The Wolfram Mathematica notebook which contains the code that produces all the Mathematica output in this web page may be downloaded at this link. Caution: This notebook will evaluate, cell-by-cell, sequentially, from top to bottom. However, due to re-use of variable names in later evaluations, once subsequent code is evaluated prior code may not render properly. Returning to and re-evaluating the first Clear[ ] expression above the expression no longer working and evaluating from that point through to the expression solves this problem.

$Post := If[MatrixQ[#1], MatrixForm[#1], #1] & (* outputs matricies in MatrixForm*)
Remove[ "Global`*"] // Quiet (* remove all variables *)

In this section, we consider vector spaces over one of the following two fields: either ℝ, a set of real numbers or ℂ, a set of complex numbers.

Inner Product

An inner product of two vectors of the same size, usually denoted by $ \left\langle {\bf x} , {\bf y} \right\rangle ,$ is a generalization of the dot product if it satisfies the following properties:

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

The fourth condition in the list above is known as the positive-definite condition. A vector space together with the inner product is called an inner product space. Every inner product space is a metric space. The metric or norm is given by

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

\[ {\bf x} \bullet {\bf y} = \overline{x_1} y_1 + \overline{x_2} y_2 + \cdots + \overline{x_n} y_n , \]

when entries are complex. Here $ \overline{\bf x} = \overline{a + {\bf j}\, b} = a - {\bf j}\,b = {\bf x}^{\ast} {\bf y} $ is a complex conjugate of a complex number x = a + jb.

Nonzero vectors u and v of the same size are orthogonal (or perpendicular) when their inner product is zero: $ \left\langle {\bf u} , {\bf v} \right\rangle = 0 . $ We abbreviate it as $ {\bf u} \perp {\bf v} . $ If A is an n × n positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf v} = {\bf u} \cdot {\bf A}^{\ast} {\bf v} \qquad\mbox{and} \qquad {\bf u} \cdot {\bf A} {\bf v} = {\bf A}^{\ast} {\bf u} \cdot {\bf v} . \]

In particular, if w₁, w₂, ... , w_n are positive real numbers, which are called weights, and if u = ( u₁, u₂, ... , u_n) and v = ( v₁, v₂, ... , v_n) are vectors in ℝⁿ, then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Example 4: The Euclidean inner product and the weighted Euclidean inner product (when $ \left\langle {\bf u} , {\bf v} \right\rangle = \sum_{k=1}^n a_k u_k v_k , $ for some positive numbers $ a_k , \ (k=1,2,\ldots , n $ ) are special cases of a general class of inner products on $ \mathbb{R}^n $ called matrix inner product. Let A be an invertible n-by-n matrix. Then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf A} {\bf v} = {\bf v}^{\mathrm T} {\bf A}^{\mathrm T} {\bf A} {\bf u} \]

defines an inner product generated by A.

Example 5: In the set of integrable functions on an interval [a,b], we can define the inner product of two functions f and g as

\[ \left\langle f , g \right\rangle = \int_a^b \overline{f} (x)\, g(x) \, {\text d}x \qquad\mbox{or} \qquad \left\langle f , g \right\rangle = \int_a^b f(x)\,\overline{g} (x) \, {\text d}x . \]

Then the norm $ \| f \| $ (also called the 2-norm or 𝔏² norm) becomes the square root of

\[ \| f \|^2 = \left\langle f , f \right\rangle = \int_a^b \left\vert f(x) \right\vert^2 \, {\text d}x . \]

In particular, the 2-norm of the function $ f(x) = 5x^2 +2x -1 $ on the interval [0,1] is

\[ \| 2 x^2 +2x -1 \| = \sqrt{\int_0^1 \left( 5x^2 +2x -1 \right)^2 {\text d}x } = \sqrt{7} . \]

Example 6: Consider a set of polynomials of degree n. If

\[ {\bf p} = p(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_n x^n \quad\mbox{and} \quad {\bf q} = q(x) = q_0 + q_1 x + q_2 x^2 + \cdots + q_n x^n \]

are two polynomials, and if $ x_0 , x_1 , \ldots , x_n $ are distinct real numbers (called sample points), then the formula

\[ \left\langle {\bf p} , {\bf q} \right\rangle = p(x_0 ) q(x_0 ) + p_1 (x_1 )q(x_1 ) + \cdots + p(x_n ) q(x_n ) \]

defines an inner product, which is called the evaluation inner product at $ x_0 , x_1 , \ldots , x_n . $ ■

The invention of Cartesian coordinates in 1649 by René Descartes (Latinized name: Cartesius) revolutionized mathematics by providing the first systematic link between Euclidean geometry and algebra.

============================================================

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

The nonzero vectors u and v of the same size are orthogonal (or perpendicular) when their inner product is zero: $ \left\langle {\bf u} , {\bf v} \right\rangle = 0 . $ We abbreviate it as $ {\bf u} \perp {\bf v} . $ If A is an n × n positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Riesz representation theorem: Let V be a finite dimensional vector space over the field 𝔽 (𝔽 = ℝ, ℂ) on which ⟨·, ·⟩ is an inner product. Let φ : V ⇾ 𝔽 be a linear functional on V. Then there exists a unique vector u ∈ V such that φ(v) = ⟨u, v⟩ for all v ∈ V.

In 1912, the Hungarian mathematician Frigyes Riesz established an isomorphism between an Euclidean space and its dual space. His result (which is also valid for some infinite dimensional spaces) restores equal right between vectors and covectors, but under a new marriage sectificate---known as the inner product, which is our next topic to discuss.

Using the Gram–Schmidt orthogonalization process we can find an orthonormal basis of V, say, {v₁, v₂, … , v_n}. Now for an arbitrary vector v ∈ V, we have \[ {\bf v} = \langle {\bf v}_1 , {\bf v} \rangle {\bf v}_1 + \langle {\bf v}_2 , {\bf v} \rangle {\bf v}_2 + \cdots + \langle {\bf v}_n , {\bf v} \rangle {\bf v}_n . \] Then it follows that \[ \varphi ({\bf v}) = \langle {\bf v}_1 , {\bf v} \rangle \varphi ({\bf v}_1 ) + \langle {\bf v}_2 , {\bf v} \rangle \varphi ({\bf v}_2 ) + \cdots + \langle {\bf v}_n , {\bf v} \rangle \varphi ({\bf v}_n ) = \left\langle {\bf v}_1 \varphi^{\ast} ({\bf v}_1 ) + \cdots + {\bf v}_n \varphi^{\ast} ({\bf v}_n ) , {\bf v} \right\rangle \] Denoting by \[ {\bf u} = {\bf v}_1 \varphi^{\ast} ({\bf v}_1 ) + \cdots + {\bf v}_n \varphi^{\ast} ({\bf v}_n ) , \] the result follows.

Suppose there exist two vectors u¹ and u² in the Riesz representation theorem. Then for w = u¹ − u², we have ⟨w, v⟩ = 0 for all v ∈ V. But then for w ∈ V, it follows that ⟨w, w⟩ = ∥ w ∥² = 0, which implies that u¹ = u².

to be checked ========================

The following famous theorem (proved independently by Frigyes Riesz and Maurice René Fréchet in 1907) establishes the converse: Any linear functional T(v) corresponds to the dot product with a weight vector u.

Riesz representation theorem: Let f be a linear form on n-dimensional vector space V. Then there exists a unique vector u such that for all v ∈ V, f(v) = u • v

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

The fourth condition in the list above is known as the positive-definite condition.

A vector space together with the inner product is called an inner product space. Every inner product space is a metric space. The metric or norm is given by

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

A generalized length function on a vector space can be imposed in many different ways, not necessarily through the inner product. What is important that this generalized length, called in mathematics a norm, should satisfy the following four axioms.

A norm on a vector space V is a nonnegative function $ \| \, \cdot \, \| \, : \, V \to [0, \infty ) $ that satisfies the following axioms for any vectors $ {\bf u}, {\bf v} \in V $ and arbitrary scalar k.

$ \| {\bf u} \| $ is real and nonnegative;
$ \| {\bf u} \| =0 $ if and only if u = 0;
$ \| k\,{\bf u} \| = |k| \, \| {\bf u} \| ;$
$ \| {\bf u} + {\bf v} \| \le \| {\bf u} \| + \| {\bf v} \| . $

With any positive definite (having positive eigenvalues) matrix one can define a corresponding norm. If A is an $ n \times n $ positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

In particular, if w₁, w₂, ... , w_n are positive real numbers, which are called weights, and if u = ( u₁, u₂, ... , u_n) and v = ( v₁, v₂, ... , v_n) are vectors in $ \mathbb{R}^n , $ then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Example: The Euclidean inner product and the weighted Euclidean inner product (when $ \left\langle {\bf u} , {\bf v} \right\rangle = \sum_{k=1}^n a_k u_k v_k , $ for some positive numbers $ a_k , \ (k=1,2,\ldots , n $ ) are special cases of a general class of inner products on $ \mathbb{R}^n $ called matrix inner product. Let A be an invertible n-by-n matrix. Then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf A} {\bf v} = {\bf v}^{\mathrm T} {\bf A}^{\mathrm T} {\bf A} {\bf u} \]

defines an inner product generated by A.

Example: In the set of integrable functions on an interval [a,b], we can define the inner product of two functions f and g as

\[ \left\langle f , g \right\rangle = \int_a^b \overline{f} (x)\, g(x) \, {\text d}x \qquad\mbox{or} \qquad \left\langle f , g \right\rangle = \int_a^b f(x)\,\overline{g} (x) \, {\text d}x . \]

Then the norm $ \| f \| $ (also called the 2-norm) becomes the square root of

\[ \| f \|^2 = \left\langle f , f \right\rangle = \int_a^b \left\vert f(x) \right\vert^2 \, {\text d}x . \]

In particular, the 2-norm of the function $ f(x) = 5x^2 +2x -1 $ on the interval [0,1] is

\[ \| 2 x^2 +2x -1 \| = \sqrt{\int_0^1 \left( 5x^2 +2x -1 \right)^2 {\text d}x } = \sqrt{7} . \]

Example: Consider a set of polynomials of degree n. If

\[ {\bf p} = p(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_n x^n \quad\mbox{and} \quad {\bf q} = q(x) = q_0 + q_1 x + q_2 x^2 + \cdots + q_n x^n \]

are two polynomials, and if $ x_0 , x_1 , \ldots , x_n $ are distinct real numbers (called sample points), then the formula

\[ \left\langle {\bf p} , {\bf q} \right\rangle = p(x_0 ) q(x_0 ) + p_1 (x_1 )q(x_1 ) + \cdots + p(x_n ) q(x_n ) \]

defines an inner product, which is called the evaluation inner product at $ x_0 , x_1 , \ldots , x_n . $ ■

With dot product, we can assign a length of a vector, which is also called the Euclidean norm or 2-norm:

\[ \| {\bf x} \|_2 = \| {\bf x} \| = \sqrt{ {\bf x}\cdot {\bf x}} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} . \]

This norm can be generalized for arbitrary real p: \[ \| {\bf x} \|_p = \left( x_1^p + x_2^p + \cdots + x_n^p \right)^{1/p} . \]

Example: Taking a vector $ {\bf v} = \left( 2, {\bf j} , -2 \right) , $ we calculate norms:

Norm[{2, \[ImaginaryJ], -2}]
Norm[{2, \[ImaginaryJ], -2}, 3/2]

Out[1]= 3
Out[2]= (1 + 4 Sqrt[2])^(2/3)

In linear algebra, functional analysis, and related areas of mathematics, a norm is a function that assigns a strictly positive length or size to each vector in a vector space—save for the zero vector, which is assigned a length of zero. On an n-dimensional complex space $ \mathbb{C}^n ,$ the most common norm is

\[ \| {\bf z} \| = \sqrt{ {\bf z}\cdot {\bf z}} = \sqrt{\overline{z_1} \,z_1 + \overline{z_2}\,z_2 + \cdots + \overline{z_n}\,z_n} = \sqrt{|z_1|^2 + |z_2 |^2 + \cdots + |z_n |^2} . \]

A unit vector u is a vector whose length equals one: $ {\bf u} \cdot {\bf u} =1 . $ We say that two vectors x and y are perpendicular if their dot product is zero. There are known many other norms.


Augustin-Louis Cauchy		Viktor Yakovlevich Bunyakovsky		Hermann Amandus Schwarz

For any norm, the Cauchy--Bunyakovsky--Schwarz (or simply CBS) inequality holds:

\[ | {\bf x} \cdot {\bf y} | \le \| {\bf x} \| \, \| {\bf y} \| . \]

The inequality for sums was published by the French mathematician and physicist Augustin-Louis Cauchy (1789--1857) in 1821, while the corresponding inequality for integrals was first proved by the Russian mathematician Viktor Yakovlevich Bunyakovsky (1804--1889) in 1859. The modern proof (which is actually a repetition of the Bunyakovsky's one) of the integral inequality was given by the German mathematician Hermann Amandus Schwarz (1843--1921) in 1888. With Euclidean norm, we can define the dot product as

\[ {\bf x} \cdot {\bf y} = \| {\bf x} \| \, \| {\bf y} \| \, \cos \theta , \]

where $ \theta $ is the angle between two vectors. ■

Introduction to Linear Algebra

Systems of Linear Equations

Matrix Algebra

Vector Spaces

Eigenvalues, Eigenvectors

Euclidean Spaces

Matrix Decompositions

Applications

Functions of Matrices

Miscellany

Preliminaries

Glossary

Reference

Inner Product