The Wolfram Mathematica notebook which contains the code that produces all the Mathematica output in this web page may be downloaded at this link. Caution: This notebook will evaluate, cell-by-cell, sequentially, from top to bottom. However, due to re-use of variable names in later evaluations, once subsequent code is evaluated prior code may not render properly. Returning to and re-evaluating the first Clear[ ] expression above the expression no longer working and evaluating from that point through to the expression solves this problem.

$Post := If[MatrixQ[#1], MatrixForm[#1], #1] & (* outputs matricies in MatrixForm*)
Remove[ "Global`*"] // Quiet (* remove all variables *)

In this section, we consider vector spaces over field ℝ, a set of real numbers or ℂ, a set of complex numbers.

Inner Product

An inner product of two vectors of the same size, usually denoted by $ \left\langle {\bf x} , {\bf y} \right\rangle ,$ is a generalization of the dot product if it satisfies the following properties:

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

The fourth condition in the list above is known as the positive-definite condition. A vector space together with the inner product is called an inner product space. Every inner product space is a metric space. The metric or norm is given by

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

\[ {\bf x} \bullet {\bf y} = \overline{x_1} y_1 + \overline{x_2} y_2 + \cdots + \overline{x_n} y_n , \]

when entries are complex. Here $ \overline{\bf x} = \overline{a + {\bf j}\, b} = a - {\bf j}\,b = {\bf x}^{\ast} {\bf y} $ is a complex conjugate of a complex number x = a + jb.

Nonzero vectors u and v of the same size are orthogonal (or perpendicular) when their inner product is zero: $ \left\langle {\bf u} , {\bf v} \right\rangle = 0 . $ We abbreviate it as $ {\bf u} \perp {\bf v} . $ If A is an n × n positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf v} = {\bf u} \cdot {\bf A}^{\ast} {\bf v} \qquad\mbox{and} \qquad {\bf u} \cdot {\bf A} {\bf v} = {\bf A}^{\ast} {\bf u} \cdot {\bf v} . \]

In particular, if w₁, w₂, ... , w_n are positive real numbers, which are called weights, and if u = ( u₁, u₂, ... , u_n) and v = ( v₁, v₂, ... , v_n) are vectors in ℝⁿ, then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Example 4: The Euclidean inner product and the weighted Euclidean inner product (when $ \left\langle {\bf u} , {\bf v} \right\rangle = \sum_{k=1}^n a_k u_k v_k , $ for some positive numbers $ a_k , \ (k=1,2,\ldots , n $ ) are special cases of a general class of inner products on $ \mathbb{R}^n $ called matrix inner product. Let A be an invertible n-by-n matrix. Then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf A} {\bf v} = {\bf v}^{\mathrm T} {\bf A}^{\mathrm T} {\bf A} {\bf u} \]

defines an inner product generated by A.

Example 5: In the set of integrable functions on an interval [a,b], we can define the inner product of two functions f and g as

\[ \left\langle f , g \right\rangle = \int_a^b \overline{f} (x)\, g(x) \, {\text d}x \qquad\mbox{or} \qquad \left\langle f , g \right\rangle = \int_a^b f(x)\,\overline{g} (x) \, {\text d}x . \]

Then the norm $ \| f \| $ (also called the 2-norm or 𝔏² norm) becomes the square root of

\[ \| f \|^2 = \left\langle f , f \right\rangle = \int_a^b \left\vert f(x) \right\vert^2 \, {\text d}x . \]

In particular, the 2-norm of the function $ f(x) = 5x^2 +2x -1 $ on the interval [0,1] is

\[ \| 2 x^2 +2x -1 \| = \sqrt{\int_0^1 \left( 5x^2 +2x -1 \right)^2 {\text d}x } = \sqrt{7} . \]

Example 6: Consider a set of polynomials of degree n. If

\[ {\bf p} = p(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_n x^n \quad\mbox{and} \quad {\bf q} = q(x) = q_0 + q_1 x + q_2 x^2 + \cdots + q_n x^n \]

are two polynomials, and if $ x_0 , x_1 , \ldots , x_n $ are distinct real numbers (called sample points), then the formula

\[ \left\langle {\bf p} , {\bf q} \right\rangle = p(x_0 ) q(x_0 ) + p_1 (x_1 )q(x_1 ) + \cdots + p(x_n ) q(x_n ) \]

defines an inner product, which is called the evaluation inner product at $ x_0 , x_1 , \ldots , x_n . $ ■

The invention of Cartesian coordinates in 1649 by René Descartes (Latinized name: Cartesius) revolutionized mathematics by providing the first systematic link between Euclidean geometry and algebra.

============================================================

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

The nonzero vectors u and v of the same size are orthogonal (or perpendicular) when their inner product is zero: $ \left\langle {\bf u} , {\bf v} \right\rangle = 0 . $ We abbreviate it as $ {\bf u} \perp {\bf v} . $ If A is an n × n positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Riesz representation theorem: Let V be a finite dimensional vector space over the field 𝔽 (𝔽 = ℝ, ℂ) on which ⟨·, ·⟩ is an inner product. Let φ : V ⇾ 𝔽 be a linear functional on V. Then there exists a unique vector u ∈ V such that φ(v) = ⟨u, v⟩ for all v ∈ V.

Using the Gram–Schmidt orthogonalization process we can find an orthonormal basis of V, say, {v₁, v₂, … , v_n}. Now for an arbitrary vector v ∈ V, we have \[ {\bf v} = \langle {\bf v}_1 , {\bf v} \rangle {\bf v}_1 + \langle {\bf v}_2 , {\bf v} \rangle {\bf v}_2 + \cdots + \langle {\bf v}_n , {\bf v} \rangle {\bf v}_n . \] Then it follows that \[ \varphi ({\bf v}) = \langle {\bf v}_1 , {\bf v} \rangle \varphi ({\bf v}_1 ) + \langle {\bf v}_2 , {\bf v} \rangle \varphi ({\bf v}_2 ) + \cdots + \langle {\bf v}_n , {\bf v} \rangle \varphi ({\bf v}_n ) = \left\langle {\bf v}_1 \varphi^{\ast} ({\bf v}_1 ) + \cdots + {\bf v}_n \varphi^{\ast} ({\bf v}_n ) , {\bf v} \right\rangle \] Denoting by \[ {\bf u} = {\bf v}_1 \varphi^{\ast} ({\bf v}_1 ) + \cdots + {\bf v}_n \varphi^{\ast} ({\bf v}_n ) , \] the result follows.

Suppose there exist two vectors u¹ and u² in the Riesz representation theorem. Then for w = u¹ − u², we have ⟨w, v⟩ = 0 for all v ∈ V. But then for w ∈ V, it follows that ⟨w, w⟩ = ∥ w ∥² = 0, which implies that u¹ = u².

to be checked ========================

$ \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . $
$ \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle $ for any scalar α.
$ \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , $ where overline means complex conjugate.
$ \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , $ and equal if and only if $ {\bf v} = {\bf 0} . $

The fourth condition in the list above is known as the positive-definite condition.

A vector space together with the inner product is called an inner product space. Every inner product space is a metric space. The metric or norm is given by

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

A generalized length function on a vector space can be imposed in many different ways, not necessarily through the inner product. What is important that this generalized length, called in mathematics a norm, should satisfy the following four axioms.

A norm on a vector space V is a nonnegative function $ \| \, \cdot \, \| \, : \, V \to [0, \infty ) $ that satisfies the following axioms for any vectors $ {\bf u}, {\bf v} \in V $ and arbitrary scalar k.

$ \| {\bf u} \| $ is real and nonnegative;
$ \| {\bf u} \| =0 $ if and only if u = 0;
$ \| k\,{\bf u} \| = |k| \, \| {\bf u} \| ;$
$ \| {\bf u} + {\bf v} \| \le \| {\bf u} \| + \| {\bf v} \| . $

With any positive definite (having positive eigenvalues) matrix one can define a corresponding norm. If A is an $ n \times n $ positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

In particular, if w₁, w₂, ... , w_n are positive real numbers, which are called weights, and if u = ( u₁, u₂, ... , u_n) and v = ( v₁, v₂, ... , v_n) are vectors in $ \mathbb{R}^n , $ then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on $ \mathbb{R}^n , $ that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n.

Example: The Euclidean inner product and the weighted Euclidean inner product (when $ \left\langle {\bf u} , {\bf v} \right\rangle = \sum_{k=1}^n a_k u_k v_k , $ for some positive numbers $ a_k , \ (k=1,2,\ldots , n $ ) are special cases of a general class of inner products on $ \mathbb{R}^n $ called matrix inner product. Let A be an invertible n-by-n matrix. Then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf A} {\bf v} = {\bf v}^{\mathrm T} {\bf A}^{\mathrm T} {\bf A} {\bf u} \]

defines an inner product generated by A.

Example: In the set of integrable functions on an interval [a,b], we can define the inner product of two functions f and g as

\[ \left\langle f , g \right\rangle = \int_a^b \overline{f} (x)\, g(x) \, {\text d}x \qquad\mbox{or} \qquad \left\langle f , g \right\rangle = \int_a^b f(x)\,\overline{g} (x) \, {\text d}x . \]

Then the norm $ \| f \| $ (also called the 2-norm) becomes the square root of

\[ \| f \|^2 = \left\langle f , f \right\rangle = \int_a^b \left\vert f(x) \right\vert^2 \, {\text d}x . \]

In particular, the 2-norm of the function $ f(x) = 5x^2 +2x -1 $ on the interval [0,1] is

\[ \| 2 x^2 +2x -1 \| = \sqrt{\int_0^1 \left( 5x^2 +2x -1 \right)^2 {\text d}x } = \sqrt{7} . \]

Example: Consider a set of polynomials of degree n. If

\[ {\bf p} = p(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_n x^n \quad\mbox{and} \quad {\bf q} = q(x) = q_0 + q_1 x + q_2 x^2 + \cdots + q_n x^n \]

are two polynomials, and if $ x_0 , x_1 , \ldots , x_n $ are distinct real numbers (called sample points), then the formula

\[ \left\langle {\bf p} , {\bf q} \right\rangle = p(x_0 ) q(x_0 ) + p_1 (x_1 )q(x_1 ) + \cdots + p(x_n ) q(x_n ) \]

defines an inner product, which is called the evaluation inner product at $ x_0 , x_1 , \ldots , x_n . $ ■

With dot product, we can assign a length of a vector, which is also called the Euclidean norm or 2-norm:

\[ \| {\bf x} \|_2 = \| {\bf x} \| = \sqrt{ {\bf x}\cdot {\bf x}} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} . \]

This norm can be generalized for arbitrary real p: \[ \| {\bf x} \|_p = \left( x_1^p + x_2^p + \cdots + x_n^p \right)^{1/p} . \]

Example: Taking a vector $ {\bf v} = \left( 2, {\bf j} , -2 \right) , $ we calculate norms:

Norm[{2, \[ImaginaryJ], -2}]
Norm[{2, \[ImaginaryJ], -2}, 3/2]

Out[1]= 3
Out[2]= (1 + 4 Sqrt[2])^(2/3)

In linear algebra, functional analysis, and related areas of mathematics, a norm is a function that assigns a strictly positive length or size to each vector in a vector space—save for the zero vector, which is assigned a length of zero. On an n-dimensional complex space $ \mathbb{C}^n ,$ the most common norm is

\[ \| {\bf z} \| = \sqrt{ {\bf z}\cdot {\bf z}} = \sqrt{\overline{z_1} \,z_1 + \overline{z_2}\,z_2 + \cdots + \overline{z_n}\,z_n} = \sqrt{|z_1|^2 + |z_2 |^2 + \cdots + |z_n |^2} . \]

A unit vector u is a vector whose length equals one: $ {\bf u} \cdot {\bf u} =1 . $ We say that two vectors x and y are perpendicular if their dot product is zero. There are known many other norms.


Augustin-Louis Cauchy		Viktor Yakovlevich Bunyakovsky		Hermann Amandus Schwarz

For any norm, the Cauchy--Bunyakovsky--Schwarz (or simply CBS) inequality holds:

\[ | {\bf x} \cdot {\bf y} | \le \| {\bf x} \| \, \| {\bf y} \| . \]

The inequality for sums was published by the French mathematician and physicist Augustin-Louis Cauchy (1789--1857) in 1821, while the corresponding inequality for integrals was first proved by the Russian mathematician Viktor Yakovlevich Bunyakovsky (1804--1889) in 1859. The modern proof (which is actually a repetition of the Bunyakovsky's one) of the integral inequality was given by the German mathematician Hermann Amandus Schwarz (1843--1921) in 1888. With Euclidean norm, we can define the dot product as

\[ {\bf x} \cdot {\bf y} = \| {\bf x} \| \, \| {\bf y} \| \, \cos \theta , \]

where $ \theta $ is the angle between two vectors. ■

Introduction to Linear Algebra

Systems of Linear Equations

Matrix Algebra

Vector Spaces

Eigenvalues, Eigenvectors

Euclidean Spaces

Matrix Decompositions

Applications

Functions of Matrices

Miscellany

Preliminaries

Glossary

Reference

Inner Product