Vladimir Dobrushkin
https://math.uri.edu/~dobrush/

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the appendix entitled GNU Free Documentation License.

Vector Spaces

In mathematics, physics, and engineering, a Euclidean vector (simply a vector) is a geometric object that has magnitude (or length) and direction. Many familiar physical notions, such as forces, velocitues, and accelerations, involve both magnitude (the amount of the force, velocity, or acceleration) and a direction. In most physical situations involving vectors, only the magnitude and direction of the vector are significant; consequently, we regard vectors with the same length and direction as being equal irrespective to their positions.

It is a custom to identify vectors with arrows (geometric object). The tail of the arrow is called the initial point of the vector and the tip the terminal point. To emphasis this approach, an arrow is placed above the initial and terminal points, for example, the notation \( {\bf v} = \vec{AB} \) tells us that A is the starting point of the vector v and its terminal point is B. In this tutorial (as in most science papers and textbooks), we will denote vectors in boldface type applied to lower case letters such as v, u, or x.

Any two vectors x and y can be added in "tail-to-head" manner; that is, either x or y may be applied to any point and then another vector is applied to the endpoint of the first. If this is done, the endpoint of the latter is the endpoint of their sum, which is denoted by x + y. Besides the operation of vector addition there is another natural operation that can be performed on vectors---multiplication by a scalar that are often taken to be real numbers. When a vector is multiplied by a real number k, its magnitude is multiplied by |k| and its direction remains the same when k is positive and the opposite direction when k is negative. Such vector is denoted by kx.

The concept of vector, as we know it today, evolved gradually over a period of more than 200 years. The Italian mathematician, senator, and municipal councilor Giusto Bellavitis (1803--1880) abstracted the basic idea in 1835. The term vector was introduced by the Irish mathematician, astronomer, and mathematical physicist William Rowan Hamilton (1805--1865) as part of a quaternion.

Vectors can be described also algebraically. Historically, the first vectors were Euclidean vectors that can be expanded through standard basic vectors that are used as coordinates. Then any vector can be uniquely represented by a sequence of scalars called coordinates or components. The set of such ordered n-tuples is denoted by \( \mathbb{R}^n . \) When scalars are complex numbers, the set of ordered n-tuples of complex numbers is denoted by \( \mathbb{C}^n . \) Motivated by these two approaches, we present the general definition of vectors.

1. Associativity of addition: \( ({\bf v} + {\bf u}) + {\bf w} = {\bf v} + ({\bf u} + {\bf w}) \) for \( ({\bf v} , {\bf u}) , {\bf w} \in V . \)
2. Commutativity of addition: \( ({\bf v} + {\bf u}) = {\bf u} + {\bf v} \) for \( ({\bf v} , {\bf u}) \in V . \)
3. Identity element of addition: there exists an element \( ({\bf 0} \in V , \) called the zero vector, such that \( {\bf v} +{\bf 0}) = {\bf v} \) for every vector from V.
4. Inverse elements of addition: for every vector v, there exists an element \( -{\bf v} \in V , \) called the additive inverse of v, such that \( {\bf v} + (-{\bf v}) = {\bf 0} . \)
5. Compatibility of scalar multiplication with field multiplication: \( a(b{\bf v}) = (ab){\bf v} \) for any scalars a and b and arbitrary vector v.
6. Identity element of scalar multiplication: \( 1{\bf v} = {\bf v} , \) where 1 denotes the multiplicative identity.
7. Distributivity of scalar multiplication with respect to vector addition: \( k\left( {\bf v} + {\bf u}\right) = k{\bf v} + k{\bf u} \) for any scalar k and arbitary vectors v and u.
8. Distributivity of scalar multiplication with respect to field addition: \( \left( a+b \right) {\bf v} = a\,{\bf v} + b\,{\bf v} \) for any two scalars a and b and arbitrary vector v. ■

Historically, the first ideas leading to vector spaces can be traced back as far as the 17th century; however, the idia crystallized with the work of the Prussian/German mathematician Hermann Günther Grassmann (1809--1877), who published a paper in 1862. He was also a linguist, physicist, neohumanist, general scholar, and publisher. His mathematical work was little noted until he was in his sixties. It is interested that while he was a student at the University of Berlin, Hermann studied theology, also taking classes in classical languages, philosophy, and literature. He does not appear to have taken courses in mathematics or physics. Although lacking university training in mathematics, it was the field that most interested him when he returned to Stettin (Province of Pomerania, Kingdom of Prussia; present-day Szczecin, Poland) in 1830 after completing his studies in Berlin.

Example: Standard Euclidean vector space.

Example: The set \( \mathbb{R}^n \) of all ordered n-tuples is our first familiar example of vector spaces. This space has a standard basis: \( {\bf e}_1 = (1,0,0,\ldots ,0 ) ,\quad {\bf e}_2 = (0,1,0,\ldots , 0 ), \ldots , {\bf e}_n = (0,0,\ldots , 0,1) .\) In \( \mathbb{R}^3 \) these unit vectors are denoted by

\[ {\bf i} = \left( 1, 0 ,0 \right) , \qquad {\bf j} = \left( 0,1,0 \right) , \qquad {\bf k} = (0,0,1) . \]

Therefore, every vector \( {\bf v} = \left( v_1 , v_2 , v_3 \right) \) can be expressed as a linear combination of standard unit vectors:

\[ {\bf v} = \left( v_1 , v_2 , v_3 \right) = v_1 {\bf i} + v_2 {\bf j} + v_3 {\bf k} . \]

 

Example: Set of continuous functions.

Example: Set of continuous functions. Let |a,b| denote an open or closed or semiclosed interval on the real axis. The set \( C(|a,b|) \) of all continuous functions on the interval |a,b| is a vector space.

Example: Set of polynomials of fixed degree.

Example: A polynomial of degree n is an expression of the form

\[ p_n (x) = a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n , \]

where n is a nonnegative integer and each coefficient a_i is a scalar/number. The zero polynomial is the polynomial having all coefficients to be zero. The polynomials \( p_n (x) = a_0 + a_1 x + \cdots + a_n x^n \) and \( q_n (x) = b_0 + b_1 x + \cdots + b_m x^m \) , where for simplicity \( n\ge m , \) can be added:

\[ p_n (x) + q_m (x) = \left( a_0 + b_0 \right) + \left( a_1 + b_1 \right) x + \cdots + \left( a_m + b_m \right) x^m + a_{m+1} x^{m+1} + \cdots + a_n x^n , \]

and multiplied by a constant

\[ k\, p_n (x) = k\,a_0 + k\,a_1 x + k\,a_2 x^2 + \cdots + k\,a_n x^n . \]

Under these operations of addition and scalar multiplication, the set of all polynomials of degree not exceeding n is a vector space.

The dot product of two vectors of the same size \( {\bf x} = \left[ x_1 , x_2 , \ldots , x_n \right] \) and \( {\bf y} = \left[ y_1 , y_2 , \ldots , y_n \right] \) (independently whether they are columns or rows) is the number, denoted either by \( {\bf x} \cdot {\bf y} \) or \( \left\langle {\bf x} , {\bf y} \right\rangle ,\)

\[ \left\langle {\bf x} , {\bf y} \right\rangle = {\bf x} \cdot {\bf y} = x_1 y_1 + x_2 y_2 + \cdots + x_n y_n , \]

when entries are real, or

\[ \left\langle {\bf x} , {\bf y} \right\rangle = {\bf x} \cdot {\bf y} = \overline{x_1} y_1 + \overline{x_2} y_2 + \cdots + \overline{x_n} y_n , \]

when entries are complex.

The dot product was first introduced by the American physicist and mathematician Josiah Willard Gibbs (1839--1903) in the 1880s. An outer product is the tensor product of two coordinate vectors \( {\bf u} = \left[ u_1 , u_2 , \ldots , u_m \right] \) and \( {\bf v} = \left[ v_1 , v_2 , \ldots , v_n \right] , \) denoted \( {\bf u} \otimes {\bf v} , \) is an m-by-n matrix W such that its coordinates satisfy \( w_{i,j} = u_i v_j . \) The outer product \( {\bf u} \otimes {\bf v} , \) is equivalent to a matrix multiplication \( {\bf u} \, {\bf v}^{\ast} , \) (or \( {\bf u} \, {\bf v}^{\mathrm T} , \) if vectors are real) provided that u is represented as a column \( m \times 1 \) vector, and v as a column \( n \times 1 \) vector. Here \( {\bf v}^{\ast} = \overline{{\bf v}^{\mathrm T}} . \)

For three dimensional vectors \( {\bf a} = a_1 \,{\bf i} + a_2 \,{\bf j} + a_3 \,{\bf k} = \left[ a_1 , a_2 , a_3 \right] \) and \( {\bf b} = b_1 \,{\bf i} + b_2 \,{\bf j} + b_3 \,{\bf k} = \left[ b_1 , b_2 , b_3 \right] \) , it is possible to define special multiplication, called cross-product:

\[ {\bf a} \times {\bf b} = \det \left[ \begin{array}{ccc} {\bf i} & {\bf j} & {\bf k} \\ a_1 & a_2 & a_3 \\ b_1 & b_2 & b_3 \end{array} \right] = {\bf i} \left( a_2 b_3 - b_2 a_3 \right) - {\bf j} \left( a_1 b_3 - b_1 a_3 \right) + {\bf k} \left( a_1 b_2 - a_2 b_1 \right) . \]

Example: Outer product.

Example: For instance, if m = 4 and n = 3, then

\[ {\bf u} \otimes {\bf v} = {\bf u} \, {\bf v}^{\mathrm T} = \begin{bmatrix} u_1 \\ u_2 \\ u_3 \\ u_4 \end{bmatrix} \begin{bmatrix} v_1 & v_2 & v_3 \end{bmatrix} = \begin{bmatrix} u_1 v_1 & u_1 v_2 & u_1 v_3 \\ u_2 v_1 & u_2 v_2 & u_2 v_3 \\ u_3 v_1 & u_3 v_2 & u_3 v_3 \\ u_4 v_1 & u_4 v_2 & u_4 v_3 \end{bmatrix} . \]

In Mathematica, the outer product has a special command:

Outer[Times, {1, 2, 3, 4}, {a, b, c}]

Out[1]= {{a, b, c}, {2 a, 2 b, 2 c}, {3 a, 3 b, 3 c}, {4 a, 4 b, 4 c}}

An inner product of two vectors of the same size, usually denoted by \( \left\langle {\bf x} , {\bf y} \right\rangle ,\) is a generalization of the dot product if it satisfies the following properties:

• \( \left\langle {\bf v}+{\bf u} , {\bf w} \right\rangle = \left\langle {\bf v} , {\bf w} \right\rangle + \left\langle {\bf u} , {\bf w} \right\rangle . \)
• \( \left\langle {\bf v} , \alpha {\bf u} \right\rangle = \alpha \left\langle {\bf v} , {\bf u} \right\rangle \) for any scalar α.
• \( \left\langle {\bf v} , {\bf u} \right\rangle = \overline{\left\langle {\bf u} , {\bf v} \right\rangle} , \) where overline means complex conjugate.
• \( \left\langle {\bf v} , {\bf v} \right\rangle \ge 0 , \) and equal if and only if \( {\bf v} = {\bf 0} . \)

The fourth condition in the list above is known as the positive-definite condition.

A vector space together with the inner product is called an inner product space. Every inner product space is a metric space. The metric or norm is given by

\[ \| {\bf u} \| = \sqrt{\left\langle {\bf u} , {\bf u} \right\rangle} . \]

The nonzero vectors u and v of the same size are orthogonal (or perpendicular) when their inner product is zero: \( \left\langle {\bf u} , {\bf v} \right\rangle = 0 . \) We abbreviate it as \( {\bf u} \perp {\bf v} . \) If A is an \( n \times n \) positive definite matrix and u and v are n-vectors, then we can define the weighted Euclidean inner product

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf v} = {\bf u} \cdot {\bf A}^{\ast} {\bf v} \qquad\mbox{and} \qquad {\bf u} \cdot {\bf A} {\bf v} = {\bf A}^{\ast} {\bf u} \cdot {\bf v} . \]

In particular, if w₁, w₂, ... , w_n are positive real numbers, which are called weights, and if u = ( u₁, u₂, ... , u_n) and v = ( v₁, v₂, ... , v_n) are vectors in \( \mathbb{R}^n , \) then the formular

\[ \left\langle {\bf u} , {\bf v} \right\rangle = w_1 u_1 v_1 + w_2 u_2 v_2 + \cdots + w_n u_n v_n \]

defines an inner product on \( \mathbb{R}^n , \) that is called the weighted Euclidean inner product with weights w₁, w₂, ... , w_n. Example: Inner product generated by a matrix.

Example: The Euclidean inner product and the weighted Euclidean inner product (when \( \left\langle {\bf u} , {\bf v} \right\rangle = \sum_{k=1}^n a_k u_k v_k , \) for some positive numbers \( a_k , \ (k=1,2,\ldots , n \) ) are special cases of a general class of inner products on \( \mathbb{R}^n \) called matrix inner product. Let A be an invertible n-by-n matrix. Then the formula

\[ \left\langle {\bf u} , {\bf v} \right\rangle = {\bf A} {\bf u} \cdot {\bf A} {\bf v} = {\bf v}^{\mathrm T} {\bf A}^{\mathrm T} {\bf A} {\bf u} \]

defines an inner product generated by A.

Example: Space of integrable functions.

Example: In the set of integrable functions on an interval [a,b], we can define the inner product of two functions f and g as

\[ \left\langle f , g \right\rangle = \int_a^b \overline{f} (x)\, g(x) \, {\text d}x \qquad\mbox{or} \qquad \left\langle f , g \right\rangle = \int_a^b f(x)\,\overline{g} (x) \, {\text d}x . \]

Then the norm \( \| f \| \) (also called the 2-norm) becomes the square root of

\[ \| f \|^2 = \left\langle f , f \right\rangle = \int_a^b \left\vert f(x) \right\vert^2 \, {\text d}x . \]

In particular, the 2-norm of the function \( f(x) = 5x^2 +2x -1 \) on the interval [0,1] is

\[ \| 2 x^2 +2x -1 \| = \sqrt{\int_0^1 \left( 5x^2 +2x -1 \right)^2 {\text d}x } = \sqrt{7} . \]

Example: Space of polynomials.

Example: Consider a set of polynomials of degree n. If

\[ {\bf p} = p(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_n x^n \quad\mbox{and} \quad {\bf q} = q(x) = q_0 + q_1 x + q_2 x^2 + \cdots + q_n x^n \]

are two polynomials, and if \( x_0 , x_1 , \ldots , x_n \) are distinct real numbers (called sample points), then the formula

\[ \left\langle {\bf p} , {\bf q} \right\rangle = p(x_0 ) q(x_0 ) + p_1 (x_1 )q(x_1 ) + \cdots + p(x_n ) q(x_n ) \]

defines an inner product, which is called the evaluation inner product at \( x_0 , x_1 , \ldots , x_n . \) ■

With dot product, we can assign a length of a vector, which is also called the Euclidean norm or 2-norm:

\[ \| {\bf x} \|_2 = \| {\bf x} \| = \sqrt{ {\bf x}\cdot {\bf x}} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} . \]

This norm can be generalized for arbitrary real p: \[ \| {\bf x} \|_p = \left( x_1^p + x_2^p + \cdots + x_n^p \right)^{1/p} . \]

The norm corresponding p=1 has a special name: Taxicab norm or Manhattan norm, which is also called 1-norm:

\[ \| {\bf x} \|_1 = \sum_{k=1}^n | x_k | = |x_1 | + |x_2 | + \cdots + |x_n | . \]

Another commonly used norm is

\[ \| {\bf x} \|_{\infty} = \sup_{1\le k \le n} | x_k | . \]

It is not hard to verify that these normes satisfy

\begin{align*} \| {\bf x} \|_{\infty} &\le \| {\bf x} \|_1 \le n\,\| {\bf x} \|_{\infty} , \\ \| {\bf x} \|_2 &\le \| {\bf x} \|_1 \le \sqrt{n} \| {\bf x} \|_2 , \\ \| {\bf x} \|_{\infty} &\le \| {\bf x} \|_2 \le \sqrt{n} \| {\bf x} \|_{\infty} , \end{align*}

i.e., these norms are equivalent. Example: Mathematica calculates the norms.

Example: Taking a vector \( {\bf v} = \left( 2, {\bf j} , -2 \right) , \) we calculate norms:

Norm[{2, \[ImaginaryJ], -2}]  

Norm[{2, \[ImaginaryJ], -2}, 3/2]

Out[1]=
Out[2]=

In linear algebra, functional analysis, and related areas of mathematics, a norm is a function that assigns a strictly positive length or size to each vector in a vector space—save for the zero vector, which is assigned a length of zero. On an n-dimensional complex space \( \mathbb{C}^n ,\) the most common norm is

\[ \| {\bf z} \| = \sqrt{ {\bf z}\cdot {\bf z}} = \sqrt{\overline{z_1} \,z_1 + \overline{z_2}\,z_2 + \cdots + \overline{z_n}\,z_n} = \sqrt{|z_1|^2 + |z_2 |^2 + \cdots + |z_n |^2} . \]

A unit vector u is a vector whose length equals one: \( {\bf u} \cdot {\bf u} =1 . \) We say that two vectors x and y are perpendicular if their dot product is zero. There are known many other norms.

Augustin-Louis Cauchy		Viktor Yakovlevich Bunyakovsky		Hermann Amandus Schwarz

For any norm, the Cauchy--Bunyakovsky--Schwarz (or simply CBS) inequality holds:

\[ | {\bf x} \cdot {\bf y} | \le \| {\bf x} \| \, \| {\bf y} \| . \]

The inequality for sums was published by the French mathematician and physicist Augustin-Louis Cauchy (1789--1857) in 1821, while the corresponding inequality for integrals was first proved by the Russian mathematicain Viktor Yakovlevich Bunyakovsky (1804--1889) in 1859. The modern proof (which is actually a repetition of the Bunyakovsky's one) of the integral inequality was given by the German mathematician Hermann Amandus Schwarz (1843--1921) in 1888. With Euclidean norm, we can define the dot product as

\[ {\bf x} \cdot {\bf y} = \| {\bf x} \| \, \| {\bf y} \| \, \cos \theta , \]

where \( \theta \) is the angle between two vectors. ■