This section is devoted to basic manipulations with matrices that have only two dimensions: rows × columns, or matrices that you could print out and lay flat on a table. Although the number of dimensions in which a matrix lives is more open to interpretation compared to what we use in this section, matrices in higher dimensions (say that would require a 3D printer) could be found elsewhere. We discuss later matrices in physical space, which are called tensors.

Matrices are ubiquitous in mathematics, physics, and engineering because of their diverse intentions. To give you an idea of their versatility, below is a non-exhaustive list of some applications of matrices.

• Representing a linear transformation or mapping;
• Using as operators acting on vectors;
• Representing a system of equations;
• Storing data (e.g., features × observations);
• Storing kernels used in filtering or convolution;
• Representing finance information from different sectors of an economy or business;
• Deriving parameters for a model that predicts changes in the spread of an infectious disease.

When referring to matrix sizes, and when referring to indices in a matrix, it is assumed that you refer first to rows then to columns. When a matrix has m rows and n columns, we speak about m-by-n or m × n matrix; so this matrix has two dimensions, and we call it m × n or (m, n).

There are two common notations for matrices, either in square brackets or parentheses. For example,

This may be abbreviated by writing only a single generic term, possibly along with indices, as in
Note:    Since matrices can be considered as operators acting on either column vectors from left or on row vectors from right, we use bracket notation for latter and embrace matrices in parentheses when they act on row vectors from right.    ■

Although entries of a matrix could be arbitrary algebraic objects, we consider in this section only naïve entries that belong to one of the following fields: set of integers, ℤ, or rational numbers, ℚ, or real numbers, ℝ, or complex numbers, ℂ. The set of all m × n matrices over field 𝔽 is denoted by 𝔽^m×n. We use capital bold letters, such as A or M, to denote matrices.

Since mathematica does not display naturally (for humans) matrices, it is convenient to force this software package to do this job for you by entering the following code:

Manipulation of Matrices

The element in the i-th row, j-th column in matrix A is denoted A[i, j] or A_{i, j} or 𝑎_{i, j} if A = [𝑎_{i, j}]. There is no standard notation for specifying a particular row or column of a matrix. Following matlab, we use the notation A[i, :] for the i-th row of A (a row vector) and A[: , j] for the j-th column. (matlab uses parentheses rather than square brackets, however). Mathematica uses notation A[i, j] for (i, j) element of matrix A, but notation for extracting corresponding rows or columns will be given later.

 

We show that matrices of the same size form a vector space. Algebraically, matrices can be added and scalar-multiplied, just like vectors, numeric and geometric vectors.

Suppose we have two matrices of the same size

This way, A and B can now be added, element by element: To multiply matrix A by a scalar (number) r ∈ 𝔽 (either from the left or from the right), just multiply every element of the matrix by the number (scalar): Moreover, (λμ)A = λ(μA) for any scalars λ and μ.

With these two operations (addition/subtraction and scalar multiplication) we can define a linear combination of matrices:

for some scalars c₁, c₂, … , c_k and some matrices A₁, A₂, … , A_k of the same size. Then this linear combination is again a matrix of the same size. This means that the set 𝔽^m×n of all m × n matrices is a vector space. Furthermore, the above operations are also distributive in two senses. On the one hand, you can push the same matrix A into parentheses: On the other hand, you can push the same scalar r into parentheses:     Example 1: Linear combination of two 3 × 4 matrices

 

In the computer algebra system Mathematica, a matrix is considered a list of lists. In case of our naïve definition of matrices, this list is filled with scalars from field 𝔽, either ℤ (which is preferred by the laziest people in academia, like me), integers, or ℚ, the rational numbers, ℝ, the real numbers, or ℂ, the set of complex numbers. Therefore, an m-by-n matrix can be considered as a list of m rows: where (in different, but equivalent, representations by different authors and software) r_i(A) = e_iA = A[i,:] is the i-th row of matrix A, with e_i = [ δ_{i, j} ] (j = 1, 2, … , n) being unit row vectors in 𝔽^1×n: Here δ_i,j is the Kronecker delta. To define a product A x of vector x = (x₁. x₂, … , x_n) by matrix A = [𝑎_{i, j}] from left, we need two things. First, we need to convert vector x ∈ 𝔽ⁿ from n-tuple into column vectors from 𝔽^n×1. This transformation is essential for matrix multiplication from left. Otherwise, you use vector/matrix multiplication for matrix acting on vector x from right; in this case, you need to convert the n-tuple vector into row vector. Since these three vector spaces, 𝔽ⁿ (Cartesian product of n copies of field 𝔽), 𝔽^n×1 (space of column vectors), and 𝔽^1×n (space of row vectors), are isomorphic, 𝔽ⁿ ≌ 𝔽^n×1 ≌ 𝔽^1×n, there exists a one-to-one and onto linear mapping that transfers one space into another.

Second, we need a special notation that abbreviates specific calculations. The dot product of two n-dimensional vectors from 𝔽ⁿ is computed by multiplying corresponding components, and then adding all these products. That is,

where x = (x₁. x₂, … , x_n), y = (y₁. y₂, … , y_n) ∈ 𝔽ⁿ. We indicate it by a large solid dot, •, because it has many different purposes of applications, not only in matrix/vector multiplication. The dot product is not defined for vectors of different dimensions.

Note:    Although dot product defines a metric (distance) in real spaces, it has some weird properties in complex spaces; for instance, (1, j) • (1, j) = 0, where j is the imaginary unit in complex plane ℂ, so j² = −1. Therefore, the dot product is not suitable for metric definition in complex spaces--there is an inner product instead. So we use dot product in complex spaces only formally, as a short-cut for linear combinations.    ■

When written this way, we can use the “two-hands" method to compute the product: the left hand moves from left to right along each row of the matrix while the right hand moves from top to bottom along the column vector.

\[ \left[ \begin{array}{cccccc} ◯ && ◯ && \cdots & & ◯ \\ ⬤ & \rightarrow & ⬤ &\rightarrow & \cdots & \rightarrow & ⬤\\ ◯ && ◯ && \cdots & & ◯ \\ \vdots && \vdots && \ddots && \vdots \\ ◯ && ◯ && \cdots & & ◯ \end{array} \right] \cdot \left[ \begin{array}{c} ⬤ \\ \downarrow \\ ⬤ \\ \downarrow \\ \vdots \\ \downarrow \\ ⬤ \end{array} \right] = \left[ \begin{array}{ccc} ⬡ \\ ⬣ \\ ⬡ \\ \vdots \\ ⬡ \end{array} \right] \]

Multiplication: row by column.

This allows us to define matrix/vector multiplication as the column m vector:

On another hand, we can write an m × n matrix A column by column (array of column vectors):

where c_j(A) = A e^T_j = A[:, j] is the j-th column of matrix A, j = 1, 2, … , n. here e^T_j is column vector obtained from row vector e_j by writing it vertically (T indicates transposition). This way, each column of A is in 𝔽^m×1. Let us consider now a different column vector, not in 𝔽^m×1 but rather in 𝔽^n×1: Since there are exactly n components in matrix A representation via columns, we can now scan A column by column, multiplying each column by the corresponding component from vector x, and sum up This indeed represents A x as a linear combination of the columns of A, with coefficients taken as components of vector x. Hence, matrix multiplication defines a linear transformation: A matrix A as an operator between column spaces, generates a linear transformation because 𝔽^n×1 is isomorphic to 𝔽ⁿ. Such multiplication of matrix and column vector given in Eq,(3) is called matrix/vector operation, or more precisely, matrix/column multiplication.

We can also define a dual multiplication on a vector from right If y = [y₁, y₂, … , y_m] ∈ 𝔽^1×m is a row vector of length m, we define its product with m×n matrix A as follows:

This rule (4) is known as vector/matrix operation or more precisely, row/matrix multiplication. The important features of matrix/vector and vector/matrix multiplications are that they provide the connection between linear transformations and matrices: To apply a transform to a vector, you convert that transform into a matrix, and then you multiply the vector by that matrix, either from left considering vector as a column or from right representing a vector as a row. Multiplying m-by-n matrix A by vector x depends on the size pf the vector. If x has length n, then you have to convert it into column vector and multiply matrix A from left. However, if x ∈ 𝔽^m, then you have no choice as to consider this vector as a row and multiply matrix A from right.

You can always convert column vectors into rows and vice versa by using transposition operation (see below) because

    Example 2: Matrix times a vector     Example 3: Nutrition problem     Example 4: Comparison of matrix multiplications from right and left

 

Before learning how (standard) matrix multiplication works, the first thing you need to know about this topic is that the matrix product exists only for compatible matrices: when matrix A is multiplied by matrix B from right, their dimensions must match. In other words the number of columns in matrix A should be equal to the number of rows in matrix B in order for the product A B to exist. Therefore, the existence of product A B does not guarantee that reverse product B A exists; but when both products hold true, it is commutative only in rare cases and generally speaking A B ≠ B A.

The following five statements provide useful terminology for saying that product A B is valid for matrices A and B:

• A times B (which is denoted by A B),
• A pre-multiplies B,
• A left-multiplies B,
• B right-multiplies A
• B post-multiplies A.

There are four ways to implement matrix multiplication of two matrices that provide insights into matrix computations in different contexts and for different problems. We denote by C [c_i,j] the product of two matrices where C_p×q means a matrix of dimensions p × q (similar description is valid for matrices A and B).

(I) The "dot-product" prospective

Each element c_i,j in product C = A B is the dot product between the i-th row in A and the j-th column in B:     Example 5: 3 × 2 Matrix times 2 × 2 matrix

(II) The "outer product" prospective

Recall that the outer product (also known as tensor product) of two vectors x ∈ 𝔽^m×1 and y ∈ 𝔽^1×n is their matrix product whose entries are all products of an element in the first vector with an element from the second vector: For complex vectors, it is customary to use the conjugate transpose of one of the vectors. However, we do not follow this rule because we just use tensor notation for shortcut definition of matrix multiplications. Thus, the entries of m × n matrix x ⊗ y can be written as

An m × p matrix A can be considered as an array of column vectors

where every column vector has size m. We represent another multiplier B as an p column vector having rows of matrix B as entries:

Every column (having size m) of matrix A is outer multiplied by every row (having size n) of matrix B; this result in a matrix of dimensions m × n,

As such our product can be considered as a layer. Then combining all layers, we obtain the result.     Example 6: 3 × 2 Matrix times 2 × 2 matrix

(III) The "row" prospective

Each row in the product matrix is the weighted sum of all rows in the right matrix, where the weights are given by the elements in each row of the left matrix. See the following example for illustration.     Example 7: 3 × 2 Matrix times 2 × 2 matrix

(IV) The "column-wise" prospective

First, we represent n × p matrix B as a list (array) of p columns:

where c_j(B) is the j-th column of matrix B. Since each column has size n, we can apply to it m × n matrix A from left and obtain In other words, every j-th column of the product (A B) is     Example 8: Column-wise product

From the column perspective, all matrices (the multiplying matrices and the product matrix) are thought of as sets of column vectors. Then the product matrix is created one column at a time.

The first column in the product matrix is a linear combination of all columns in the left matrix, where the weights are defined by the elements in the first column of the right matrix. The second column in the product matrix is again a weighted combination of all columns in the left matrix, except that the weights now come from the second column in the right matrix. And so on for all n columns in the right matrix. The following example demonstrates this approach.     Example 9: 3 × 2 Matrix times 2 × 2 matrix

Before we go to advanced applications, we consider a simple, but very important example of multiplication by diagonal matrix:

and A diagonal matrix having diagonal entries λ₁, λ₂, … , λ_n (as in the preceding equality) may safely be abbreviated by diag (λ₁, λ₂, … , λ_n) because such matrix is isomorphic to a vector. Mathematica has a dedicated command: We will widely use the main property of the identity matrix: it acts as multiplicative unity (either from left or from right) on any rectangular matrix. Multiplication by the identity matrix does not change the output:    I_mA = A I_n = A ∈ 𝔽^m×n.     Example 10: Product is not commutative Proof of Theorem 1:     This theorem tells us that in finite sequence of matrix products you can distribute parenthesis as you want, so the order in which individual products are evaluated does not matter: However, as final answer of a sequence of matrix products does not depend on the order in which pairwise products are executed, this order may substantially effect the time of matrix product evaluation (see Dobrushkin's book).

Other properties of Matrix Multiplication are formulated in the following theorem. Proof of Theorem 2:

The concept of matrix multiplication is due to the German mathematician Gotthold Eisenstein (1823--1852) of Jewish descent. He introduced the idea of matrix multiplication around 1844 (this year, he published alone 23 papers) to simplify the process of making substitutions in linear systems. He suffered various health problems throughout his life, including meningitis as an infant, a disease that took the lives of all five of his brothers and sisters.

Gotthold's idea was further expanded on and formalized by the British mathematician Arthur Cayley (1821--1895) in his Memoir on the Theory of Matrices that was published in 1858. Eisenstein was a pupil of Carl Friedrich Gauss who ranked him as the equal of Issac Newton and Archimedes.