Recall that we denote by 𝔽^m×n the vector space of all m-byn matrices with entries from field 𝔽, which is either ℤ, the set of integers, or ℚ, the set of rational numbers, or ℝ, the set of real numbers, or ℂ, the set of complex numbers.

Manipulations with vectors

Usually, an n-tuple (x₁, x₂, … , x_n), written in parenthesis and a row vector [x₁, x₂, … , x_n], written in brackets, look as the same object to human eyes. One of the pedagogical virtues of any software package is its requirement to pay close attention to the types of objects used in specific contexts. In particular, the system Python treats these two versions of a vector differently because

If we work only with vectors as one dimension arrays, numpy is good for their representations. The operations of addition of vectors and multiplication by scalars work as expected. When we want to work with matrix multiplication, it's better to define the dimension of the vectors explicitly. Below we will show how vectors works in numpy first. Then we introduce matrix and matrix opearitions in numpy.

Comment: break code into three parts: (1) define n-tuple ∈ 𝔽ⁿ as x = (1, 2, 3)

(3) define column vector z = [100, 200, 300]

append their dimensions!!

When we work with matrices, it is better to define the dimension of the row vector and column vector explicitly. Otherwise, we will get unexpected values. Matrix multiplication use the @ peration.

Row vector multiply column vector as inner product calculation. Column vector multiply row vector as outter procuct calculatin.

Transpose and conjugate transpose in Python package numpy

To extract partitioned matrix, we use row and column index. Pay attention that matrix in numpy starts row and column index from zero, not one. Below we show how to extract row, column and block from matrix defined in numpy. Please note that the ending index after : is not included. If there is no index number before or after :, it means from the start or to the end respectively.

Partition Matrices

A block matrix or a partitioned matrix is a matrix that is interpreted as having been broken into sections called blocks or submatrices.

  Partitoned matrices appear in most modern applications of linear algebra because the notation highlights essential structures of matrices. Partititoning a matrix is a generalization to used previously a list of columns or rows. Intuitively, a matrix interpreted as a block matrix can be visualized as the original matrix with a collection of horizontal and vertical lines, which break it up, or partition it, into a collection of smaller matrices. Especially when dimensions of a matrix are large, it may be beneficial to view a matrix as combined from smaller submatrices. If we simultaneously partition adjacent rows and adjacent columns of a matrix into groups, this partitions the matrix into submatrices or blocks, resulting in a representation of the matrix as a partitioned or block matrix.

  It is possible to use a block partitioned matrix product that involves only algebra on submatrices of the factors. The partitioning of the factors is not arbitrary, however, and requires "conformable partitions" between two matrices A and B such that all submatrix products that will be used are defined in usual row-column rule.

M = {{ 1, -2, 3, -1, 4},{3, 1, -2, 4, -2},{5, 4, -3, 1, 1},{2, -3, 4, 2, -3}}
B = {{6}, {-3}, {1}, {4}, {-1}}
M.B

The Schur complement is named after Issai Schur (1875--1941), who introduced it in 1917 (I. Schur, Potenzreihen im Innern des Einheitskreises, J. Reine Angew. Math., 147, 1917, 205–232). The USA mathematician Emilie Virginia Haynsworth (1916--1985) was the first in 1968 paper to call it the Schur complement. The Schur complement is a key tool in the fields of numerical analysis, statistics and matrix analysis.

  Issai Schur was a Russian mathematician (he was born in Mogilev, now Belarus) who worked in Germany for most of his life. Issai spoke German without a trace of an accent, and nobody even guessed that it was not his first language. He obtained his doctorate in 1901, became lecturer in 1903 and, after a stay at the University of Bonn, professor in 1919. As a student of Ferdinand Georg Frobenius (1849--1917), he worked on group representations. He is perhaps best known today for his result on the existence of the Schur decomposition, which we will discuss later.

  In 1922 Schur was elected to the Prussian Academy, proposed by Planck, the secretary of the Academy. From 1933 events in Germany made Schur's life increasingly difficult. Schur considered himself as a German, not a Jew, but the Nazis had different opinion. Later in 1935 Schur was dismissed from his chair in Berlin but he continued to work there suffering great hardship and difficulties. Schur left Germany for Palestine in 1939, broken in mind and body, having the final humiliation of being forced to find a sponsor to pay the 'Reichs flight tax' to allow him to leave Germany. Without sufficient funds to live in Palestine he was forced to sell his beloved academic books to the Institute for Advanced Study in Princeton.

Example 3: Schur complements.

Example 3: Consider the matrix

\[ {\bf M} = \begin{bmatrix} 1&-2&3&-1&4 \\ 3&1&-2&4&-2 \\ 5&4&-3&1&1 \\ 2&-3&4&2&-3 \end{bmatrix} = \left[ \begin{array}{cc|ccc} 1&-2&3&-1&4 \\ 3&1&-2&4&-2 \\ \hline 5&4&-3&1&1 \\ 2&-3&4&2&-3 \\ 0&1&3&-1&2 \end{array} \right] = \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} , \]

where

\[ {\bf A} = \begin{bmatrix} 1&-2 \\ 3&1 \end{bmatrix} , \quad {\bf B} = \begin{bmatrix} 3&-1&4 \\ -2&4&2 \end{bmatrix} , \quad {\bf C} = \begin{bmatrix} 5&4 \\ 2&-3 \\ 0&1 \end{bmatrix} , \quad {\bf D} = \begin{bmatrix} -3&1&1 \\ 4&2& -3 \\ 3&-1&2 \end{bmatrix} . \]

Since both square matrices A and D are not singular (\( \det{\bf A} =7 , \quad \det{\bf D} = -30 \) ), we can evaluate both Schur complements:

\begin{align*} {\bf M}/{\bf A} &= {\bf D} - {\bf C}\,{\bf A}^{-1} {\bf B} = \frac{1}{7} \begin{bmatrix} 28&-56&7 \\ -3&21&-67 \\ 32&-14&24 \end{bmatrix} , \\ {\bf M}/{\bf D} &= {\bf A} - {\bf B}\,{\bf D}^{-1} {\bf C} = - \frac{1}{3} \begin{bmatrix} 7&19 \\ 42 & 27 \end{bmatrix} . \end{align*}

We check calculations with Python:

M = {{1, -2, 3, -1, 4}, {3, 1, -2, 4, -2}, {5, 4, -3, 1, 1}, {2, -3,
   4, 2, -3}, {0, 1, 3, -1, 2}}
DD = {{-3, 1, 1}, {4, 2, -3}, {3, -1, 2}}
A = {{1, -2}, {3, 1}}
B = {{3, -1, 4}, {-2, 4, 2}}
CC = {{5, 4}, {2, -3}, {0, 1}}

MA = (DD - CC.Inverse[A].B)*7 

Out[6]= {{28, -56, 7}, {-3, 21, -67}, {32, -14, 24}}

MD = (A - B.Inverse[DD].CC)*3

Out[7]= {{-7, -19}, {-42, -27}}

 

Block Matrix Determinant

For a block matrix \( {\bf M} = \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} \) with square matrices A and D, we have

\[ \det \left( {\bf M} \right) = \det \left( {\bf A} \right) \left( {\bf M} / {\bf A} \right) \qquad\mbox{and} \qquad \det \left( {\bf M} \right) = \det \left( {\bf D} \right) \left( {\bf M} / {\bf D} \right) , \]

provided that inverse matrices exist. The latter follows from the equation

\[ \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} \begin{bmatrix} {\bf I} & {\bf 0} \\ - {\bf D}^{-1} {\bf C} & {\bf I} \end{bmatrix} = \begin{bmatrix} {\bf A} - {\bf B}\, {\bf D}^{-1} {\bf C} & {\bf B} \\ {\bf 0} & {\bf D} \end{bmatrix} , \]

where I is the identity matrix (taken to be of the appropriate dimension), and square matrix D is assumed to be nonsingular. Similar formula is valid for invertible matrix A. The following block diagonalization forms learly display the Schur complement role. If A is nonsingular,

\begin{align*} &\begin{bmatrix} {\bf I} & {\bf 0} \\ -{\bf C}\, {\bf A}^{-1} & {\bf I} \end{bmatrix} \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} \begin{bmatrix} {\bf I} & - {\bf A}^{-1} {\bf B} \\ {\bf 0} & {\bf I} \end{bmatrix} = \begin{bmatrix} {\bf A} & {\bf 0} \\ {\bf 0} & {\bf M} / {\bf A} \end{bmatrix} , \\ &\begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} = \begin{bmatrix} {\bf I} & {\bf 0} \\ {\bf C}\, {\bf A}^{-1} & {\bf I} \end{bmatrix} \begin{bmatrix} {\bf A} & {\bf 0} \\ {\bf 0} & {\bf M} / {\bf A} \end{bmatrix} \begin{bmatrix} {\bf I} & {\bf A}^{-1} {\bf B} \\ {\bf 0} & {\bf I} \end{bmatrix} . \end{align*}

If D is nonsingular, we have

\begin{align*} &\begin{bmatrix} {\bf I} & - {\bf B}\,{\bf D}^{-1} \\ {\bf 0} & {\bf I} \end{bmatrix} \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} \begin{bmatrix} {\bf I} & {\bf 0} \\ -{\bf D}^{-1} {\bf C} & {\bf I} \end{bmatrix} = \begin{bmatrix} {\bf M} / {\bf D} & {\bf 0} \\ {\bf 0} & {\bf D} \end{bmatrix} , \\ &\begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} = \begin{bmatrix} {\bf I} & {\bf B}\,{\bf D}^{-1} \\ {\bf 0} & {\bf I} \end{bmatrix} \begin{bmatrix} {\bf M} / {\bf D} & {\bf 0} \\ {\bf 0} & {\bf D} \end{bmatrix} \begin{bmatrix} {\bf I} & {\bf 0} \\ {\bf D}^{-1} {\bf C} & {\bf I} \end{bmatrix} . \end{align*}

All of these equations can be directly verified by matrix multiplication. Example 4: Determinant of partitioned 5-by-5 matrix.

Example 4: Consider four block matrices:

\[ {\bf A}_1 = \begin{bmatrix} 3&1 \\ 5&2 \end{bmatrix} , \quad {\bf A}_2 = \begin{bmatrix} 1&2&3 \\ 2&3&4 \end{bmatrix} , \quad {\bf A}_3 = \begin{bmatrix} -1&1 \\ 2&3 \\ 3&4 \end{bmatrix} , \quad {\bf A}_4 = \begin{bmatrix} 2&1&-1 \\ 1&1&1 \\ 2&3&4 \end{bmatrix} . \]

With Mathematica, we build 5-by-5 matrix from these blockes:

A1 = {{3, 1}, {5, 2}}
A2 = {{1, 2, 3}, {2, 3, 4}}
A3 = {{-1, 1}, {2, 3}, {3, 4}}
A4 = {{2, 1, -1}, {1, 1, 1}, {2, 3, 4}}
M = ArrayFlatten[{{A1, A2}, {A3, A4}}]

\[ {\bf M} = \begin{bmatrix} {\bf A}_1&{\bf A}_2 \\ {\bf A}_3&{\bf A}_4 \end{bmatrix} = \begin{bmatrix} 3&1&1&2&3 \\ 5&2&2&3&4 \\ -1&1&2&1&-1 \\ 2&3&1&1&1 \\ 3&4&2&3&4 \end{bmatrix} . \]

Then we calculate Schur complements

\[ {\bf M}/{\bf A} = {\bf A}_4 - {\bf A}_3 {\bf A}_1^{-1} {\bf A}_2 = \begin{bmatrix} 1&3&4 \\ -2&2&6 \\ -2&4&10 \end{bmatrix} , \qquad {\bf M}/{\bf D} = {\bf A}_1 - {\bf A}_2 {\bf A}_4^{-1} {\bf A}_3 = \begin{bmatrix} 2&0 \\ 2&-2 \end{bmatrix} . \]

MA = A4 - A3.Inverse[A1].A2

Out[6]= {{1, 3, 4}, {-2, 2, 6}, {-2, 4, 10}}

Det[MA]

Out[7]= 4

MD = A1 - A2.Inverse[A4].A3

Out[8]= {{2, 0}, {2, -2}}

Det[MD]

Out[9]= -4

Therefore,

\[ \det{\bf M} = \det {\bf A}_1 \det \left( {\bf M}/{\bf A} \right) = \det {\bf A}_4 \det \left( {\bf M}/{\bf D} \right) =4 . \]

■

 

Block Matrix Inversion

To find the inverse matrix to a block matrix \( {\bf M} = \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} , \) we have

\begin{align*} {\bf M}^{-1} &= \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix}^{-1} = \begin{bmatrix} {\bf A}^{-1} + {\bf A}^{-1} {\bf B} \left( {\bf M}/{\bf A} \right)^{-1} {\bf C} \,{\bf A}^{-1} & - {\bf A}^{-1} {\bf B}\left( {\bf M}/{\bf A} \right)^{-1} \\ - \left( {\bf M}/{\bf A} \right)^{-1} {\bf C} \,{\bf A}^{-1} & \left( {\bf M}/{\bf A} \right)^{-1} \end{bmatrix} , \\ \\ {\bf M}^{-1} &= \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix}^{-1} = \begin{bmatrix} \left( {\bf M}/{\bf D} \right)^{-1} & - \left( {\bf M}/{\bf D} \right)^{-1} {\bf B} \,{\bf D}^{-1} \\ - {\bf D}^{-1} {\bf C} \left( {\bf M}/{\bf D} \right)^{-1} & {\bf D}^{-1} + {\bf D}^{-1} {\bf C} \left( {\bf M}/{\bf D} \right)^{-1} {\bf B} \, {\bf D}^{-1} \end{bmatrix} . \end{align*}

When m = n so that A, B, C, and D are all square, and all of them are nonsingular, one obtains the elegant formula

\[ {\bf M}^{-1} = \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix}^{-1} = \begin{bmatrix} \left( {\bf M}/{\bf D} \right)^{-1} & \left( {\bf M}/{\bf B} \right)^{-1} \\ \left( {\bf M}/{\bf C} \right)^{-1} & \left( {\bf M}/{\bf A} \right)^{-1} \end{bmatrix} . \]

Example: Inverse of partitioned matrix.

Example: Let us consider a 5-by-5 matrix that is partitioned into four blocks:

\[ {\bf M} = \begin{bmatrix} {\bf A} & {\bf B} \\ {\bf C} & {\bf D} \end{bmatrix} , \]

where

\[ {\bf A} = \begin{bmatrix} 3&5 \\ 1&2 \end{bmatrix} , \quad {\bf B} = \begin{bmatrix} 2 &-1&3 \\ -1&1&2 \end{bmatrix} , \quad {\bf C} = \begin{bmatrix} -3&2 \\ 1&-3 \\ 2&1 \end{bmatrix} , \quad {\bf D} = \begin{bmatrix} 2&4&-6 & \\ 3&-11&18 \\ -2&-8&13 \end{bmatrix} . \]

Next we calculate Schur's complements:

\begin{align*} {\bf M}/{\bf A} &= {\bf D} - {\bf C}\,{\bf A}^{-1} {\bf B} = \begin{bmatrix} 39&-25&-24 \\ -27&8&31 \\ -15&2&18\end{bmatrix} , \\ {\bf M}/{\bf D} &= {\bf A} - {\bf B}\,{\bf D}^{-1} {\bf C} = \begin{bmatrix} 19/2 & 3 \\ 37& 69 \end{bmatrix} . \end{align*}

We check calculations with Mathematica:

A = {{3, 5}, {1, 2}}
B = {{2, -1, 3}, {-1, 1, 2}}
CC = {{-3, 2}, {1, -3}, {2, 1}}
DD= {{2, 4, -6}, {-3, -11, 18}, {-2, -8, 13}}
MA = DD - CC.Inverse[A].B

Out[5]= {{39, -25, -24}, {-27, 8, 31}, {-15, 2, 18}}

MD = A - B.Inverse[DD].CC

Out[6]= {{19/2, 3}, {37, 69}}

Now we calculate components in the inverse matrix (of course, using Mathematica):

Inverse[MD]

Out[7]= {{46/363, -(2/363)}, {-(74/1089), 19/1089}}

Inverse[MA]

Out[8]= {{82/1089, 134/363, -(53/99)}, {7/363, 38/121, -(17/33)}, {2/33, 3/11, -(1/3)}}

-Inverse[MD].B.Inverse[DD]

Out[9]= {{-(109/363), -(4/121), -(4/33)}, {128/1089, -(83/363), 38/99}}

-Inverse[DD].CC.Inverse[MD]
Inverse[DD] + Inverse[DD].CC.Inverse[MD].B.Inverse[DD]
Inverse[A] + Inverse[A].B.Inverse[MA].CC.Inverse[A]
-Inverse[MA].CC.Inverse[A]
-Inverse[A].B.Inverse[MA]

Out[10]= {{395/1089, -(175/1089)}, {47/363, 140/363}, {4/33, 7/33}}
Out[11]= {{82/1089, 134/363, -(53/99)}, {7/363, 38/121, -(17/33)}, {2/33, 3/11, -(1/3)}}
Out[12]= {{46/363, -(2/363)}, {-(74/1089), 19/1089}}
Out[13]= {{395/1089, -(175/1089)}, {47/363, 140/363}, {4/33, 7/33}}
Out[14]= {{-(109/363), -(4/121), -(4/33)}, {128/1089, -(83/363), 38/99}}

Finally, we check the answer:

M = {{3, 5, 2, -1, 3}, {1, 2, -1, 1, 2}, {-3, 2, 2, 4, -6}, {1, -3, 3, -11, 18}, {2, 1, -2, -8, 13}}
Inverse[M]*3501 

Out[16]= {{522, -54, -1035, -36, -540}, {304, -179, 524, -249, 544}, {395, -175, 82, 402, -583}, {-291, 1680, -87, 342, -705}, {-222, 1029, 78, 297, -213}}

Therefore,

\[ {\bf M}^{-1} = \frac{1}{3501} \begin{bmatrix} 522&-54&-1035&-36&540 \\ 304&-179&524&-249&544 \\ 395&-175&82&402&-583 \\ -291&1680& -87&342&-705 \\ -222&1029&78&297&-213 \end{bmatrix} . \qquad ■ \]

Block Diagonal Matrices

A block diagonal matrix is a block matrix that is a square matrix, and having main diagonal blocks square matrices, such that the off-diagonal blocks are zero matrices. A block diagonal matrix M has the form

\[ {\bf M} = \begin{bmatrix} {\bf A}_1 & {\bf 0} & \cdots & {\bf 0} \\ {\bf 0} & {\bf A}_2 & \cdots & {\bf 0} \\ \vdots & \vdots& \ddots & \vdots \\ {\bf 0} & {\bf 0} & \cdots & {\bf A}_n \end{bmatrix} , \]

where A_k is a square matrix; in other words, it is the direct sum of ₁, ... , A_n: \( {\bf M} = {\bf A}_1 \oplus {\bf A}_2 \oplus \cdots \oplus {\bf A}_n . \) It can also be indicated as diag(\( {\bf A}_1 , {\bf A}_2 , \ldots , {\bf A}_n \) ). Any square matrix can trivially be considered a block diagonal matrix with only one block.

 

For the determinant and trace, the following properties hold

\begin{align*} \det \left( {\bf M} \right) &= \det {\bf A}_1 \times \det {\bf A}_2 \times \cdots \times \det {\bf A}_n , \\ \mbox{tr}\left( {\bf M} \right) &= \mbox{tr} \left( {\bf A}_1 \right) + \mbox{tr} \left( {\bf A}_2 \right) + \cdots + \mbox{tr} \left( {\bf A}_n \right) . \end{align*}

The inverse of a block diagonal matrix is another block diagonal matrix, composed of the inverse of each block, as follows:

\[ {\bf M}^{-1} = \begin{bmatrix} {\bf A}_1 & {\bf 0} & \cdots & {\bf 0} \\ {\bf 0} & {\bf A}_2 & \cdots & {\bf 0} \\ \vdots & \vdots& \ddots & \vdots \\ {\bf 0} & {\bf 0} & \cdots & {\bf A}_n \end{bmatrix}^{-1} = \begin{bmatrix} {\bf A}_1^{-1} & {\bf 0} & \cdots & {\bf 0} \\ {\bf 0} & {\bf A}_2^{-1} & \cdots & {\bf 0} \\ \vdots & \vdots& \ddots & \vdots \\ {\bf 0} & {\bf 0} & \cdots & {\bf A}_n^{-1} \end{bmatrix} . \]

Example: Block diagonal matrix. n

Example: Suppose we are given two square matrices:

\[ {\bf A}_1 = \begin{bmatrix} 3&1 \\ 5&2 \end{bmatrix} , \quad {\bf A}_4 = \begin{bmatrix} 2&1&-1 \\ 1&1&1 \\ 2&3&4 \end{bmatrix} . \]

With Mathematica, we build 5-by-5 block diagonal matrix from these two blockes:

A1 = {{3, 1}, {5, 2}}
A4 = {{2, 1, -1}, {1, 1, 1}, {2, 3, 4}}
zero23 = ConstantArray[0, {2, 3}]
zero32 = ConstantArray[0, {3, 2}]
A = ArrayFlatten[{{A1, zero23}, {zero32, A4}}]

\[ {\bf A} = \begin{bmatrix} {\bf A}_1&{\bf 0} \\ {\bf 0}&{\bf A}_4 \end{bmatrix} = \begin{bmatrix} 3&1&0&0&0 \\ 5&2&0&0&0 \\ 0&0&2&1&-1 \\ 0&0&1&1&1 \\ 0&0&2&3&4 \end{bmatrix} . \]

Then we calculate its determinant and determinants of corresponding blocks:

 
Det[A]
 

Out[1]= -1

 
Det[A1]
 

Out[1]= -1

Det[A4]

Out[1]= 1

which is in accordance with the formula \( \det{\bf A} = \det {\bf A}_1 \det {\bf A}_2 . \) Finally, we calculate its inverse:
Inverse[A]

Out[4]= {{2, -1, 0, 0, 0}, {-5, 3, 0, 0, 0}, {0, 0, -1, 7, -2}, {0, 0, 2, -10, 3}, {0, 0, -1, 4, -1}}

Inverse[A1]

Out[5]= {{2, -1}, {-5, 3}}

Inverse[A4]

Out[6]= {{-1, 7, -2}, {2, -10, 3}, {-1, 4, -1}}

Therefore,

\[ {\bf A}^{-1} = \begin{bmatrix} 3&1&0&0&0 \\ 5&2&0&0&0 \\ 0&0&2&1&-1 \\ 0&0&1&1&1 \\ 0&0&2&3&4 \end{bmatrix}^{-1} = \begin{bmatrix} 2&-1&0&0&0 \\ -5&3&0&0&0 \\ 0&0&-1&7&-2 \\ 0&0&2&-10&3 \\ 0&0&-1&4&-1 \end{bmatrix} . \]

■

 

Block Tridiagonal Matrices

A block tridiagonal matrix is another special block matrix, which is just like the block diagonal matrix a square matrix, having square matrices (blocks) in the lower diagonal, main diagonal and upper diagonal, with all other blocks being zero matrices. It is essentially a tridiagonal matrix but has submatrices in places of scalars. A block tridiagonal matrix M has the form

\[ {\bf M} = \begin{bmatrix} {\bf B}_1 & {\bf C}_1 & {\bf 0} &&& \cdots & {\bf 0} \\ {\bf A}_2 & {\bf B}_2 & {\bf C}_2 &&& \cdots & {\bf 0} \\ \vdots& \ddots & \ddots & \ddots &&& \vdots \\ && {\bf A}_k & {\bf B}_k & {\bf C}_k & & {\bf 0} \\ &&& \ddots & \ddots & \ddots && \\ &&&&{\bf A}_{n-1}&{\bf B}_{n-1} & {\bf C}_{n-1} \\ {\bf 0}&&\cdots&&&{\bf A}_n&{\bf B}_n \end{bmatrix} , \]

where A_k, B_k and C_k are square sub-matrices of the lower, main and upper diagonal respectively.

Direct Sum

For any arbitrary matrices \( {\bf A} = [a_{i,j} ] \) (of size \( n \times m \) ) and \( {\bf B} = [b_{i,j} \) (of size \( p \times q \) ), we have the direct sum of A and B, denoted by \( {\bf A} \oplus {\bf B} \) and defined as

\[ {\bf A} \oplus {\bf B} = \begin{bmatrix} a_{11} & \cdots & a_{1m} & 0 & \cdots & 0 \\ \vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\ a_{n,1} & \cdots & a_{nm} & 0 & \cdots & 0 \\ 0& \cdots & 0 & b_{11} & \cdots & b_{1q} \\ \vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\ 0& \cdots & 0 & b_{p1} & \cdots & b_{pq} \end{bmatrix} . \]

Direct Product

Let A be n-by-m matrix and B be p-by-q matrix, then their Kronecker product or direct product, denoted by \( \otimes ,\) is the \( pn \times mq \) block matrix:

\[ {\bf A} \otimes {\bf B} = \begin{bmatrix} a_{11} {\bf B} & \cdots & a_{1m} {\bf B} \\ \vdots& \ddots & \vdots \\ a_{n1} {\bf B} & \cdots & a_{nm} {\bf B} \end{bmatrix} . \]

More explicitly:

\[ {\bf A} \otimes {\bf B} = \begin{bmatrix} a_{11} b_{11} & a_{11} b_{12} & \cdots & a_{1m} b_{1q} & \cdots & \cdots & a_{1m} b_{11} & a_{1n} b_{12} & \cdots & a_{1m} b_{1q} \\ a_{11} b_{21} & a_{11} b_{21} & \cdots & a_{11} b_{2q} & \cdots & \cdots & a_{1m} b_{21} & a_{1m} b_{22} & \cdots & a_{1m} b_{2q} \\ \vdots & \vdots & \ddots & \vdots & && \vdots & \vdots & \ddots & \vdots \\ a_{11} b_{p1} & a_{11} b_{p2} & \cdots & a_{11} b_{pq} & \cdots & \cdots & a_{1m} b_{p1} & a_{1m} b_{p2} & \cdots & a_{1m} b_{pq} \\ \vdots & \vdots & & \vdots & \ddots && \vdots & \vdots & & \vdots \\ \vdots & \vdots & & \vdots & &\ddots & \vdots & \vdots & & \vdots \\ a_{n1} b_{11} & a_{n1} b_{12} & \cdots & a_{n1} b_{1q} & \cdots & \cdots & a_{nm} b_{11} & a_{nm} b_{12} & \cdots & a_{nm} b_{1q} \\ a_{n1} b_{21} & a_{n1} b_{22} & \cdots & a_{n1} b_{2q} & \cdots & \cdots & a_{nm} b_{21} & a_{nm} b_{22} & \cdots & a_{nm} b_{2q} \\ \vdots & \vdots & \ddots & \vdots & && \vdots & \vdots & \ddots & \vdots \\ a_{n1} b_{p1} & a_{n1} b_{p2} & \cdots & a_{n1} b_{pq} & \cdots & \cdots & a_{nm} b_{p1} & a_{nm} b_{p2} & \cdots & a_{nm} b_{pq} \end{bmatrix} . \]

The Kronecker product is named after the German mathematician Leopold Kronecker (1823--1891), even though there is little evidence that he was the first to define and use it. Indeed, the Kronecker product should be called the Zehfuss product because Johann Georg Zehfuss published a paper in 1858 (Zeit. fur Math und Physik, 3, 1858, 298--301), in which he described the matrix operation we now know as the Kronecker product.