Lecture Notes

5. Interpretations of Linear Maps

Part of the Series on Linear Algebra.

By Akshay Agrawal. Last updated Sept. 19, 2017.

Previous entry: Linear Maps and Matrices; Next entry: Null Spaces and Ranges

Linear maps and matrices are linear algebra’s principal actors, so it is imperative for the reader to be comfortable with them. In this section, we provide a variety of intuitions for and interpretations of these two actors.

5.1 Linear Maps and Matrices are Two Sides of the Same Coin

This was stated in the previous section, but it bears repeating:

Every linear map can be encoded with a matrix, and every matrix-vector multiplication can be interpreted as applying a linear map to a vector.

The reduction from linear maps to matrices is made possible by the fact that linear maps are completely determined by their action on bases. The number of columns in the matrix of a linear map is equal to the dimension of the domain, which in turn is equal to the number of vectors in any basis for the domain.

As an example, let $T: \Reals^{2} \rightarrow \Reals^{2}$ be defined such that $T(c_1, c_2) = (2 c_1, 3 c_2)$. Then the matrix $A$ for $T$ with respect to the standard basis is

$$\begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$$

because $T(1, 0) = (2, 0)$ and $T(0, 1) = (0, 3)$. So in particular the product

$$A \begin{bmatrix}c_1 \\ c_2 \end{bmatrix}$$

can be intepreted as $A(c_1e_1 + c_2e_2) = c_1 A e_1 + c_2 A e_2 = c_1 \rowvec{2 & 0} + c_2 \rowvec{0 & 3},$ ie, as the sum of $A$ applied to the standard basis vectors $e_1 = \rowvec{1 & 0}$ and $e_2 = \rowvec{0 & 1}$, each of which is scaled by its corresponding coordinate.

5.2 Linear Maps Preserve Lines

Every linear map preserves lines — this provides the intuition for why linear maps are named as such. In particular, consider the line $f(\lambda) = v + \lambda \cdot d$, where $\lambda$ is a scalar $d$ is a direction vector, and $v$ a vector in $V$; let $T : V \rightarrow W$ be a linear map. Applying $T$ to $f$ results in another line precisely because linear maps are additive and homogeneous:

$$\begin{align*} T(f(\lambda)) &= T(v + \lambda d) \\ &= T(v) + T(\lambda d) \\ &= T(v) + \lambda T(d). \end{align*}$$

5.3 Linear Maps as Geometric Transformations

In this subsection, we present a few important examples of geometric transformations — dilations, rotations, and reflections — that are linear maps. Because compositions of linear maps are themselves linear maps, the transformations presented here may be thought of as building blocks for assembling more complicated maps. In fact,

every linear map can be decomposed into the composition of dilations, rotations, and reflections.

We will encounter this deep result later, when studying the singular value decomposition.

(Unless otherwise stated, all matrices in this section are with respect to the standard bases.)

Dilations

Any function that operates on vectors by stretching or shrinking vectors (that is, by dilating them) is a linear map whose matrix is given by

$$\begin{bmatrix} \lambda_1 & 0 & \ldots & 0 \\ 0 & \lambda_2 & \ldots & 0 \\ \vdots & & \ddots & \vdots \\ 0 & 0 & \ldots & \lambda_n \end{bmatrix}$$

for some $\lambda_i$ are positive scalars.

Rotations

Any function that rotates vectors by some angle $\theta$ counter-clockwise about the origin is a linear map. In two dimensions, a rotation is given by the matrix

$$\begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}.$$

This rotation matrix is derived by considering where the standard basis vectors land on the unit circle after passing through a rotation of $\theta$ radians.

Reflections

Any function that reflects vectors across a line passing through the origin is a linear map. For example, below is the matrix for a reflection across the $x$-axis:

$$\begin{bmatrix} -1 & 0 \\ 0 & 1 \end{bmatrix}.$$

The simplest way to represent reflections across arbitrary lines through the origin is to change coordinates to a basis that makes the reflection natural; we’ll return to this idea in a subsequent section.

5.4 Special Instances of Matrix Multiplication

Here we present some special instances of matrix multiplications. In order to represent these products, it will be convenient to introduce the transpose of a matrix. The transpose of a matrix $A$ is denoted $A^T$ and is defined such that $A^T_{ij} = A_{ji}$. It is a useful fact that $(AB)^T = B^TA^T$.

Dot product

The matrix product of a row vector by a column vector is called a dot product. If $a$ and $b$ are vectors from the same $n$-dimensional vector space, then

$$a^Tb = \sum_{i=1}^{n} a_ib_i.$$

Outer product

The matrix product of a column vector by a row vector is called an outer product. If $a$ and $b$ are vectors from the same $n$-dimensional vector space then

$$ab^T = \begin{bmatrix} a_1b_1 & a_1b_2 & \ldots & a_1b_n \\ a_2b_1 & a_2b_2 & \ldots & a_2b_n \\ \vdots & \cdots & \ddots & \vdots \\ a_nb_1 & a_nb_2 & \ldots & a_nb_n \end{bmatrix},$$

as you can verify yourself using the definition of matrix multiplication.

Example: Gram matrix

The Gram matrix of a set of vectors $a_1, a_2, \ldots a_n$ is the $n\times n$ matrix $G$ such that $G_{ij} = a_i^Ta_j$. You can check for yourself that if $A$ is the matrix whose $i$-th column is $a_i$, then $G = A^TA$. Moreover, letting $B = A^T$, it is also true that $A^TA = BB^T = \sum_{i=1}^{n} b_ib_i^T$, where $b_1, b_2, \ldots, b_n$ are the columns of $B$ (and hence the rows of $A$).

5.5 Different Views on Matrix Multiplication

There are at least two useful ways to interpret matrix-vector and matrix-matrix multiplication: from the perspective of the columns of the matrix on the left-hand side and from the perspective of its rows. In the following, let $A$ be an $m \times n$ matrix and $B$ an $n \times p$ matrix, and let $a_i$ be the columns of $A$, $\tilde{a}_i$ the rows of $A$ (represented as column vectors), $b_i$ the columns of $B$, and $\tilde{b_i}$ the rows of $B$.

Linear combinations of column vectors

As defined in the previous section, the matrix-vector product $Ab$, $b = \rowvec{c_1 & c_2 & \ldots c_n}$, is given by forming a linear combination of the columns of $A$ with the coordinates of $b$ as the coefficients:

$$Ab = \sum_{i=1}^{n} c_ia_i.$$

An engineering interpretation of this view is that the columns of $A$ are actuators that represent the movement of $e_i$ to $a_i$, and $Ab$ represents the result of applying the actuators to each of $c_ie_i$.

The matrix product $AB$ can be calculated by horizontally concatenating the vectors $Ab_1, Ab_2, \ldots, Ab_p$:

$$AB = \begin{bmatrix} Ab_1 & Ab_2 & \cdots & Ab_p. \end{bmatrix}$$

Linear combinations of row vectors

The vector-matrix product $a^T B$, where $a^T$ is a row vector, can be cast as a linear combination of the rows $b_i^T$ of $B$ with the coordinates of $a$ as coefficients:

$$a^T B = \sum_{i=1}^{n} c_i b_i^T.$$

The matrix product $AB$ can be calculated by vertically concatenating the vectors $a_1^T B, a_2^T B, \ldots, a_n^T B$:

$$AB = \begin{bmatrix} a_1^TB \\ a_2^TB \\ \vdots \\ a_n^TB \end{bmatrix}.$$

Dot products with row vectors

The matrix-vector product $Ab$ can also be cast in terms of the rows of $A$:

$$Ab = \begin{bmatrix} \tilde{a}_1^T b \\ \tilde{a}_2^T b \\ \vdots \\ \tilde{a}_m^T b \end{bmatrix}.$$

An engineering interpretation is that the rows of $A$ are sensors whose measurements of an input signal or data point $b$ are recorded in the rows of $Ab$.

Sum of outer products

The matrix product $AB$ can be expressed as a sum of outer products

$$AB = \sum_{i=1}^{n} a_i\tilde{b_i}^T,$$

as you should verify.

5.6 Exercises

1. Prove that if two lines in a vector space $V$ are parallel, then applying a linear map $T: V \rightarrow W$ to each of them yields two lines that are parallel in $W$.