**(C) 2018-2019 by Damir Cavar**

**Version:** 0.2, September 2019

**Download:** This and various other Jupyter notebooks are available from my GitHub repo.

*Linear Algebra Review and Reference* by Zico Kolter (updated by Chuong Do) from September 30, 2015. This means, many passages are literally copied, many are rewritten. I do not mark sections that are added or different. Consider this notebook a extended annotation of Kolter's (and Do's) notes. See also James E. Gentle (2017) Matrix Algebra: Theory, Computations and Applications in Statistics. Second edition. Springer. Another good resource is Philip N. Klein (2013) Coding the Matrix: Linear Algebra through Applications to Computer Science, Newtonian Press.

See for the introduction to Linear Algebra the Notebook "Linear Algebra".

**Introduction in Philip Klein's Coding the Matrix:** Chapter 12

Assume that after $t$ years the amounts in the two accounts are represented by a 2-vector:

$x^{(t)} = \begin{bmatrix} amount in Account 1 \\[0.3em] amount in Account 2 \end{bmatrix}$

The growth of the amounts in one year can be described in a matrix:

$x^{(t+1)} = \begin{bmatrix} a_{11} & a_{12} \\[0.3em] a_{21} & a_{22} \end{bmatrix} x^{(t)}$

Given the specification of the interest rate above, this simple case gives us:

$x^{(t+1)} = \begin{bmatrix} 1.05 & 0 \\[0.3em] 0 & 1.03 \end{bmatrix} x^{(t)}$

Let $A$ denote the matrix: $\begin{bmatrix} 1.05 & 0 \\[0.3em] 0 & 1.03 \end{bmatrix}$

$A$ is a diagonal.

```
In [10]:
```import numpy as np
x = np.array([[100],
[100]])
A = np.array([[1.05, 0],
[0, 1.03]])
A.dot(x)

```
Out[10]:
```

After two years the accounts would be:

```
In [11]:
```A.dot(A.dot(x))

```
Out[11]:
```

If we might want to know how $x^{(100)}$ compares to $x^{(0)}$, we could iterate over:

We can also write the product as $A^{100}$.

Note that $A$ is a diagonal, thus the entries of $A^{100}$ are $1.05^{100}$ and $1.03^{100}$:

$A^{100} = \begin{bmatrix} 131.50125784630401 & 0 \\[0.3em] 0 & 19.218631980856298 \end{bmatrix}$

Now consider the definition below:

**Basic definition:**

**eigenvalue** of $A$, if there is a non-zero vector $x$, the corresponding **eigenvector**, if the following is true:

$Ax = \lambda x, x \neq 0$

**eigenvalue** of $A$ and $x \in \mathbb{C}^n$ is the corresponding **eigenvector**.

**eigenvector** $x \in \mathbb{C}^n$, and scalar $t \in \mathbb{C}, A(cx) = cAx = c\lambda x = \lambda(cx)$, so $cx$ is also an **eigenvector**. For this reason when we talk about **“the” eigenvector** associated with $\lambda$, we usually assume that the **eigenvector** is normalized to have length $1$ (this still creates some ambiguity, since $x$ and $−x$ will both be **eigenvectors**, but we will have to live with this).

$\{ x : A x = \lambda x \}$

$(\lambda I − A)x = 0, x \neq 0$

$|(\lambda I − A)| = 0$

- The trace of a $A$ is equal to the sum of its eigenvalues,

$\mathrm{tr}A = \sum_{i=1}^n \lambda_i$ - The determinant of $A$ is equal to the product of its eigenvalues,

$|A| = \prod_{i=1}^n \lambda_i$ - The rank of $A$ is equal to the number of non-zero eigenvalues of $A$
- If $A$ is non-singular then $1/\lambda_i$ is an eigenvalue of $A^{−1}$ with associated eigenvector $x_i$, i.e., $A^{−1}x_i = (1/\lambda_i)x_i$. (To prove this, take the eigenvector equation, $Ax_i = \lambda_i x_i$ and left-multiply each side by $A^{−1}$.)
- The eigenvalues of a diagonal matrix $D = \mathrm{diag}(d_1, \dots{}, d_n)$ are just the diagonal entries $d_1, \dots{}, d_n$.

We can write all the eigenvector equations simultaneously as:

$A X = X \Lambda$

**diagonalizable**.

We can compute the eigenvalues and eigenvectors in numpy in the following way:

```
In [12]:
```A = np.array([[ 2., 1. ],
[ 1., 2. ]])
A

```
Out[12]:
```

```
In [13]:
```from numpy import linalg as lg
l = lg.eigvals(A)
print(l)

```
```

Two remarkable properties come about when we look at the eigenvalues and eigenvectors of a symmetric matrix $A \in \mathbb{S}^n$.

- it can be shown that all the eigenvalues of $A$ are real
- the eigenvectors of $A$ are orthonormal, i.e., the matrix $X$ defined above is an orthogonal matrix (for this reason, we denote the matrix of eigenvectors as $U$ in this case).

$x^T A x = x^T U \Lambda U^T x = y^T \Lambda y = \sum_{i=1}^n \lambda_i y^2_i$

$\mathrm{max}_{x\in \mathbb{R}^n} x^T A x \mbox{ subject to } \|x\|^2_2 = 1$

$\mathrm{min}_{x\in \mathbb{R}^n} x^T A x \mbox{ subject to } \|x\|^2_2 = 1$

...

**(C) 2018-2019 by Damir Cavar**