# Linear Algebra Review Notes



In [1]:

import numpy as np



## Gaussian Elimination

• one way to solve $Ax = b$

For example:

$\begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = \begin{bmatrix} 6 \\ 11 \\ 20 \\ \end{bmatrix},\; A = \begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix},\; x = \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ \end{bmatrix},\; b = \begin{bmatrix} 6 \\ 11 \\ 20 \\ \end{bmatrix}$

$[A, b] = \begin{bmatrix}1 & 1 & 1 & 6\\ 3 & 1 & 2 & 11\\ 2 & 3 & 4 & 20\\ \end{bmatrix}$

row2 - row1 * 3: $\begin{bmatrix}1 & 1 & 1 & 6\\ 0 & -2 & -1 & -7\\ 2 & 3 & 4 & 20\\ \end{bmatrix}$

row3 - row1 * 2: $\begin{bmatrix}1 & 1 & 1 & 6\\ 0 & -2 & -1 & -7\\ 0 & 1 & 2 & 8\\ \end{bmatrix}$

exchange row2 with row3: $\begin{bmatrix}1 & 1 & 1 & 6\\ 0 & 1 & 2 & 8\\ 0 & -2 & -1 & -7\\ \end{bmatrix}$

row1 - row2: $\begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & -2 & -1 & -7\\ \end{bmatrix}$

row3 + row2 * 2: $\begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & 0 & 3 & 9\\ \end{bmatrix}$

divide row3 by 3: $\begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & 0 & 1 & 3\\ \end{bmatrix}$

row1 + row3: $\begin{bmatrix}1 & 0 & 0 & 1\\ 0 & 1 & 2 & 8\\ 0 & 0 & 1 & 3\\ \end{bmatrix}$

row2 - row3 * 2: $\begin{bmatrix}1 & 0 & 0 & 1\\ 0 & 1 & 0 & 2\\ 0 & 0 & 1 & 3\\ \end{bmatrix}$

$x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ \end{bmatrix}$



In [7]:

A = np.array([[1,1,1], [3,1,2], [2,3,4]])
b = np.array([6, 11, 20])




In [8]:

A




Out[8]:

array([[1, 1, 1],
[3, 1, 2],
[2, 3, 4]])




In [9]:

b




Out[9]:

array([ 6, 11, 20])




In [10]:

x = np.linalg.solve(A, b)




In [11]:

x




Out[11]:

array([ 1.,  2.,  3.])



## Gaussian-Jordan Elimination

• one way to find $A^{-1}$

For example:

$A = \begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix}$

$[A, I] = \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 3 & 1 & 2 & 0 & 1 & 0\\ 2 & 3 & 4 & 0 & 0 & 1\\ \end{bmatrix}$

row2 - row1 * 3: $\begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & -2 & -1 & -3 & 1 & 0\\ 2 & 3 & 4 & 0 & 0 & 1\\ \end{bmatrix}$

row3 - row1 * 2: $\begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & -2 & -1 & -3 & 1 & 0\\ 0 & 1 & 2 & -2 & 0 & 1\\ \end{bmatrix}$

exchange row2 with row3: $\begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & -2 & -1 & -3 & 1 & 0\\ \end{bmatrix}$

row1 - row2: $\begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & -2 & -1 & -3 & 1 & 0\\ \end{bmatrix}$

row3 + row2 * 2: $\begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 3 & -7 & 1 & 2\\ \end{bmatrix}$

divide row3 by 3: $\begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3} \\ \end{bmatrix}$

row1 + row3: $\begin{bmatrix}1 & 0 & 0 & \frac{2}{3} & \frac{1}{3} & -\frac{1}{3}\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3}\\ \end{bmatrix}$

row2 - row3 * 2: $\begin{bmatrix}1 & 0 & 0 & \frac{2}{3} & \frac{1}{3} & -\frac{1}{3}\\ 0 & 1 & 0 & \frac{8}{3} & -\frac{2}{3} & -\frac{1}{3}\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3}\\ \end{bmatrix}$



In [17]:

A = np.matrix([[1,1,1], [3,1,2], [2,3,4]])




In [18]:

A




Out[18]:

matrix([[1, 1, 1],
[3, 1, 2],
[2, 3, 4]])




In [19]:

np.linalg.inv(A)




Out[19]:

matrix([[ 0.66666667,  0.33333333, -0.33333333],
[ 2.66666667, -0.66666667, -0.33333333],
[-2.33333333,  0.33333333,  0.66666667]])



## Column space

• All linear combinations of columns

• $Ax = b$ is solvable if and only if b is in the column space of A

## Rank

• It is the number of pivots

• It can tell the number of independent columns

• It is the dimension of column space, row space and nullspace

For example:

$A = \begin{bmatrix} 1 & 2 & 2 \\ 2 & 4 & 1 \\ 3 & 6 & 4 \\ \end{bmatrix}$

row2 - row1 * 2 : $\begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 3 & 6 & 4 \\ \end{bmatrix}$

row3 - row1 * 3 : $\begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 0 & 0 & -2 \\ \end{bmatrix}$

row3 - row2 * 3/2 : $\begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 0 & 0 & 0 \\ \end{bmatrix}$

So, $rank(A) = 2$



In [4]:

A = np.matrix([[1,2,2],[2,4,1],[3,6,4]])




In [5]:

A




Out[5]:

matrix([[1, 2, 2],
[2, 4, 1],
[3, 6, 4]])




In [6]:

np.linalg.matrix_rank(A)




Out[6]:

2



## Projection Matrix

$P = A(A^TA)^{-1}A^T$

## Transpose

• $A^T (i,j) = A(j,i)$

For example:

$A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \\ \end{bmatrix} ,\; A^T = \begin{bmatrix}1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \\ \end{bmatrix}$



In [25]:

A = np.matrix([[1,2,3], [4,5,6], [7,8,9]])




In [26]:

A




Out[26]:

matrix([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])




In [27]:

A.transpose()




Out[27]:

matrix([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])




In [28]:

A.T




Out[28]:

matrix([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])



## Eigenvalue & Eigenvector

$$If \; Av = \lambda v ,\; then \; \lambda \;is\; the\; eigenvalue\; of\; matrix\; A,\; and \;v \;is \;a \;eigenvector.\;$$

• solve

• solve