Linear Algebra Review Notes



In [1]:

    
import numpy as np

Gaussian Elimination

one way to solve $ Ax = b $

For example:

$ \begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = \begin{bmatrix} 6 \\ 11 \\ 20 \\ \end{bmatrix},\; A = \begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix},\; x = \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ \end{bmatrix},\; b = \begin{bmatrix} 6 \\ 11 \\ 20 \\ \end{bmatrix}$

$ [A, b] = \begin{bmatrix}1 & 1 & 1 & 6\\ 3 & 1 & 2 & 11\\ 2 & 3 & 4 & 20\\ \end{bmatrix}$

row2 - row1 * 3: $ \begin{bmatrix}1 & 1 & 1 & 6\\ 0 & -2 & -1 & -7\\ 2 & 3 & 4 & 20\\ \end{bmatrix} $

row3 - row1 * 2: $ \begin{bmatrix}1 & 1 & 1 & 6\\ 0 & -2 & -1 & -7\\ 0 & 1 & 2 & 8\\ \end{bmatrix} $

exchange row2 with row3: $ \begin{bmatrix}1 & 1 & 1 & 6\\ 0 & 1 & 2 & 8\\ 0 & -2 & -1 & -7\\ \end{bmatrix} $

row1 - row2: $ \begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & -2 & -1 & -7\\ \end{bmatrix} $

row3 + row2 * 2: $ \begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & 0 & 3 & 9\\ \end{bmatrix} $

divide row3 by 3: $ \begin{bmatrix}1 & 0 & -1 & -2\\ 0 & 1 & 2 & 8\\ 0 & 0 & 1 & 3\\ \end{bmatrix} $

row1 + row3: $ \begin{bmatrix}1 & 0 & 0 & 1\\ 0 & 1 & 2 & 8\\ 0 & 0 & 1 & 3\\ \end{bmatrix} $

row2 - row3 * 2: $ \begin{bmatrix}1 & 0 & 0 & 1\\ 0 & 1 & 0 & 2\\ 0 & 0 & 1 & 3\\ \end{bmatrix} $

$ x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ \end{bmatrix}$



In [7]:

    
A = np.array([[1,1,1], [3,1,2], [2,3,4]])
b = np.array([6, 11, 20])



In [8]:

    
A









    Out[8]:





array([[1, 1, 1],
       [3, 1, 2],
       [2, 3, 4]])



In [9]:

    
b









    Out[9]:





array([ 6, 11, 20])



In [10]:

    
x = np.linalg.solve(A, b)



In [11]:

    
x









    Out[11]:





array([ 1.,  2.,  3.])

Gaussian-Jordan Elimination

one way to find $A^{-1}$

For example:

$ A = \begin{bmatrix}1 & 1 & 1 \\ 3 & 1 & 2 \\ 2 & 3 & 4 \\ \end{bmatrix} $

$ [A, I] = \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 3 & 1 & 2 & 0 & 1 & 0\\ 2 & 3 & 4 & 0 & 0 & 1\\ \end{bmatrix}$

row2 - row1 * 3: $ \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & -2 & -1 & -3 & 1 & 0\\ 2 & 3 & 4 & 0 & 0 & 1\\ \end{bmatrix} $

row3 - row1 * 2: $ \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & -2 & -1 & -3 & 1 & 0\\ 0 & 1 & 2 & -2 & 0 & 1\\ \end{bmatrix} $

exchange row2 with row3: $ \begin{bmatrix}1 & 1 & 1 & 1 & 0 & 0\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & -2 & -1 & -3 & 1 & 0\\ \end{bmatrix} $

row1 - row2: $ \begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & -2 & -1 & -3 & 1 & 0\\ \end{bmatrix} $

row3 + row2 * 2: $ \begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 3 & -7 & 1 & 2\\ \end{bmatrix} $

divide row3 by 3: $ \begin{bmatrix}1 & 0 & -1 & 3 & 0 & -1\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3} \\ \end{bmatrix} $

row1 + row3: $ \begin{bmatrix}1 & 0 & 0 & \frac{2}{3} & \frac{1}{3} & -\frac{1}{3}\\ 0 & 1 & 2 & -2 & 0 & 1\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3}\\ \end{bmatrix} $

row2 - row3 * 2: $ \begin{bmatrix}1 & 0 & 0 & \frac{2}{3} & \frac{1}{3} & -\frac{1}{3}\\ 0 & 1 & 0 & \frac{8}{3} & -\frac{2}{3} & -\frac{1}{3}\\ 0 & 0 & 1 & -\frac{7}{3} & \frac{1}{3} & \frac{2}{3}\\ \end{bmatrix} $



In [17]:

    
A = np.matrix([[1,1,1], [3,1,2], [2,3,4]])



In [18]:

    
A









    Out[18]:





matrix([[1, 1, 1],
        [3, 1, 2],
        [2, 3, 4]])



In [19]:

    
np.linalg.inv(A)









    Out[19]:





matrix([[ 0.66666667,  0.33333333, -0.33333333],
        [ 2.66666667, -0.66666667, -0.33333333],
        [-2.33333333,  0.33333333,  0.66666667]])

Column space

All linear combinations of columns
$ Ax = b $ is solvable if and only if b is in the column space of A

Nullspace

Solutions about Ax = 0

Solutions about Ax = b

Rank

It is the number of pivots
It can tell the number of independent columns
It is the dimension of column space, row space and nullspace

For example:

$ A = \begin{bmatrix} 1 & 2 & 2 \\ 2 & 4 & 1 \\ 3 & 6 & 4 \\ \end{bmatrix}$

row2 - row1 * 2 : $\begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 3 & 6 & 4 \\ \end{bmatrix}$

row3 - row1 * 3 : $\begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 0 & 0 & -2 \\ \end{bmatrix}$

row3 - row2 * 3/2 : $ \begin{bmatrix} 1 & 2 & 2 \\ 0 & 0 & -3 \\ 0 & 0 & 0 \\ \end{bmatrix} $

So, $ rank(A) = 2 $



In [4]:

    
A = np.matrix([[1,2,2],[2,4,1],[3,6,4]])



In [5]:

    
A









    Out[5]:





matrix([[1, 2, 2],
        [2, 4, 1],
        [3, 6, 4]])



In [6]:

    
np.linalg.matrix_rank(A)









    Out[6]:





2

Projection Matrix

$ P = A(A^TA)^{-1}A^T$

Transpose

$A^T (i,j) = A(j,i)$

For example:

$ A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \\ \end{bmatrix} ,\; A^T = \begin{bmatrix}1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \\ \end{bmatrix}$



In [25]:

    
A = np.matrix([[1,2,3], [4,5,6], [7,8,9]])



In [26]:

    
A









    Out[26]:





matrix([[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]])



In [27]:

    
A.transpose()









    Out[27]:





matrix([[1, 4, 7],
        [2, 5, 8],
        [3, 6, 9]])



In [28]:

    
A.T









    Out[28]:





matrix([[1, 4, 7],
        [2, 5, 8],
        [3, 6, 9]])

Determinant

Inverse

Pseudo-inverse

Eigenvalue & Eigenvector

$$ If \; Av = \lambda v ,\; then \; \lambda \;is\; the\; eigenvalue\; of\; matrix\; A,\; and \;v \;is \;a \;eigenvector.\;$$

Linear Algebra Review Notes

Gaussian Elimination

Gaussian-Jordan Elimination

Column space

Nullspace

Solutions about Ax = 0

Solutions about Ax = b

Rank

Projection Matrix

Transpose

Determinant

Inverse

Pseudo-inverse

Eigenvalue & Eigenvector

How to find eigenvalue and eigenvector for a 2X2 matrix:

SVD

References