Linear algebra

Motivation

In data science, machine learning (ML), and artificial intelligence (AI), we usually deal not with single numbers but with multivariate (i.e. containing multiple elements or entries) lists of numbers — mathematically speaking, vectors, — and multivariate tables of numbers — mathematically speaking, matrices. Therefore we solve multivariate equations, apply multivariate calculus to find optima of multivariate functions, etc.

The branch of mathematics that studies vectors, matrices, and related mathematical objects is called linear algebra. It is one of the most practically useful areas of mathematics in applied work and a prerequisite for data science, machine learning (ML), and artificial intelligence (AI).

Objectives

• To consider matrices.
• To introduce the transpose of a matrix.
• To demonstrate that matrices also form a vector space.
• To introduce matrix multiplication and consider four different views of matrix multiplication.
• To introduce the identity matrix.
• To introduce linear systems.
• To introduce matrix inverses.
• To demonstrate that matrices "work" on vectors as linear operators.
• To introduce the notion of a determinant.
• To consider in more detail the homogeneous and nonhomogeneous linear systems.
• To consider when a square matrix is invertible and introduce the invertible matrix theorem.
• To introduce gradients (grad), Jacobians, and Hessians.

Question: Why are linear systems so important?

Richard Feynman's answer: Finally, we make some remarks on why linear systems are so important. The answer is simple: because we can solve them! (from The Feynman Lectures on Physics, mainly mechanics, radiation, and heat (1963)).

One of the most useful techniques in all of finance

Regression analysis is one of the most widely used techniques for analyzing multi-factor data. Its broad appeal and usefulness result from the conceptually logical process of using an equation to express the relationship between a variable of interest (the response) and a set of related predictor variables. Regression analysis is also interesting theoretically because of elegant underlying mathematics and a well-developed statistical theory. Successful use of regression requires an appreciation of both the theory and the practical problems that typically arise when the technique is employed with real-world data. \cite{montgomery-2012}

\begin{frame} \frametitle{Linear regression model in matrix form (i)} For $n \in \Nnz$ (the number of \defn{data points}), $p \in \Nnz$ (the number of \defn{features}), $i \in [1, \ldots, n]$, \begin{equation*} y_i = \beta_1 x_{i1} + \ldots + \beta_p x_{ip} + \epsilon_i, \end{equation*} or, in matrix form, \begin{equation*} \V{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{small} \begin{equation*} \V{y} = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}, \quad \M{X} = \begin{pmatrix} \transpose{\V{x}_1} \\ \transpose{\V{x}_2} \\ \vdots \\ \transpose{\V{x}_n} \\ \end{pmatrix} = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1p} \\ x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{np} \\ \end{pmatrix}, \quad \V{\beta} = \begin{pmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_p \end{pmatrix}, \quad \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{pmatrix}. \end{equation*} \end{small} \begin{itemize} \item When $p = 1$ the model is referred to as a \defn{simple} linear regression model, when $p > 1$ it is referred to as a \defn{multivariate} linear regression model. \item $\V{x}_i$ are referred to as the \defn{inputs} (but also \defn{regressors}, \defn{independent variables}, and \defn{explanatory variables}). \item $y_i$ are referred to as the \defn{outputs} (but also \defn{regressands} and \defn{dependent variables}). \item $\epsilon_i$ are referred to as \defn{disturbance} (but also \defn{noise}, \defn{error}) terms. \end{itemize} \end{frame}

\begin{frame} \frametitle{Linear regression model in matrix form (ii)} \begin{itemize} \item In \begin{equation*} \V{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} $\M{X}$ and $\V{y}$ are \nb{given} and \nb{fixed}, perhaps as a table: \begin{center} \begin{tabular}{cccc|c} $x_{11}$ & $x_{12}$ & $\cdots$ & $x_{1p}$ & $y_1$ \\ $x_{21}$ & $x_{22}$ & $\cdots$ & $x_{2p}$ & $y_2$ \\ $\vdots$ & $\vdots$ & $\ddots$ & $\vdots$ & $\vdots$ \\ $x_{n1}$ & $x_{n2}$ & $\cdots$ & $x_{np}$ & $y_n$ \\ \end{tabular} \end{center} \item $\V{\epsilon}$ is a (vector) \nb{random variable}. \item $\V{\beta}$ is unknown. \end{itemize} \end{frame}

\begin{frame} \frametitle{Disturbance versus residual} \begin{itemize} \item Notice that, in \begin{equation*} \V{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} $\V{\epsilon}$ is a (vector) random variable modelling the difference between the regressand $\V{y}$ and the deterministic function $\M{X}\V{\beta}$ involving the regressands $\M{X}$. \item Since $\V{y}$ depends on $\V{\epsilon}$, it is itself a random variable. \item But when we are actually \alert{given} $\V{y}$ as well as $\M{X}$ as data, we can then compute the \alert{deterministic} quantity \begin{equation} \hat{\V{\epsilon}} = \V{y} - \M{X}\V{\beta}, \label{eq:residual-in-linear-regression-wrong} \end{equation} which we call the \alert{residual}. \item Confusingly enough, we use the same notation for $\V{y}$ when it is a random variable as for its fixed sample when it is given. \item To recap, the \alert{disturbance} is random, the \alert{residual} is deterministic. \item Equation (\ref{eq:residual-in-linear-regression-wrong}) is all very well, but we don't actually know $\V{\beta}$ in it! Given the data, $\V{y}$ and $\M{X}$, we need to \alert{fit} the linear regression model and find a ``good enough'' estimate for $\V{\beta}$, say $\hat{\V{\beta}}$. The equation (\ref{eq:residual-in-linear-regression-wrong}) is actually wrong, it should be \begin{equation*} \hat{\V{\epsilon}} = \V{y} - \M{X}\hat{\V{\beta}}. \label{eq:residual-in-linear-regression} \end{equation*} \item We find a ``good enough'' $\hat{\V{\beta}}$ by minimising some sensible function of the residual $\hat{\V{\epsilon}}$. We shall start by considering a particular method of finding the ``good enough'' $\hat{\V{\beta}}$ --- the \defn{ordinary least squares (OLS)}. \end{itemize} \end{frame}

\begin{frame} \frametitle{Who invented linear regression? (i)} \begin{columns} \begin{column}{0.3\textwidth} \begin{figure} \centering \includegraphics[width=1.\textwidth]{francis-galton.pdf} \end{figure} \begin{small} A photograph of Sir~Francis~Galton (1822--1911) from~\cite{pearson-1914} \end{small} \end{column} \begin{column}{0.7\textwidth} \begin{itemize} \item The term \defn{``regression''} was introduced in an 1886 paper on genetics, \atitle{Regression towards Mediocrity in Hereditary Stature}~\cite{galton-1886} by \nb{Sir~Francis~Galton}, a Victorian statistician and polymath interested, among many other things, in genetics, evolution, and natural selection. \item Galton used the word in the sense of \defn{``regression towards the mean''} or, put more bluntly, \defn{``regression towards mediocrity''}. \item His inquiry was prompted by his earlier interest in hereditary genius (Galton was himself a child prodigy): \begin{quote} If a man breeds from strong, well-shaped dogs, but of mixed pedigree, the puppies will be sometimes, but rarely, the equals of their parents. They will commonly be of a mongrel, nondescript type, because \nb{ancestral peculiarities are apt to crop out in the offspring}.~\cite{galton-1869} \end{quote} \end{itemize} \end{column} \end{columns} \end{frame}

\begin{frame} \frametitle{Who invented linear regression? (ii)} Galton was a pioneer in statistics. He computed correlations to make sense of empirical data and made wide use of what would later become known as \defn{regression lines}~\cite{bulmer-2003}. Here is his Plate~X from~\cite{galton-1886}. \begin{center} \begin{figure} \centering \includegraphics[width=.7\textwidth]{galton-1875.pdf} \end{figure} \end{center} \end{frame}

\begin{frame} \frametitle{The classical linear regression model} In the \alert{classical} linear regression model (CLRM), the following assumptions are made: \begin{itemize} \item for all $i \in [1, \ldots, n]$, $\epsilon_i$ have a mean of zero, $\E[\epsilon_i] = 0$; \item for all $i \in [1, \ldots, n]$, $\epsilon_i$ have the same finite variance, $\Var[\epsilon_i] = \sigma^2 < \infty$, in other words, the random variables $\epsilon_i$ are \alert{homoscedastic}; \item for all $i, j \in [1, \ldots, n]$, $i \neq j$, the $\epsilon_i$ are uncorrelated, $\Cov[\epsilon_i, \epsilon_j] = 0$; \item the regressors are \alert{non-random} and \alert{observable}; \item the regressors are not perfectly linearly correlated with each other. \end{itemize} \end{frame}

\section{Ordinary least squares}

\begin{frame} \frametitle{The ordinary least squares (OLS) approach} \begin{itemize} \item The idea is to find such a $\V{\beta}$ in \begin{equation*} \V{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} --- we'll refer to it as $\hat{\V{\beta}}$ --- that will give us the \defn{estimate} of $\V{y}$, \begin{equation*} \hat{\V{y}} \defeq \M{X}\hat{\V{\beta}}, \end{equation*} \item ...that will minimize a certain \nb{loss function} of the \defn{residual} \begin{equation*} \hat{\V{\epsilon}} \defeq \V{y} - \hat{\V{y}}, \end{equation*} \item ...namely the squared error, which we'll now call the \defn{residual sum of squares (RSS)}, \begin{equation*} \text{RSS} = \transpose{\hat{\V{\epsilon}}}\hat{\V{\epsilon}} = \transpose{\left(\V{y} - \M{X}\hat{\V{\beta}}\right)} \left(\V{y} - \M{X}\hat{\V{\beta}}\right). \end{equation*} \end{itemize} \end{frame}

\begin{frame} \frametitle{Derivation of OLS} \begin{theorem} \begin{equation*} \hat{\V{\beta}} = \left(\transpose{\M{X}}\M{X}\right)^{-1}\transpose{\M{X}}\V{y}. \end{equation*} \begin{proof} \begin{equation*} \text{RSS} = \transpose{\hat{\V{\epsilon}}}\hat{\V{\epsilon}} = \transpose{\left(\V{y} - \M{X}\V{\beta}\right)} \left(\V{y} - \M{X}\V{\beta}\right) = \transpose{\V{y}}\V{y} - 2\transpose{\V{y}}\M{X}\V{\beta} + \transpose{\V{\beta}}\transpose{\M{X}}\M{X}\V{\beta}. \end{equation*} Using the denominator notation for vector-by-vector and scalar-by-vector derivatives, \begin{equation*} \nabla_{\V{\beta}}\text{RSS} = \frac{\partial\text{RSS}}{\partial\V{\beta}} = -2\transpose{\M{X}}\V{y} + 2\transpose{\M{X}}\M{X}\V{\beta}. \end{equation*} When $\nabla_{\V{\beta}}\text{RSS} = 0$, $\transpose{\M{X}}\M{X}\V{\beta} = \transpose{\M{X}}\V{y}$ (this is the so-called \alert{normal equation}), hence $\V{\beta} = \left(\transpose{\M{X}}\M{X}\right)^{-1}\transpose{\M{X}}\V{y}$. To check that this critical point is a minimum, we examine the Hessian matrix \begin{equation*} \frac{\partial}{\partial{\V{\beta}}}\left(\frac{\partial\text{RSS}}{\partial\V{\beta}}\right) = 2\transpose{\M{X}}\M{X}. \end{equation*} If $\M{X}$ is full rank (i.e. has rank $p+1$ if we allow the intercept and $p$ if we don't; this implies that $\M{X}$ is skinny or square), then $\transpose{\M{X}}\M{X}$ is positive definite and the critical point is indeed a minimum. \end{proof} \end{theorem} \end{frame}

\begin{frame} \frametitle{The answer is...} ...\nb{yes}, it was Gau{\ss} \nb{or} Legendre. In~\cite{stigler-1981}, \begin{quote} It is argued (though not conclusively) that Gauss probably possessed the method well before Legendre, but that he was unsuccessful in communicating it to his contemporaries. \end{quote} \end{frame}

\section{The geometry of linear regression}

\begin{frame} \frametitle{A word to Sir Isaac Newton} \begin{columns} \begin{column}{0.3\textwidth} \begin{figure} \centering \includegraphics[width=1.\textwidth]{isaac-newton.pdf} \end{figure} \begin{small} Portrait of Sir~Isaac~Newton at~46 (1642--1726/7) by Godfrey~Kneller \end{small} \end{column} \begin{column}{0.7\textwidth} In~\atitle{Rules for methodizing the Apocalypse}: \begin{quote} Truth is ever to be found in \nb{simplicity}, and not in the multiplicity and confusion of things. \end{quote} So let us look at a simple, low-dimensional example. (We would like to thank \nb{Douglas Machado Vieira} for this suggestion.) \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{Let us understand the geometry of this linear regression problem} \begin{itemize} \item We are given the linear model \begin{equation*} \M{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{equation*} \M{X} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}, \V{y} = \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix}, \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \epsilon_3 \end{pmatrix}. \end{equation*} \item We need to estimate $\V{\beta}$. \item Here $p = 1$ (we have a single feature) and $n = 3$ (we have three data points). Our dataset consists of three data points, three $(x_i, y_i)$ pairs: $(1, 2)$, $(3, 1)$, $(4, 2)$. Because $p = 1$, all our $x_i$ are univariate: 1, 3, and 4. \end{itemize} \end{frame}\begin{frame} \frametitle{What are we minimising? (i)} \begin{center} We are fitting the regression line (shown in \nb{red}) to minimize... \\ ...the sum of squares of the lengths of these \nb{vertical} dashed lines? \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} Given the linear model \begin{equation*} \M{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{equation*} \M{X} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}, \V{y} = \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix}, \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \epsilon_3 \end{pmatrix}, \end{equation*} estimate $\V{\beta}$. \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{What are we minimising? (ii)} \begin{center} We are fitting the regression line (shown in \nb{red}) to minimize... \\ ...the sum of squares of the lengths of these \nb{orthogonal} (to the fitted line) dashed lines? \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1.6/0.8/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzDefPoints{3/1/B, 2.8/1.4/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} Given the linear model \begin{equation*} \M{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{equation*} \M{X} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}, \V{y} = \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix}, \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \epsilon_3 \end{pmatrix}, \end{equation*} estimate $\V{\beta}$. \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{Ordinary least squares (OLS)} \begin{align*} \hat{\V{\beta}} &= (\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}\V{y} \\ &= (1^2 + 3^2 + 4^2)^{-1} (1 \cdot 2 + 3 \cdot 1 + 4 \cdot 2) \\ &= \frac{13}{26} = \frac{1}{2}. \end{align*} \begin{align*} \hat{\V{y}} &= \M{X}\hat{\V{\beta}} \\ &= \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix} \cdot \frac{1}{2} \\ &= \begin{pmatrix} \sfrac{1}{2} \\ \sfrac{3}{2} \\ 2 \end{pmatrix}. \end{align*} \end{frame}\begin{frame} \frametitle{So indeed we are minimising...} \begin{center} We are fitting the regression line (shown in \nb{red}) to minimize... \\ ...the sum of squares of the \nb{vertical} dashed lines, $\V{y} - \hat{\V{y}} = \V{y} - \M{X}\hat{\V{\beta}}$. \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,\frac{1}{2})$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,\frac{3}{2})$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} Given the linear model \begin{equation*} \M{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{equation*} \M{X} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}, \V{y} = \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix}, \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \epsilon_3 \end{pmatrix}, \end{equation*} estimate $\V{\beta}$. \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{Moore--Penrose pseudoinverse} Let's have a look at our $\hat{\V{\beta}}$: \begin{equation*} \hat{\V{\beta}} = \underbrace{(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}}_{\M{X}^+}\V{y} \end{equation*} \begin{itemize} \item $\M{X}^+$ is the \defn{Moore--Penrose pseudoinverse} matrix of $\M{X}$. \item In our example, \begin{align*} \M{X}^+ &= (\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} \\ &= (1^2 + 3^2 + 4^2)^{-1} \begin{pmatrix} 1 & 3 & 4 \end{pmatrix} \\ &= \begin{pmatrix} \frac{1}{26} & \frac{3}{26} & \frac{4}{26} \end{pmatrix}. \end{align*} \item One of its properties is $(\M{X}\M{X}^+)\M{X} = \M{X}$; in other words, $\M{X}\M{X}^+$ maps all column vectors of $\M{X}$ to themselves, while $\M{X}\M{X}^+$ need not be the identity matrix. \item In our example, \begin{equation*} (\M{X}\M{X}^+)\M{X} = \left[ \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix} \begin{pmatrix} \frac{1}{26} & \frac{3}{26} & \frac{4}{26} \end{pmatrix} \right] \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix} = \underbrace{\begin{pmatrix} \sfrac{1}{26} & \sfrac{3}{26} & \sfrac{4}{26} \\ \sfrac{3}{26} & \sfrac{9}{26} & \sfrac{12}{26} \\ \sfrac{4}{26} & \sfrac{12}{26} & \sfrac{16}{26} \end{pmatrix}}_{\text{not an identity matrix, but...}} \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}. \end{equation*} \end{itemize} \end{frame}\begin{frame} \frametitle{The hat matrix} Let's have a look at our $\hat{\V{y}}$: \begin{align*} \hat{\V{y}} &= \M{X}\hat{\V{\beta}} \\ &= \underbrace{\M{X}\underbrace{(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}}_{\M{X}^+}}_{\M{H}}\V{y} \end{align*} \begin{itemize} \item The matrix $\M{H} \defeq \M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}$ is known as the \defn{hat matrix} --- because it takes $\V{y}$ and puts a hat on it, $\hat{\V{y}}$. \item In our example, \begin{equation*} \M{H} = \M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} = \M{X}\M{X}^+ = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix} \begin{pmatrix} \frac{1}{26} & \frac{3}{26} & \frac{4}{26} \end{pmatrix} = \begin{pmatrix} \sfrac{1}{26} & \sfrac{3}{26} & \sfrac{4}{26} \\ \sfrac{3}{26} & \sfrac{9}{26} & \sfrac{12}{26} \\ \sfrac{4}{26} & \sfrac{12}{26} & \sfrac{16}{26} \end{pmatrix}. \end{equation*} \item But this is just the matrix from the previous slide that pretended to be an identity matrix (although only for $\V{X}$)... \end{itemize} \end{frame}\begin{frame} \frametitle{Orthogonal projection matrices} \begin{itemize} \item We recognize $\M{H}$ as one of the so-called \defn{orthogonal projection matrices}. \item Let's check that it has the two properties of orthogonal projection matrices: \begin{itemize} \item it is \defn{symmetric}: \begin{align*} \transpose{\M{H}} &= \transpose{(\M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}})} \\ &= \transpose{(\transpose{\M{X}})}\transpose{[(\transpose{\M{X}}\M{X})^{-1}]}\transpose{\M{X}} \\ &= \M{X}[\transpose{(\transpose{\M{X}}\M{X})}]^{-1}\transpose{\M{X}} \\ &= \M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} \\ &= \M{H}; \end{align*} \item it is \defn{idempotent}: \begin{align*} \M{H}^2 &= (\M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}) (\M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}}) \\ &= \M{X}[(\transpose{\M{X}}\M{X})^{-1}(\transpose{\M{X}} \M{X})](\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} \\ &= \M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} \\ &= \M{H}. \end{align*} \end{itemize} \item \nb{STOP!} Did we not decide that we were minimising the sum of squares of the lengths of the \nb{vertical} dashed lines and not of the \nb{orthogonal} dashed lines? \item \nb{Where is the orthogonal projection?} \end{itemize} \end{frame}\begin{frame} \frametitle{The mystery of the missing orthogonal projection (i)} \begin{center} We are fitting the regression line (shown in \nb{red} on the left plot) to minimize... \\ ...the sum of squares of the \nb{vertical} dashed lines, $\V{y} - \hat{\V{y}} = \V{y} - \M{X}\hat{\V{\beta}}$. \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,\frac{1}{2})$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,\frac{3}{2})$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (0.5,1.5,2){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[below right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat); \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \end{tikzpicture} \end{column} \end{columns} \begin{center} We find our (OLS) solution by orthogonally projecting $\V{y}$ (the \textcolor{green}{green} line) onto the column space of $\M{X}$ (the \textcolor{blue}{blue} line or hyperplane) --- see the \nb{dashed red line} on the right plot. \end{center} \end{frame}\begin{frame} \frametitle{The mystery of the missing orthogonal projection (ii)} \begin{itemize} \item We spotted the orthogonal projection by \nb{changing our point of view}. \item The left-hand plot represents the so-called \defn{variable space} view, $\R^p$, view of the linear regression: here each data point is indeed represented by a point and the dimensions represent the features. Here we can see the regression line. \item The right-hand plot represents the so-called \defn{subject space} view, $\R^n$. Here $y$ and $\hat{y}$ appear as vectors represented by lines and the dimensions represent the data points. \item The orthogonal projection was \nb{hidden} on the variable space view, but became \nb{revealed} on the subject space view. \item This in itself is a useful lesson in data science. \end{itemize} \end{frame}\begin{frame} \frametitle{The importance of point of view in data science} \begin{columns} \begin{column}{0.3\textwidth} \begin{figure} \centering \includegraphics[width=1.\textwidth]{obi-wan-kenobi.pdf} \end{figure} \end{column} \begin{column}{0.7\textwidth} You're going to find that many of the truths we cling to depend greatly on our own point of view. \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{The mystery of the missing orthogonal projection (iii)} \begin{itemize} \item The hat matrix, $\M{H}$ is an \defn{orthogonal projection matrix} that projects $\V{y}$ onto the \defn{column space} of $\M{X}$ --- the space spanned by the column vectors of $\M{X}$. \item The result, $\hat{\V{y}}$, is a \defn{linear combination} of the columns of $\M{X}$. \item The \defn{weights} in this linear combination are given by the elements of $\hat{\V{\beta}}$. \item In our example, $p = 1$, so $\M{X}$ has a single column. Thus the result, $\hat{\V{y}}$, is a scalar multiple of that column, the scalar being our $\hat{\beta} = \frac{1}{2}$: \begin{equation*} \hat{\V{y}} = \M{H}\V{y} = \begin{pmatrix} \sfrac{1}{26} & \sfrac{3}{26} & \sfrac{4}{26} \\ \sfrac{3}{26} & \sfrac{9}{26} & \sfrac{12}{26} \\ \sfrac{4}{26} & \sfrac{12}{26} & \sfrac{16}{26} \end{pmatrix} \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix} = \begin{pmatrix} \sfrac{1}{2} \\ \sfrac{3}{2} \\ 2 \end{pmatrix} = \underbrace{\frac{1}{2}}_{\hat{\beta}} \underbrace{\begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}}_{\text{the single column of $\M{X}$}}. \end{equation*} \end{itemize} \end{frame}\begin{frame} \frametitle{Would we actually draw this line?} If we were drawing a line of best fit, would we actually draw it through $(x_1, \hat{y}_1) = (1, \sfrac{1}{2})$, $(x_2, \hat{y}_2) = (3, \sfrac{3}{2})$, $(x_3, \hat{y}_3) = (4, 2)$? Or would we draw the line on the right? \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,\frac{1}{2})$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,\frac{3}{2})$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/1.7857/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){} \tkzDefPoints{3/1/B, 3/1.6428/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){} \tkzDefPoints{4/2/C, 4/1.5714/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){} \tkzDefPoints{0/1.8571/J, 5/1.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{The $\V{y}$-intercept} \begin{itemize} \item If we didn't have to start our line of best fit at the origin, we would draw the line on the right. \item Fortunately, it is easy to introduce the \defn{($\V{y}$-)~intercept} into our linear model without changing much of its structure: \begin{itemize} \item prepend a column of ones to $\M{X}$, so it becomes the $n \times (p+1)$-dimensional matrix \begin{equation*} \M{X} = \begin{pmatrix} 1 & 1 \\ 1 & 3 \\ 1 & 4 \end{pmatrix}; \end{equation*} \item prepend $\beta_0$ to the column vector $\hat{\V{\beta}}$, so it becomes the $p+1$-dimensional column vector \begin{equation*} \hat{\V{\beta}} = \begin{pmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{pmatrix}. \end{equation*} \end{itemize} \item And then proceed as before... \end{itemize} \end{frame}\begin{frame} \frametitle{Linear regression model with $\V{y}$-intercept in matrix form} For $n \in \Nnz$, $p \in \Nnz$, $i \in [1, \ldots, n]$, \begin{equation*} y_i = \beta_0 + \beta_1 x_{i1} + \ldots + \beta_p x_{ip} + \epsilon_i, \end{equation*} or, in matrix form, \begin{equation*} \V{y} = \M{X}\V{\beta} + \V{\epsilon}, \end{equation*} where \begin{small} \begin{equation*} \V{y} = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}, \quad \M{X} = \begin{pmatrix} 1 & \transpose{\V{x}_1} \\ 1 & \transpose{\V{x}_2} \\ \vdots & \vdots \\ 1 & \transpose{\V{x}_n} \\ \end{pmatrix} = \begin{pmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \\ \end{pmatrix}, \quad \V{\beta} = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{pmatrix}, \quad \V{\epsilon} = \begin{pmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{pmatrix}. \end{equation*} \end{small} \end{frame}\begin{frame} \frametitle{An OLS solution with the $\V{y}$-intercept permitted} \begin{itemize} \item The Moore--Penrose pseudoinverse matrix of $\M{X}$ is given by \begin{align*} \M{X}^+ = (\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} &= \left[ \begin{pmatrix} 1 & 1 & 1 \\ 1 & 3 & 4 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & 3 \\ 1 & 4 \end{pmatrix} \right]^{-1} \begin{pmatrix} 1 & 1 & 1 \\ 1 & 3 & 4 \end{pmatrix} \\ &= \begin{pmatrix} 3 & 8 \\ 8 & 26 \end{pmatrix}^{-1} \begin{pmatrix} 1 & 1 & 1 \\ 1 & 3 & 4 \end{pmatrix} \\ &= \frac{1}{14} \begin{pmatrix} 26 & -8 \\ -8 & 3 \end{pmatrix} \begin{pmatrix} 1 & 1 & 1 \\ 1 & 3 & 4 \end{pmatrix} \\ &= \frac{1}{14} \begin{pmatrix} 18 & 2 & -6 \\ -5 & 1 & 4 \end{pmatrix}. \end{align*} \item Hence \begin{equation*} \hat{\V{\beta}} = \M{X}^+ \V{y} = \frac{1}{14} \begin{pmatrix} 18 & 2 & -6 \\ -5 & 1 & 4 \end{pmatrix} \begin{pmatrix} 2 \\ 1 \\ 2 \end{pmatrix} = \begin{pmatrix} \sfrac{13}{7} \\ -\sfrac{1}{14} \end{pmatrix} = \begin{pmatrix} \beta_0 \\ \beta_1 \end{pmatrix}, \end{equation*} \item and \begin{equation*} \hat{\V{y}} = \M{X}\hat{\V{\beta}} = \begin{pmatrix} 1 & 1 \\ 1 & 3 \\ 1 & 4 \end{pmatrix} \begin{pmatrix} \sfrac{13}{7} \\ -\sfrac{1}{14} \end{pmatrix} = \begin{pmatrix} \sfrac{25}{14} \\ \sfrac{23}{14} \\ \sfrac{11}{7} \end{pmatrix} = \hat{\beta}_0 \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} + \hat{\beta}_1 \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}. \end{equation*} \end{itemize} \end{frame}\begin{frame} \frametitle{The $\M{X}$-$\V{y}$ plot} \begin{columns} \begin{column}{0.5\textwidth} Without the intercept: \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,\frac{1}{2})$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,\frac{3}{2})$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} With the intercept: \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/1.7857/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,1\sfrac{11}{14})$} \tkzDefPoints{3/1/B, 3/1.6428/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,1\sfrac{9}{14})$} \tkzDefPoints{4/2/C, 4/1.5714/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,1\sfrac{4}{7})$} \tkzDefPoints{0/1.8571/J, 5/1.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{The orthogonal projection without the intercept} \begin{center} We are fitting the regression line (shown in \nb{red} on the left plot) to minimize... \\ ...the sum of squares of the \nb{vertical} dashed lines, $\V{y} - \hat{\V{y}} = \V{y} - \M{X}\hat{\V{\beta}}$. \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/0.5/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,\frac{1}{2})$} \tkzDefPoints{3/1/B, 3/1.5/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,\frac{3}{2})$} \tkzDefPoints{4/2/C, 4/2/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,2)$} \tkzDefPoints{0/0/J, 5/2.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (0.5,1.5,2){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[below right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat); \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \end{tikzpicture} \end{column} \end{columns} \begin{center} We find our (OLS) solution by orthogonally projecting $\V{y}$ (the \textcolor{green}{green} line) onto the column space of $\M{X}$ (the \textcolor{blue}{blue} line or hyperplane) --- see the \nb{dashed red line} on the right plot. \end{center} \end{frame}\begin{frame} \frametitle{The orthogonal projection with the intercept} \begin{center} We are fitting the regression line (shown in \nb{red} on the left plot) to minimize... \\ ...the sum of squares of the \nb{vertical} dashed lines, $\V{y} - \hat{\V{y}} = \V{y} - \M{X}\hat{\V{\beta}}$. \end{center} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}] \tkzInit[xmin=-1,ymin=-1,xmax=5,ymax=5] \tkzAxeXY \tkzGrid \tkzDefPoints{1/2/A, 1/1.7857/A1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](A,A1) \tkzDrawPoints[size=10,color=red](A, A1) \tkzLabelPoint[above,color=blue](A){$(1,2)$} \tkzLabelPoint[below,color=red](A1){$(1,1\sfrac{11}{14})$} \tkzDefPoints{3/1/B, 3/1.6428/B1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](B,B1) \tkzDrawPoints[size=10,color=red](B, B1) \tkzLabelPoint[above right,color=blue](B){$(3,1)$} \tkzLabelPoint[above left,color=red](B1){$(3,1\sfrac{9}{14})$} \tkzDefPoints{4/2/C, 4/1.5714/C1} \tkzDrawSegments[color=blue, ultra thick, densely dashed](C,C1) \tkzDrawPoints[size=10,color=red](C, C1) \tkzLabelPoint[above left,color=blue](C){$(4,2)$} \tkzLabelPoint[below right,color=red](C1){$(4,1\sfrac{4}{7})$} \tkzDefPoints{0/1.8571/J, 5/1.5/K} \tkzDrawSegments[color=red, ultra thick](J,K) \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (X0) at (1,1,1){}; \coordinate (X01) at (1,0,0){}; \coordinate (X02) at (0,1,0){}; \coordinate (X03) at (0,0,1){}; \coordinate (X012) at (1,1,0){}; \coordinate (X013) at (1,0,1){}; \coordinate (X023) at (0,1,1){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (1.7857,1.6428,1.5714){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}_{:,1}$}; \draw[color=blue, thick] (O)--(X0); \draw[color=blue, dashed, thin] (X01)--(X012); \draw[color=blue, dashed, thin] (X01)--(X013); \draw[color=blue, dashed, thin] (X02)--(X012); \draw[color=blue, dashed, thin] (X02)--(X023); \draw[color=blue, dashed, thin] (X03)--(X013); \draw[color=blue, dashed, thin] (X03)--(X023); \draw[color=blue, dashed, thin] (X012)--(X0); \draw[color=blue, dashed, thin] (X013)--(X0); \draw[color=blue, dashed, thin] (X023)--(X0); \draw[fill=black] (X0) circle (0.2em) node[below right] {$\V{X}_{:,0}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[above left] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat); \coordinate (aa) at (3,1,0){}; \filldraw[color=blue, fill opacity=0.3, dashed, thin] (O)--(aa)--(X); \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \end{tikzpicture} \end{column} \end{columns} \begin{center} We find our (OLS) solution by orthogonally projecting $\V{y}$ (the \textcolor{green}{green} line) onto the column space of $\M{X}$ (the \textcolor{blue}{blue} line or hyperplane) --- see the \nb{dashed red line} on the right plot. \end{center} \end{frame}\begin{frame} \frametitle{The hat matrix now projects onto a plane, not onto a line} \begin{itemize} \item When we allow the intercept, the hat matrix becomes \begin{align*} \M{H} &= \M{X}(\transpose{\M{X}}\M{X})^{-1}\transpose{\M{X}} \\ &= \M{X}\M{X}^+ \\ &= \begin{pmatrix} 1 & 1 \\ 1 & 3 \\ 1 & 4 \end{pmatrix} \left[ \frac{1}{14} \begin{pmatrix} 18 & 2 & -6 \\ -5 & 1 & 4 \end{pmatrix} \right] \\ &= \frac{1}{14} \begin{pmatrix} 13 & 3 & -2 \\ 3 & 5 & 6 \\ -2 & 6 & 10 \end{pmatrix}. \end{align*} \item As before, it projects $\V{y}$ onto the column space of $\M{X}$, the resulting projection being $\hat{\V{y}}$. \item But now $\M{X}$ has two (linearly independent) columns, not one, so the projection is onto a plane, not onto a line. If we increase $p$ and add more columns to $\M{X}$, if these columns are linearly independent, we'll further increase the dimension of the column space of $\M{X}$ onto which we project. \end{itemize} \end{frame}\begin{frame} \frametitle{The equation of a plane} \begin{itemize} \item Recall that the equation of a plane orthogonal to a vector $\V{a}$ and containing a point $\V{x}_0$ is given by $\transpose{\V{a}} (\V{x} - \V{x}_0) = 0$. (We are simply stating a fact: $\V{a}$ and $\V{x} - \V{x}_0$ (the latter lies on the plane) are orthogonal.) \item In our case, \begin{equation*} \hat{\V{\epsilon}} = \V{y} - \hat{\V{y}} = \begin{pmatrix} \sfrac{3}{14} \\ -\sfrac{9}{14} \\ \sfrac{3}{7} \end{pmatrix} \end{equation*} is orthogonal to the plane that contains the column vectors of $\M{X}$ --- $\transpose{\begin{pmatrix} 1 & 1 & 1 \end{pmatrix}}$ and $\transpose{\begin{pmatrix} 1 & 3 & 4 \end{pmatrix}}$. \item We can use either of them to work out the equation of the plane: \begin{equation*} \frac{3}{14} (x_1 - 1) - \frac{9}{14} (x_2 - 3) + \frac{3}{7} (x_3 - 4) = 0 \Rightarrow \frac{3}{14} x_1 - \frac{9}{14} x_2 + \frac{3}{7} x_3 = 0. \end{equation*} \end{itemize} \end{frame}\begin{frame} \frametitle{The OLS solution} \begin{itemize} \item The resulting OLS solution is a linear combination of the columns of $\M{X}$, the weights being given by the elements of $\hat{\V{\beta}}$: \begin{equation*} \hat{\V{y}} = \M{X}\hat{\V{\beta}} = \begin{pmatrix} \sfrac{25}{14} \\ \sfrac{23}{14} \\ \sfrac{11}{7} \end{pmatrix} = \hat{\beta}_0 \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} + \hat{\beta}_1 \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}. \end{equation*} \item Here $\hat{\beta}_0$ is, by construction, the $y$-intercept of the regression line and $\hat{\beta}_1$ is its gradient. \end{itemize} \end{frame}\begin{frame} \frametitle{So have we...} ...solved the mystery of the missing orthogonal projection (and understood the geometry of~OLS)? \end{frame}\begin{frame} \frametitle{Just one more thing} \begin{columns} \begin{column}{0.3\textwidth} \begin{figure} \centering \includegraphics[width=1.\textwidth]{columbo.pdf} \end{figure} \end{column} \begin{column}{0.7\textwidth} Another orthogonal projection is lurking right in front of us and we have nearly missed it! \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{The other projection matrix} \begin{itemize} \item The matrix $\M{M} = \M{I}_n - \M{H}$, known as the \defn{annihilator} matrix, is \nb{also} an orthogonal projection matrix, and it is closely linked to the hat matrix $\M{H}$. \item Notice that $\M{M}$ inherits the following properties from $\M{H}$: it is also \begin{itemize} \item \nb{symmetric}: \begin{equation*} \transpose{\M{M}} = \transpose{\M{I}_n - \M{H}} = \transpose{\M{I}_n} - \transpose{\M{H}} = \M{I}_n - \M{H} = \M{M}; \end{equation*} \item \nb{idempotent}: \begin{equation*} \M{M}^2 = (\M{I}_n - \M{H}) (\M{I}_n - \M{H}) = \M{I}_n \M{I}_n - \M{H} \M{I}_n - \M{I}_n \M{H} + \M{H} \M{H} = \M{I}_n - \M{H} - \M{H} + \M{H} = \M{I}_n - \M{H} = \M{M}. \end{equation*} \end{itemize} \item This matrix projects onto the space that is orthogonal\footnote{Two vector subspaces $A$ and $B$ are \defn{orthogonal} if each vector in $A$ is orthogonal to each vector in $B$.} to the column space of $\M{X}$ --- its so called \defn{orthogonal complement}. \item In fact, \begin{equation*} \hat{\V{\epsilon}} = \V{y} - \hat{\V{y}} = \V{y} - \M{H}\V{y} = (\M{I}_n - \M{H})\V{y} = \M{M}\V{y}, \end{equation*} and we see that the residuals lie in this vector space. \item The orthogonal projection matrix $\M{M}$ acts on $\M{y}$ to obtain the residuals. \end{itemize} \end{frame}\begin{frame} \frametitle{The orthogonal decomposition} Thus we have an \defn{orthogonal decomposition} of $\V{y}$: $\V{y} = \hat{\V{y}} + \V{y} - \hat{\V{y}} = \hat{\V{y}} + \hat{\V{\epsilon}} = \underbrace{\M{H}\V{y}}_{\hat{\V{y}}} + \underbrace{\M{M}\V{y}}_{\hat{\V{\epsilon}}}$: \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (0.5,1.5,2){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[below right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat) node[midway, above, color=black] {$\hat{\V{\epsilon}}$}; \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \tkzFindAngle(yhat,O,y) \tkzGetAngle{angleyhatOy}; \tkzDrawArc[R with nodes, color=red](O,1cm)(yhat,y) \tkzLabelAngle[above, pos=-0.1](yhat,O,y){$\theta$} \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (X0) at (1,1,1){}; \coordinate (X01) at (1,0,0){}; \coordinate (X02) at (0,1,0){}; \coordinate (X03) at (0,0,1){}; \coordinate (X012) at (1,1,0){}; \coordinate (X013) at (1,0,1){}; \coordinate (X023) at (0,1,1){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (1.7857,1.6428,1.5714){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}_{:,1}$}; \draw[color=blue, thick] (O)--(X0); \draw[color=blue, dashed, thin] (X01)--(X012); \draw[color=blue, dashed, thin] (X01)--(X013); \draw[color=blue, dashed, thin] (X02)--(X012); \draw[color=blue, dashed, thin] (X02)--(X023); \draw[color=blue, dashed, thin] (X03)--(X013); \draw[color=blue, dashed, thin] (X03)--(X023); \draw[color=blue, dashed, thin] (X012)--(X0); \draw[color=blue, dashed, thin] (X013)--(X0); \draw[color=blue, dashed, thin] (X023)--(X0); \draw[fill=black] (X0) circle (0.2em) node[below right] {$\V{X}_{:,0}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[above right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat) node[midway, above, color=black] {$\hat{\V{\epsilon}}$}; \coordinate (aa) at (3,1,0){}; \filldraw[color=blue, fill opacity=0.3, dashed, thin] (O)--(aa)--(X); \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \end{tikzpicture} \end{column} \end{columns} \end{frame}

\section{How well are we doing?}

\begin{frame} \frametitle{Variance partitioning (i)} \begin{itemize} \item Let $\bar{y} = \sum_{i=1}^n y_i$ be the sample mean. Let us define \begin{itemize} \item \defn{Total Sum of Squares (TSS)}, sometimes also referred to as \nb{Sum of Squares, Total (SST, SSTO)} and \defn{SYY}: $\sum_{i=1}^n (y_i - \bar{y})^2$; \item \defn{Estimated Sum of Squares (ESS)}, sometimes also referred to as \nb{Sum of Squares, Model (SSM)}, \nb{Estimated Sum of Squares (ESS)}, and \nb{Sum of Squares, Regression (SSR, SSreg)}: $\sum_{i=1}^n (\hat{y}_i - \bar{y})^2$; \item \defn{Residual Sum of Squares (RSS)}, sometimes also referred to as \nb{Sum of Squares, Error (SSE)}: $\sum_{i=1}^n (y_i - \hat{y}_i)^2$. \end{itemize} \item Notice that the abbreviations used in the literature can be quite \nb{confusing}: does ``E'' stand for ``estimated'' or for ``error''; does ``R'' stand for ``regression'' or for ``residual''? To avoid confusion, we shall stick with \defn{Total Sum of Squares (TSS)}, \defn{Estimated Sum of Squares (ESS)}, \defn{Residual Sum of Squares (RSS)}. \item If we define $\bar{\V{y}} \in \R^n$ as a vector all of whose elements are the sample mean $\bar{y}$, we can write these down in vector notation: \begin{itemize} \item $\text{TSS} = \sum_{i=1}^n (y_i - \bar{y})^2 = \transpose{(\V{y} - \bar{\V{y}})}(\V{y} - \bar{\V{y}})$; \item $\text{ESS} = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 = \transpose{(\hat{\V{y}} - \bar{\V{y}})}(\hat{\V{y}} - \bar{\V{y}})$; \item $\text{RSS} = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \transpose{(\V{y} - \hat{\V{y}})}(\V{y} - \hat{\V{y}})$. \end{itemize} \end{itemize} \end{frame}\begin{frame} \frametitle{Variance partitioning (ii)} \begin{itemize} \item Now notice that \begin{itemize} \item $\text{TSS} = \norm{\V{y} - \bar{\V{y}}}^2 = \norm{\V{y}}^2 - 2\transpose{\V{y}}\bar{\V{y}} + \norm{\bar{\V{y}}}^2$; \item $\text{ESS} = \norm{\hat{\V{y}} - \bar{\V{y}}}^2 = \norm{\hat{\V{y}}}^2 - 2\transpose{\hat{\V{y}}}\bar{\V{y}} + \norm{\bar{\V{y}}}^2$; \item $\text{RSS} = \norm{\V{y} - \hat{\V{y}}}^2 = \norm{\V{\epsilon}}^2$. \end{itemize} \item If the model includes the intercept, the first column of $\M{X}$ is a column of ones and, since $\transpose{\M{X}}\hat{\V{\epsilon}} = \V{0}$, the sum of the residuals is zero. This is equivalent to $\transpose{\V{y}}\bar{\V{y}} = \transpose{\hat{\V{y}}}\bar{\V{y}}$. \item By Pythagoras's theorem, \begin{equation*} \norm{\V{y}}^2 = \norm{\hat{\V{y}}}^2 + \norm{\hat{\V{\epsilon}}}^2. \end{equation*} \item Hence \begin{equation*} \text{TSS} = \text{ESS} + \text{RSS}, \end{equation*} a fundamental result known as \defn{variance partitioning}. \item Irrespective of whether our model includes the intercept, if we center $\V{y}$ by subtracting the sample mean $\bar{y} = \sum_{i=1}^n y_i$ from each element, we notice that $\text{TSS} = \text{ESS} + \text{RSS}$ simply amounts to the above application of Pythagoras's theorem. \end{itemize} \end{frame}\begin{frame} \frametitle{From Christopher Clavius's edition of Euclid's Elements} \begin{figure} \centering \includegraphics[width=.72\textwidth]{euclidis-elementorum.pdf} \end{figure} \begin{small} From \url{https://www.maa.org/press/periodicals/convergence/mathematical-treasures-christopher-claviuss-edition-of-euclids-elements} \end{small} \end{frame}\begin{frame} \frametitle{Pythagoras's theorem} \begin{figure} \centering \includegraphics[width=.72\textwidth]{pythagoras-theorem.pdf} \end{figure} \end{frame}\begin{frame} \frametitle{Question: can you spot $R^2$ on these graphs?} \begin{columns} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (0.5,1.5,2){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[below right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat) node[midway, above, color=black] {$\hat{\V{\epsilon}}$}; \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \tkzFindAngle(yhat,O,y) \tkzGetAngle{angleyhatOy}; \tkzDrawArc[R with nodes, color=red](O,1cm)(yhat,y) \tkzLabelAngle[above, pos=-0.1](yhat,O,y){$\theta$} \end{tikzpicture} \end{column} \begin{column}{0.5\textwidth} \begin{tikzpicture}[scale=0.7, every node/.style={scale=0.7}, tdplot_main_coords, axis/.style={->,dashed}, thick, rotate around x=15, rotate around z=-3] % -- remove these 3 lines if no axis is preferred \draw[axis] (0, 0, 0) -- (3, 0, 0) node [below] {1st coordinate}; \draw[axis] (0, 0, 0) -- (0, 4, 0) node [below left] {2nd coordinate}; \draw[axis] (0, 0, 0) -- (0, 0, 5) node [above] {3rd coordinate}; \coordinate (O) at (0,0,0){}; \coordinate (X) at (1,3,4){}; \coordinate (X1) at (1,0,0){}; \coordinate (X2) at (0,3,0){}; \coordinate (X3) at (0,0,4){}; \coordinate (X12) at (1,3,0){}; \coordinate (X13) at (1,0,4){}; \coordinate (X23) at (0,3,4){}; \coordinate (X0) at (1,1,1){}; \coordinate (X01) at (1,0,0){}; \coordinate (X02) at (0,1,0){}; \coordinate (X03) at (0,0,1){}; \coordinate (X012) at (1,1,0){}; \coordinate (X013) at (1,0,1){}; \coordinate (X023) at (0,1,1){}; \coordinate (y) at (2,1,2){}; \coordinate (y1) at (2,0,0){}; \coordinate (y2) at (0,1,0){}; \coordinate (y3) at (0,0,2){}; \coordinate (y12) at (2,1,0){}; \coordinate (y13) at (2,0,2){}; \coordinate (y23) at (0,1,2){}; \coordinate (yhat) at (1.7857,1.6428,1.5714){}; \draw[fill=black] (O) circle (0.2em) node[below] {$0$}; \draw[color=blue, thick] (O)--(X); \draw[color=blue, dashed, thin] (X1)--(X12); \draw[color=blue, dashed, thin] (X1)--(X13); \draw[color=blue, dashed, thin] (X2)--(X12); \draw[color=blue, dashed, thin] (X2)--(X23); \draw[color=blue, dashed, thin] (X3)--(X13); \draw[color=blue, dashed, thin] (X3)--(X23); \draw[color=blue, dashed, thin] (X12)--(X); \draw[color=blue, dashed, thin] (X13)--(X); \draw[color=blue, dashed, thin] (X23)--(X); \draw[fill=black] (X) circle (0.2em) node[below right] {$\V{X}_{:,1}$}; \draw[color=blue, thick] (O)--(X0); \draw[color=blue, dashed, thin] (X01)--(X012); \draw[color=blue, dashed, thin] (X01)--(X013); \draw[color=blue, dashed, thin] (X02)--(X012); \draw[color=blue, dashed, thin] (X02)--(X023); \draw[color=blue, dashed, thin] (X03)--(X013); \draw[color=blue, dashed, thin] (X03)--(X023); \draw[color=blue, dashed, thin] (X012)--(X0); \draw[color=blue, dashed, thin] (X013)--(X0); \draw[color=blue, dashed, thin] (X023)--(X0); \draw[fill=black] (X0) circle (0.2em) node[below right] {$\V{X}_{:,0}$}; \draw[color=green, thick] (O)--(y); \draw[color=green, dashed, thin] (y1)--(y12); \draw[color=green, dashed, thin] (y1)--(y13); \draw[color=green, dashed, thin] (y2)--(y12); \draw[color=green, dashed, thin] (y2)--(y23); \draw[color=green, dashed, thin] (y3)--(y13); \draw[color=green, dashed, thin] (y3)--(y23); \draw[color=green, dashed, thin] (y12)--(y); \draw[color=green, dashed, thin] (y13)--(y); \draw[color=green, dashed, thin] (y23)--(y); \draw[fill=black] (y) circle (0.2em) node[above left] {$\V{y}$}; \draw[dashed, color=green] (O)--(yhat); \draw[fill=black] (yhat) circle (0.2em) node[above right] {$\hat{\V{y}}$}; \draw[dashed, color=red] (y)--(yhat) node[midway, above, color=black] {$\hat{\V{\epsilon}}$}; \coordinate (aa) at (3,1,0){}; \filldraw[color=blue, fill opacity=0.3, dashed, thin] (O)--(aa)--(X); \tkzMarkRightAngle[draw=black, thin](O,yhat,y) \end{tikzpicture} \end{column} \end{columns} \end{frame}\begin{frame} \frametitle{Coefficient of determination ($R^2$)} \begin{itemize} \item Recall that, in a right triangle, \begin{equation*} \cos(\theta) = \frac{|\text{adjacent}|}{|\text{hypotenuse}|}. \end{equation*} \item Thus \begin{equation*} (\cos \theta)^2 = \frac{\norm{\hat{\V{y}}}^2}{\norm{\V{y}}^2} = \frac{\text{ESS}}{\text{TSS}} = \frac{\text{TSS} - \text{RSS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}} \eqdef R^2. \end{equation*} \item This quantity is also known as the \defn{coefficient of determination}. \item It indicates how much better the function predicts the dependent variable than the default predictor in the absence of any inputs --- the mean value of the output. \item It is also sometimes described as the \defn{proportion of variance explained}. \end{itemize} \end{frame}\begin{frame} \frametitle{$R^2$ never decreases (increases) when features are added (removed)} \begin{itemize} \item Regression can be seen as an orthogonal projection of $\V{y}$ onto the column space of $\M{X}$. \item An orthogonal projection minimizes the length of the vector that represents the difference between $\V{y}$ and the fitted values, $\hat{\V{y}}$, i.e. minimizes $\V{\epsilon}$. \item The fitted values are a vector in the column space of $\M{X}$. \item If you add a column to $\M{X}$, the column space of $\M{X}$ either stays the same or gets bigger. \item Therefore, the orthogonal projection onto this new space cannot possibly be longer since the original column space is a subspace of the new space. \end{itemize} \end{frame}\begin{frame} \frametitle{Theory and practice} \begin{itemize} \item So far we have been careful to address an academic economist's very valid concern: \begin{quote} That works very well in practice, but does it work in theory? \end{quote} \item This said, \begin{quote} The difference between theory and practice is considerably greater in practice than in theory. \end{quote} \item OK, it's time for some practice! \end{itemize} \end{frame}