This is a very quick run-through of some basic statistical concepts, adapted from Lab 4 in Harvard's CS109 course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

- Linear Regression Models
- Prediction using linear regression
- Some re-sampling methods
- Train-Test splits
- Cross Validation

Linear regression is used to model and predict continuous outcomes while logistic regression is used to model binary outcomes. We'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`

, `seaborn`

, and `scikit-learn`

. While we don't explicitly teach `statsmodels`

and `seaborn`

in the Springboard workshop, those are great libraries to know.

```
In [39]:
```# special IPython command to prepare the notebook for matplotlib and other libraries
%pylab inline
import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn
import seaborn as sns
# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```
```

Given a dataset $X$ and $Y$, linear regression can be used to:

- Build a
**predictive model**to predict future values of $X_i$ without a $Y$ value. - Model the
**strength of the relationship**between each dependent variable $X_i$ and $Y$ - Sometimes not all $X_i$ will have a relationship with $Y$
- Need to figure out which $X_i$ contributes most information to determine $Y$
- Linear regression is used in so many applications that I won't warrant this with examples. It is in many cases, the first pass prediction algorithm for continuous outcomes.

Linear Regression is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$. This method assumes the relationship between each predictor $X$ is linearly related to the dependent variable $Y$.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. This is the simplest form of linear regression (one variable), we'll call this the simple model.

$\beta_0$ is the intercept of the linear model

Multiple linear regression is when you have more than one independent variable

- $X_1$, $X_2$, $X_3$, $\ldots$

- Back to the simple model. The model in linear regression is the
*conditional mean*of $Y$ given the values in $X$ is expressed a linear function.

- The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

- Once you estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, you can use these to predict new values of $Y$

- How do you estimate the coefficients?
- There are many ways to fit a linear regression model
- The method called
**least squares**is one of the most common methods - We will discuss least squares today

Least squares is a method that can estimate the coefficients of a linear model by minimizing the difference between the following:

$$ S = \sum_{i=1}^N r_i = \sum_{i=1}^N (y_i - (\beta_0 + \beta_1 x_i))^2 $$where $N$ is the number of observations.

- We will not go into the mathematical details, but the least squares estimates $\hat{\beta}_0$ and $\hat{\beta}_1$ minimize the sum of the squared residuals $r_i = y_i - (\beta_0 + \beta_1 x_i)$ in the model (i.e. makes the difference between the observed $y_i$ and linear model $\beta_0 + \beta_1 x_i$ as small as possible).

The solution can be written in compact matrix notation as

$$\hat\beta = (X^T X)^{-1}X^T Y$$We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

**Note**: The "hat" means it is an estimate of the coefficient.

The Boston Housing data set contains information about the housing values in suburbs of Boston. This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

`sklearn`

This data set is available in the sklearn python module which is how we will access it today.

```
In [40]:
```from sklearn.datasets import load_boston
boston = load_boston()

```
In [41]:
```boston.keys()

```
Out[41]:
```

```
In [42]:
```boston.data.shape

```
Out[42]:
```

```
In [43]:
```# Print column names
print (boston.feature_names)

```
```

```
In [44]:
```# Print description of Boston housing data set
print (boston.DESCR)

```
```

Now let's explore the data set itself.

```
In [45]:
```bos = pd.DataFrame(boston.data)
bos.head()

```
Out[45]:
```

There are no column names in the DataFrame. Let's add those.

```
In [46]:
```bos.columns = boston.feature_names
bos.head()

```
Out[46]:
```

`bos`

containing all the data we want to use to predict Boston Housing prices. Let's create a variable called `PRICE`

which will contain the prices. This information is contained in the `target`

data.

```
In [47]:
```print (boston.target.shape)

```
```

```
In [48]:
```bos['PRICE'] = boston.target
bos.head()

```
Out[48]:
```

```
In [49]:
```bos.describe()

```
Out[49]:
```

```
In [50]:
```plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")

```
Out[50]:
```

**Your turn**: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. What do you notice?

```
In [51]:
```#your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("average number of rooms per dwelling (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")

```
Out[51]:
```

```
In [52]:
```#your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("pupil-teacher ratio by town (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")

```
Out[52]:
```

**Your turn**: What are some other numeric variables of interest? Plot scatter plots with these variables and *PRICE*.

```
In [53]:
```#your turn: create some other scatter plots
plt.scatter(bos.AGE, bos.PRICE)
plt.xlabel("proportion of owner-occupied units built prior to 1940 (AGE)")
plt.ylabel("Housing Price")
plt.title("Relationship between House Ages and Price")

```
Out[53]:
```

Seaborn is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use seaborn regplot for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```
In [54]:
```sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)

```
Out[54]:
```

```
In [55]:
```plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequency")
plt.show()

```
```

**Your turn**: Plot separate histograms and one for *RM*, one for *PTRATIO*. Any interesting observations?

```
In [56]:
```#your turn
plt.hist(bos.RM)
plt.title("RM")
plt.xlabel("average number of rooms per dwelling")
plt.ylabel("Frequency")
plt.show()

```
```

```
In [57]:
```# Histogram for pupil-teacher ratio by town
plt.hist(bos.PTRATIO)
plt.title("PTRATIO")
plt.xlabel("pupil-teacher ratio by town")
plt.ylabel("Frequency")
plt.show()

```
```

Here,

$Y$ = boston housing prices (also called "target" data in python)

and

$X$ = all the other features (or independent variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least squares method as the way to estimate the coefficients.

`statsmodels`

Statsmodels is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive, it's a good library to have in your toolbox. Here's a quick example of what you could do with it.

```
In [58]:
```# Import regression modules
# ols - stands for Ordinary least squares, we'll use this
import statsmodels.api as sm
from statsmodels.formula.api import ols

```
In [59]:
```# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print (m.summary())

```
```

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM`

coefficient (9.1021) by first noticing that the p-value (under `P>|t|`

) is so small, basically zero. We can interpret the coefficient as, if we compare two groups of towns, one where the average number of rooms is say $5$ and the other group is the same except that they all have $6$ rooms. For these two groups the average difference in house prices is about $9.1$ (in thousands) so about $\$9,100$ difference. The confidence interval fives us a range of plausible values for this difference, about ($\$8,279, \$9,925$), deffinitely not chump change.

`statsmodels`

formulasThis formula notation will seem familiar to `R`

users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels`

(`ols`

or `logit`

) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`

, `X1`

,`X2`

, ...), bbut you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex, for a quick run-down of further uses see the `statsmodels`

help page.

**Your turn:** Create a scatterpot between the predicted prices, available in `m.fittedvalues`

and the original prices. How does the plot look?

```
In [73]:
```# your turn
plt.scatter(bos.PRICE, m.fittedvalues)
plt.xlabel("Housing Price")
plt.ylabel("Predicted Housing Price")
plt.title("Relationship between Predicted and Actual Price")

```
Out[73]:
```

```
In [61]:
```from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)
# This creates a LinearRegression object
lm = LinearRegression()
lm

```
Out[61]:
```

Check out the scikit-learn docs here. We have listed the main functions here.

Main functions | Description |
---|---|

`lm.fit()` |
Fit a linear model |

`lm.predit()` |
Predict Y using the linear model with estimated coefficients |

`lm.score()` |
Returns the coefficient of determination (R^2). A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model |

```
In [62]:
```# Look inside lm object
#lm.<tab>

Output | Description |
---|---|

`lm.coef_` |
Estimated coefficients |

`lm.intercept_` |
Estimated intercept |

```
In [63]:
```# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)

```
Out[63]:
```

**Your turn:** How would you change the model to not fit an intercept term? Would you recommend not having an intercept?

Let's look at the estimated coefficients from the linear model using `1m.intercept_`

and `lm.coef_`

.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$```
In [64]:
```print ('Estimated intercept coefficient:', lm.intercept_)

```
```

```
In [65]:
```print ('Number of coefficients:', len(lm.coef_))

```
```

```
In [67]:
```# The coefficients
pd.DataFrame(list(zip(X.columns, lm.coef_)), columns = ['features', 'estimatedCoefficients'])

```
Out[67]:
```

```
In [68]:
```# first five predicted prices
lm.predict(X)[0:5]

```
Out[68]:
```

**Your turn:**

- Histogram: Plot a histogram of all the predicted prices
- Scatter Plot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with
`statsmodels`

before).

```
In [74]:
```# your turn
# Plot a histogram of all the predicted prices
plt.hist(lm.predict(X))
plt.title("Predicted Prices")
plt.xlabel("Predicted Prices")
plt.ylabel("Frequency")
plt.show()
# Let's plot the true prices compared to the predicted prices to see they disagree
plt.scatter(bos.PRICE, lm.predict(X))
plt.xlabel("Housing Price")
plt.ylabel("Predicted Housing Price")
plt.title("Relationship between Predicted and Actual Price")

```
Out[74]:
```

```
In [75]:
```print (np.sum((bos.PRICE - lm.predict(X)) ** 2))

```
```

```
In [80]:
```#your turn
print ('Mean squared error: ', np.mean((bos.PRICE - lm.predict(X)) ** 2))

```
```

```
In [81]:
```lm = LinearRegression()
lm.fit(X[['PTRATIO']], bos.PRICE)

```
Out[81]:
```

```
In [82]:
```msePTRATIO = np.mean((bos.PRICE - lm.predict(X[['PTRATIO']])) ** 2)
print (msePTRATIO)

```
```

We can also plot the fitted linear regression line.

```
In [83]:
```plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-to-Teacher Ratio (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")
plt.plot(bos.PTRATIO, lm.predict(X[['PTRATIO']]), color='blue', linewidth=3)
plt.show()

```
```

```
In [101]:
```# your turn
lm.fit(X[['CRIM']], bos.PRICE)
print ('(MSE) Per capita crime rate by town: ', np.mean((bos.PRICE - lm.predict(X[['CRIM']])) ** 2))
lm.fit(X[['RM']], bos.PRICE)
print ('(MSE) Average number of rooms per dwelling: ', np.mean((bos.PRICE - lm.predict(X[['RM']])) ** 2))
lm.fit(X[['PTRATIO']], bos.PRICE)
print ('(MSE) Pupil-teacher ratio by town: ', np.mean((bos.PRICE - lm.predict(X[['PTRATIO']])) ** 2))

```
```

- **Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$.
- **Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X.
- e.g. PTRATIO

- **Normal distribution for errors**. The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. $$ \epsilon_i \sim N(0, \sigma^2)$$ Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$. e.g. skewed positively or negatively.
- **Independent errors**. The observations are assumed to be obtained independently.
- e.g. Observations across time may be correlated

```
In [123]:
```sns.set(font_scale=.8)
sns.heatmap(X.corr(), vmax=.8, square=True, annot=True)

```
Out[123]:
```

Let's stick to the linear regression example:

- We built our model with the requirement that the model fit the data well.
- As a side-effect, the model will fit
**THIS**dataset well. What about new data? - We wanted the model for predictions, right?
- One simple solution, leave out some data (for
**testing**) and**train**the model on the rest - This also leads directly to the idea of cross-validation, next section.

One way of doing this is you can create training and testing data sets manually.

```
In [85]:
```X_train = X[:-50]
X_test = X[-50:]
Y_train = bos.PRICE[:-50]
Y_test = bos.PRICE[-50:]
print (X_train.shape)
print (X_test.shape)
print (Y_train.shape)
print (Y_test.shape)

```
```

`train_test_split`

in `sklearn.cross_validation`

. Here's the documentation.

```
In [86]:
```X_train, X_test, Y_train, Y_test = sklearn.cross_validation.train_test_split(
X, bos.PRICE, test_size=0.33, random_state = 5)
print (X_train.shape)
print (X_test.shape)
print (Y_train.shape)
print (Y_test.shape)

```
```

**Your turn:** Let's build a linear regression model using our new training data sets.

- Fit a linear regression model to the training set
- Predict the output on the test set

```
In [105]:
```# your turn
# Fit a linear regression model to the training set
lm.fit(X_train, Y_train)
lm.predict(X_test)

```
Out[105]:
```

**Your turn:**

Calculate the mean squared error

- using just the test data
- using just the training data

Are they pretty similar or very different? What does that mean?

```
In [107]:
```# your turn
# Calculate MSE using just the test data
print ('(MSE) using just the test data: ', np.mean((Y_test - lm.predict(X_test)) ** 2))
# Calculate MSE using just the training data
print ('(MSE) using just the training data: ', np.mean((Y_train - lm.predict(X_train)) ** 2))

```
```

```
In [108]:
```plt.scatter(lm.predict(X_train), lm.predict(X_train) - Y_train, c='b', s=40, alpha=0.5)
plt.scatter(lm.predict(X_test), lm.predict(X_test) - Y_test, c='g', s=40)
plt.hlines(y = 0, xmin=0, xmax = 50)
plt.title('Residual Plot using training (blue) and test (green) data')
plt.ylabel('Residuals')

```
Out[108]:
```

**Your turn:** Do you think this linear regression model generalizes well on the test data?
-> ** No, the scatter points are not close to zero so the model needs improvements. Check the features to see highly correlated predictors and remove one of them or check the parameters of the model and do fine-tuning.**

A simple extension of the Test/train split is called K-fold cross-validation.

Here's the procedure:

- randomly assign your $n$ samples to one of $K$ groups. They'll each have about $n/k$ samples
- For each group $k$:
- Fit the model (e.g. run regression) on all data excluding the $k^{th}$ group
- Use the model to predict the outcomes in group $k$
- Calculate your prediction error for each observation in $k^{th}$ group (e.g. $(Y_i - \hat{Y}_i)^2$ for regression, $\mathbb{1}(Y_i = \hat{Y}_i)$ for logistic regression).
- Calculate the average prediction error across all samples $Err_{CV} = \frac{1}{n}\sum_{i=1}^n (Y_i - \hat{Y}_i)^2$

Luckily you don't have to do this entire process all by hand (`for`

loops, etc.) every single time, `sci-kit learn`

has a very nice implementation of this, have a look at the documentation.

**Your turn (extra credit):** Implement K-Fold cross-validation using the procedure above and Boston Housing data set using $K=4$. How does the average prediction error compare to the train-test split above?

```
In [127]:
```from sklearn import cross_validation, linear_model
# If the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used.
# In all other cases, KFold is used
scores = cross_validation.cross_val_score(lm, X, bos.PRICE, scoring='mean_squared_error', cv=4)
# This will print metric for evaluation
print ('(MSE) Using k-fold: ', np.mean(scores))
print ('The K-fold cross-validation is not performaing well compared to the previous train-test split above')

```
```