In this notebook, we will run ridge regression multiple times with different L2 penalties to see which one produces the best fit. We will revisit the example of polynomial regression as a means to see the effect of L2 regularization. In particular, we will:

- Use a pre-built implementation of regression (GraphLab Create) to run polynomial regression
- Use matplotlib to visualize polynomial regressions
- Use a pre-built implementation of regression (GraphLab Create) to run polynomial regression, this time with L2 penalty
- Use matplotlib to visualize polynomial regressions under L2 regularization
- Choose best L2 penalty using cross-validation.
- Assess the final fit using test data.

We will continue to use the House data from previous notebooks. (In the next programming assignment for this module, you will implement your own ridge regression learning algorithm using gradient descent.)

```
In [1]:
```import graphlab

`polynomial_sframe`

from Week 3:

```
In [2]:
```def polynomial_sframe(feature, degree):
# assume that degree >= 1
# initialize the SFrame:
poly_sframe = graphlab.SFrame()
# and set poly_sframe['power_1'] equal to the passed feature
poly_sframe['power_1'] = feature
# first check if degree > 1
if degree > 1:
# then loop over the remaining degrees:
# range usually starts at 0 and stops at the endpoint-1. We want it to start at 2 and stop at degree
for power in range(2, degree+1):
# first we'll give the column a name:
name = 'power_' + str(power)
name_left = 'power_' + str(power-1)
# then assign poly_sframe[name] to the appropriate power of feature
poly_sframe[name] = feature * poly_sframe[name_left]
return poly_sframe

Let's use matplotlib to visualize what a polynomial regression looks like on the house data.

```
In [3]:
```import matplotlib.pyplot as plt
%matplotlib inline

```
```

```
In [4]:
```sales = graphlab.SFrame('kc_house_data.gl/')

```
In [5]:
```sales = sales.sort(['sqft_living','price'])

`polynomial_sframe()`

and fit a model with these features. When fitting the model, use an L2 penalty of `1e-5`

:

```
In [6]:
```l2_small_penalty = 1e-5

Note: When we have so many features and so few data points, the solution can become highly numerically unstable, which can sometimes lead to strange unpredictable results. Thus, rather than using no regularization, we will introduce a tiny amount of regularization (`l2_penalty=1e-5`

) to make the solution numerically stable. (In lecture, we discussed the fact that regularization can also help with numerical stability, and here we are seeing a practical example.)

With the L2 penalty specified above, fit the model and print out the learned weights.

Hint: make sure to add 'price' column to the new SFrame before calling `graphlab.linear_regression.create()`

. Also, make sure GraphLab Create doesn't create its own validation set by using the option `validation_set=None`

in this call.

```
In [8]:
```poly_sframe = polynomial_sframe(sales['sqft_living'], 15)
my_features = poly_sframe.column_names()
poly_sframe['price'] = sales['price']
model = graphlab.linear_regression.create(poly_sframe, 'price', features=my_features,
validation_set=None, l2_penalty=1e-5)

```
```

```
In [12]:
```model.get('coefficients')

```
Out[12]:
```

*QUIZ QUESTION: What's the learned value for the coefficient of feature power_1?*

*high variance*. We will see in a moment that ridge regression reduces such variance. But first, we must reproduce the experiment we did in Week 3.

`set_1`

, `set_2`

, `set_3`

, and `set_4`

. Use `.random_split`

function and make sure you set `seed=0`

.

```
In [13]:
```(semi_split1, semi_split2) = sales.random_split(.5,seed=0)
(set_1, set_2) = semi_split1.random_split(0.5, seed=0)
(set_3, set_4) = semi_split2.random_split(0.5, seed=0)

Next, fit a 15th degree polynomial on `set_1`

, `set_2`

, `set_3`

, and `set_4`

, using 'sqft_living' to predict prices. Print the weights and make a plot of the resulting model.

Hint: When calling `graphlab.linear_regression.create()`

, use the same L2 penalty as before (i.e. `l2_small_penalty`

). Also, make sure GraphLab Create doesn't create its own validation set by using the option `validation_set = None`

in this call.

```
In [17]:
```def fitAndPlot(data, degree, l2):
poly_data = polynomial_sframe(data['sqft_living'], degree)
my_features = poly_data.column_names() # get the name of the features
poly_data['price'] = data['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', l2_penalty=l2, verbose=False,
features = my_features, validation_set = None)
plt.plot(poly_data['power_1'],poly_data['price'],'.',
poly_data['power_1'], model.predict(poly_data),'-')
model.get("coefficients").print_rows(num_rows = 16)

```
In [18]:
```for data in [set_1, set_2, set_3, set_4]:
fitAndPlot(data, 15, l2_small_penalty)

```
```

```
In [ ]:
```power_1 | None | 1247.59037346 | 7944.94142547
power_1 | None | -759.251889293 | 7591.2364892
power_1 | None | 783.493762459 | nan
power_1 | None | 585.865810528 | 2868.03758336

The four curves should differ from one another a lot, as should the coefficients you learned.

** QUIZ QUESTION: For the models learned in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature ** (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)

`power_1`

?Generally, whenever we see weights change so much in response to change in data, we believe the variance of our estimate to be large. Ridge regression aims to address this issue by penalizing "large" weights. (Weights of `model15`

looked quite small, but they are not that small because 'sqft_living' input is in the order of thousands.)

With the argument `l2_penalty=1e5`

, fit a 15th-order polynomial model on `set_1`

, `set_2`

, `set_3`

, and `set_4`

. Other than the change in the `l2_penalty`

parameter, the code should be the same as the experiment above. Also, make sure GraphLab Create doesn't create its own validation set by using the option `validation_set = None`

in this call.

```
In [19]:
```for data in [set_1, set_2, set_3, set_4]:
fitAndPlot(data, 15, 1e5)

```
```

These curves should vary a lot less, now that you applied a high degree of regularization.

** QUIZ QUESTION: For the models learned with the high level of regularization in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature ** (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)

`power_1`

?Just like the polynomial degree, the L2 penalty is a "magic" parameter we need to select. We could use the validation set approach as we did in the last module, but that approach has a major disadvantage: it leaves fewer observations available for training. **Cross-validation** seeks to overcome this issue by using all of the training set in a smart way.

We will implement a kind of cross-validation called **k-fold cross-validation**. The method gets its name because it involves dividing the training set into k segments of roughtly equal size. Similar to the validation set method, we measure the validation error with one of the segments designated as the validation set. The major difference is that we repeat the process k times as follows:

Set aside segment 0 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

Set aside segment 1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

...

Set aside segment k-1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

After this process, we compute the average of the k validation errors, and use it as an estimate of the generalization error. Notice that all observations are used for both training and validation, as we iterate over segments of data.

To estimate the generalization error well, it is crucial to shuffle the training data before dividing them into segments. GraphLab Create has a utility function for shuffling a given SFrame. We reserve 10% of the data as the test set and shuffle the remainder. (Make sure to use `seed=1`

to get consistent answer.)

```
In [20]:
```(train_valid, test) = sales.random_split(.9, seed=1)
train_valid_shuffled = graphlab.toolkits.cross_validation.shuffle(train_valid, random_seed=1)

`n/k`

elements, where `n`

is the number of observations in the training set and `k`

is the number of segments. Since the segment 0 starts at index 0 and contains `n/k`

elements, it ends at index `(n/k)-1`

. The segment 1 starts where the segment 0 left off, at index `(n/k)`

. With `n/k`

elements, the segment 1 ends at index `(n*2/k)-1`

. Continuing in this fashion, we deduce that the segment `i`

starts at index `(n*i/k)`

and ends at `(n*(i+1)/k)-1`

.

```
In [21]:
```n = len(train_valid_shuffled)
k = 10 # 10-fold cross-validation
for i in xrange(k):
start = (n*i)/k
end = (n*(i+1))/k-1
print i, (start, end)

```
```

`train_valid_shuffled`

. Notice that the first index (0) is included in the slice but the last index (10) is omitted.

```
In [22]:
```train_valid_shuffled[0:10] # rows 0 to 9

```
Out[22]:
```

`train_valid_shuffled`

dataframe into k=10 segments of roughly equal size, with starting and ending indices computed as above.
Extract the fourth segment (segment 3) and assign it to a variable called `validation4`

.

```
In [25]:
```validation4 = train_valid_shuffled[5818:7758]

```
In [26]:
```print int(round(validation4['price'].mean(), 0))

```
```

`append()`

method that pastes together two disjoint sets of rows originating from a common dataset. For instance, the following cell pastes together the first and last two rows of the `train_valid_shuffled`

dataframe.

```
In [27]:
```n = len(train_valid_shuffled)
first_two = train_valid_shuffled[0:2]
last_two = train_valid_shuffled[n-2:n]
print first_two.append(last_two)

```
```

*excluding* fourth segment (segment 3) and assign the subset to `train4`

.

```
In [29]:
```train4 = train_valid_shuffled[0:5818].append(train_valid_shuffled[7758:])

```
In [30]:
```print int(round(train4['price'].mean(), 0))

```
```

Now we are ready to implement k-fold cross-validation. Write a function that computes k validation errors by designating each of the k segments as the validation set. It accepts as parameters (i) `k`

, (ii) `l2_penalty`

, (iii) dataframe, (iv) name of output column (e.g. `price`

) and (v) list of feature names. The function returns the average validation error using k segments as validation sets.

- For each i in [0, 1, ..., k-1]:
- Compute starting and ending indices of segment i and call 'start' and 'end'
- Form validation set by taking a slice (start:end+1) from the data.
- Form training set by appending slice (end+1:n) to the end of slice (0:start).
- Train a linear model using training set just formed, with a given l2_penalty
- Compute validation error using validation set just formed

```
In [42]:
```def k_fold_cross_validation(k, l2_penalty, data, output_name, features_list):
n = len(data)
RSS_total = 0.
for i in xrange(k):
start = (n*i)/k
end = (n*(i+1))/k
validation = data[start:end]
train = data[:start].append(data[end:])
model = graphlab.linear_regression.create(train, target=output_name, l2_penalty=l2_penalty,
verbose=False, features=features_list, validation_set=None)
predictions = model.predict(validation)
errors = predictions - validation[output_name]
RSS = (errors * errors).sum()
RSS_total = RSS_total + RSS
return RSS_total / k

Once we have a function to compute the average validation error for a model, we can write a loop to find the model that minimizes the average validation error. Write a loop that does the following:

- We will again be aiming to fit a 15th-order polynomial model using the
`sqft_living`

input - For
`l2_penalty`

in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, you can use this Numpy function:`np.logspace(1, 7, num=13)`

.)- Run 10-fold cross-validation with
`l2_penalty`

- Run 10-fold cross-validation with
- Report which L2 penalty produced the lowest average validation error.

Note: since the degree of the polynomial is now fixed to 15, to make things faster, you should generate polynomial features in advance and re-use them throughout the loop. Make sure to use `train_valid_shuffled`

when generating polynomial features!

```
In [40]:
```train_data = polynomial_sframe(train_valid_shuffled['sqft_living'], 15)
feature_list = train_data.column_names()
train_data['price'] = train_valid_shuffled['price']

```
In [37]:
```import numpy as np

```
In [46]:
```validation_error = float('inf')
l2_min = 0
errors = []
l2_list = np.logspace(1, 7, num=13)
for l2_penalty in l2_list:
new_error = k_fold_cross_validation(10, l2_penalty, train_data, 'price', feature_list)
errors.append(new_error)
if new_error < validation_error:
l2_min = l2_penalty
validation_error = new_error
print l2_min

```
```

*QUIZ QUESTIONS: What is the best value for the L2 penalty according to 10-fold validation?*

```
In [49]:
```# Plot the l2_penalty values in the x axis and the cross-validation error in the y axis.
# Using plt.xscale('log') will make your plot more intuitive.
plt.xscale('log')
plt.plot(np.logspace(1, 7, num=13), errors)

```
Out[49]:
```

`l2_penalty`

. This way, your final model will be trained on the entire dataset.

```
In [51]:
```final_model = graphlab.linear_regression.create(train_data, target='price', l2_penalty=1000,
verbose=False, features=feature_list, validation_set=None)

```
In [52]:
```test_data = polynomial_sframe(test['sqft_living'], 15)
test_data['price'] = test['price']
predictions = final_model.predict(test_data)

```
In [53]:
```test_error = predictions - test['price']

```
In [54]:
```RSS = (test_error * test_error).sum()

```
In [55]:
``````
RSS
```

```
Out[55]:
```

```
In [ ]:
```