Machine Learning 101++ in Python

by Pieter Buteneers (@pieterbuteneers) and Bart De Vylder from CoScale

1. Imports

Let's first start with importing all the necessary packages. Some imports will be repeated in the exercises but if you want to skip some parts you can just execute the imports below and start with any exercise.

As you can see we also import packages from __future__. This is to improve the compatibility with Python 3, but will not guarantee it.

2. Linear Regression

Linear Regression assumes a linear realationship between 2 variables.

As an example we'll consider the historical page views of a web server and compare it to its CPU usage. We'll try to predict the CPU usage of the server based on the page views of the different pages.

2.1 Data import and inspection

Let's import the data and take a look at it.

The orange line on the plot above is the number of page views and the blue line is the CPU load that viewing this pages generates on the server.

2.2 Simple linear regression

First, we're going to work with the total page views on the server, and compare it to the CPU usage. We can make use of a PyPlot's scatter plot to understand the relation between the total page views and the CPU usage:

There clearly is a strong correlation between the page views and the CPU usage. Because of this correlation we can build a model to predict the CPU usage from the total page views. If we use a linear model we get a formula like the following:

$$ \text{cpu_usage} = c_0 + c_1 \text{total_page_views} $$

Since we don't know the exact values for $c_0$ and $c_1$ we will have to compute them. For that we'll make use of the scikit-learn machine learning library for Python and use least-squares linear regression

Now we need to feed the data to the model to fit it. The model.fit(X,y) method in general takes a matrix X and vector y as arguments:

      X = [[x_11, x_12, x_13, ...],                  y = [y_1,
           [x_21, x_22, x_23, ...],                       y_2,  
           [x_31, x_32, x_33, ...],                       y_3,
           ...]                                           ...]

and tries to find coefficients that allow to predict the y_i's from the x_ij's. In our case the matrix X will consist of only 1 column containing the total page views. Our total_page_views variable however, is still only a one-dimensional vector, so we need to np.reshape() it into a two-dimensional array. Since there is only 1 feature the second dimension should be 1.

Then we fit our model using the the total page views and cpu. The coefficients found are automatically stored in the simple_lin_model object.

We can now inspect the coefficient $c_1$ and constant term (intercept) $c_0$ of the model:

So this means that each additional page view adds about 0.11% CPU load to the server and all the other processes running on the server consume on average 0.72% CPU.

Once the model is trained we can use it to predict the outcome for a given input (or array of inputs). Note that the predict function requires a 2-dimensional array similar to the fit function.

What is the expected CPU usage when we have 880 page views per second?

Now we plot the linear model together with our data to verify it captures the relationship correctly (the predict method can accept the entire total_page_views array at once).

Our model can calculate the R2 score indicating how well the linear model captures the data. A score of 1 means there is perfect linear correlation and the model can fit the data perfectly, a score of 0 (or lower) means that there is no correlation at all (and it does not make sense to try to model it that way). The score method takes the same arguments as the fit method.

2.3 Extrapolation

Now let's repeat this experiment with different but similar data. We will try to predict what the CPU usage will be if there will be 8 page views (per second).

Now let's plot what you have done.

Is this what you would expect? Can you see what's wrong?

Let's plot the time series again to get a different view at the data.

The spikes of CPU usage are actually backups that run at night and they can be ignored. So repeat the exersize again but ignore these data points.

Hint: The selection variable should contain True where there is no backup going on and False when the backup occurs. This is an easy shortcut to do a selection of specific data points in numpy arrays.

So what you should have learned from the previous exercise is that you should always look at your data and/or write scripts to inspect your data. Additinally extrapolation does not always work because there are no training examples in that area.

3. Multiple linear regression

A server can host different pages and each of the page views will generate load on the CPU. This load will however not be the same for each page.

Now let us consider the separate page views and build a linear model for that. The model we try to fit takes the form:

$$\text{cpu_usage} = c_0 + c_1 \text{page_views}_1 + c_2 \text{page_views}_2 + \ldots + c_n \text{page_views}_n$$

where the $\text{page_views}_i$'s correspond the our different pages:

We start again by creating a LinearRegression model.

Next we fit the model on the data, using multi_lin_model.fit(X,y). In contrast to the case above our page_views variable already has the correct shape to pass as the X matrix: it has one column per page.

Now, given the coefficients calculated by the model, which capture the contribution of each page view to the total CPU usage, we can start to answer some interesting questions. For example, which page view causes most CPU usage, on a per visit basis?

For this we can generate a table of page names with their coefficients in descending order:

From this table we see that 'resources/js/basket.js' consumes the most per CPU per view. It generates about 0.30% CPU load for each additional page view. 'products/science.html' on the other hand is much leaner and only consumes about 0.04% CPU per view.

Now let us investigate the constant term again.

As you can see this term is very similar to the result achieved in single linear regression, but it is not entirely the same. This means that these models are not perfect. However, they seem to be able to give a reliable estimate.

4. Non-linear Regression

Sometimes linear relations don't cut it anymore, so you might want a more complex method. There are 2 approaches to this:

• Use a non-linear method (such as Neural Networks, Support Vector Machines, Random Forests and Gaussian Processes)
• Use non-linear features as pre-processing for a linear method

Actually both methods are in essence identical and there is not always a clear distinction between the two. We will use the second approach in this section since it is easier to understand what is going on.

Please note that it is very often not even necessary to use non-linear methods, since the linear methods can be extremely powerful on their own and they are quite often very stable and reliable (in contrast to non-linear methods).

4.1. Fitting a sine function with linear regression

Als an example task we will try to fit a sine function. We will use the np.sine() function to compute the sine of the elements in a numpy array.

Let's first try this with linear regression.

For training we will draw 10 samples of this function as our train set.

Now let's try to fit this function with linear regression.

As you can see this fit is not optimal.

4.2. Fitting a sine function using polynomial expansion

One of the easiest ways to make your machine learning technique more intelligent is to extract relevant features from the data. These features can be anything that you can find that will make it easier for the metod to be able to fit the data. This means that as a machine learning engineer it is best to know and understand your data.

As some of you might remember from math class is that you can create an approximation of any function (including a sine function) using a polynomial function with the Taylor expansion. So we will use that approach to learn a better fit.

In this case we will create what we call features using a polynomial expansion. If you set the degree to 3 it will generate data of the 0d, 1st, 2nd and 3rd order (including cross products) as shown in the example below (change x and degree to see the different expansions of x to a certain degree).

As you can see above this function transforms $x$ into [$x^0$, $x^1$, $x^2$, $x^3$] with $x^0=1$ and $x^1 = x$. If you have 2 inputs it will also take the cross products so that [$x_1$, $x_2$] is transformed into: [1, $x_1$, $x_2$, $x_1^2$, $x_1x_2$, $x_2^2$, $x_1^3$, $x_1^2x_2$, $x_1x_2^2$, $x_2^3$] as shown below.

In this example we only have 1 input so the number of features is always the degree + 1.

Because of this polynomial features extraction finding of the coefficients of the polynomial becomes a linear problem, so similar to the previous exercise on multiple linear regression you can find the optimal weights as follows:

$$y = c_0 + c_1 x + c_2 x^2 + c_3 x^3 + \cdots + c_n x^n$$

So for multiple values of $x$ and $y$ you can minimize the error of this equation using linear regression. How this is done in practice is shown below.

The more relevant these features are the better your model can fit the data.

Now play with the degree of the polynomal expansion function below to create better features. Search for the optimal degree.

Now let's test this on new and unseen data.

If everything is correct your score is very close to 1. Which means that we have built a model that can fit this data (almost) perfectly.

4.3. Add noise to the equation

Sadly all the data that we measure or gather doesn'thave the mathematical precision of the data we used here. Quite often our measurements contain noise.

So let us repeat this process for data with more noise. Similarly as above, you have to choose the optimal degree of the polynomials.

Now let's see what this results to in the test set.

As you can clearly see, this result is not that good. Why do you think this is?

Now plot the result to see the function you created.

Is this what you expect?

Now repeat the process below a couple of times for random noise.

What did you observe? And what is the method learning? And how can you avoid this?

Try to figure out a solution for this problem without changing the noise level.

Check your solution a couple of times to make sure your solution works for different noise samples.

5. Over-fitting and Cross-Validation

What you have experienced above is called over-fitting and happens when your model learns the noise that is inherrent in the data.

This problem was caused because there were to many parameters in the model. So the model was too advanced so that it became capable of learning the noise in the data by heart. Reducing the number of parameters solves this problem. But how do you know how many parameters is optimal?

(Another way to solve this problem is to use more data. Because if there are more data points in the data and if there is more noise, your model isn't able to learn all that noise anymore and you get a better result. Since it usually is not possible to gather more data we will not take this approach.)

In the exercise above you had to set the number of polynomial functions to get a better result, but how can you estimate this in a reliable way without manually selection the optimal parameters?

5.1. Validation set

A common way to solve this problem is through the use of a validation set. This means that you use a subset of the training data to train your model on, and another subset of the training data to validate your parameters. Based on the score of your model on this validation set you can select the optimal parameter.

So use this approach to select the best number of polynomials for the noisy sine function.

Now test this result on the test set with the following code.

As you can see this approach works to select het optimal degree. Usually the test score is lower than the validation score, but in this case it is not because the test data doesn't contain noise.

5.2. Cross-Validation

To improve this procedure you can repeat the process above for different train and validation sets so that the optimal parameter is less dependent on the way the data was selected.

One basic strategy for this is leave-one-out cross validation, where each data point is left out of the train set once, and the model is then validated on this point. Now let's implement this. First make a 2-dimensional array results to store all your results using the np.ones() function: 1 dimension (row) for each validation set and 1 dimension (column) for each degree of the PolynomialFeatures() function. Then you loop over all the validation sets followed by a loop over all the degrees of the PolynomialFeatures() function you want to try out. Then set the result for that experiment in the right element of the results array.

We will use the mean squared error (MSE) instead of R2 because that is more stable. Since the MSE measures the error, smaller values are better.

Now that you have your results average them over all validation sets (using the np.mean() function over the correct axis) so that you know the average error for each degree over all validation sets. Now find the the degree with the smallest error using the np.argmin() function.

Now let's have a look at the result.

As you can see this automatic way of selecting the optimal degree has resulted in a good fit for the sine function.

5.3 Regularisation

When you have too many parameters in your model, there is a risk of overfitting, i.e. your model learns the noise. To avoid this, techniques have been developed to make an estimation of this noise.

One of these techniques is Ridge Regression. This linear regression technique has an additional parameter called the regularisation parameter. This parameter basically sets the standard deviation of the noise you want to remove. The effect in practice is that it makes sure the weights of linear regression remain small and thus less over-fitting.

Since this is an additional parameter that needs to be set, it needs to be set using cross-validation as well. Luckily sklearn developed a method that does this for us in a computational efficient way called sklearn.linear_model.RidgeCV()

As you can see above, the result of Ridge Regression is not as good as reducing the number of features in this example. However it works a lot better than without regularisation (try that). In the example above you will notice that it makes the result a lot smoother and removes the unwanted spikes. It will actually make sure that if you have too many features you still get a reasonable result. So this means that it should be in your standard toolkit.

The removal of the extra features can be automated using feature selection. A very short introduction to sklearn on the topic can be found here.

Another method that is often used is sklearn.linear_model.LassoCV() which actually combines removal of features and estimation of the noise. It is however very dependent on the dataset which of the two methods performs best.

Cross-validation should be applied to any parameter you set in your function and that without looking at the test set.

Over-fitting is one of the biggest issues in machine learning and most of the research that is currently being done in machine learning is a search for techniques to avoid over-fitting. As a starting point we list a few of the techniques that you can use to avoid over-fitting:

• Use more data
• Artificially generate more data based on the original data
• Use a smaller model (with less parameters)
• Use less features (and thus less parameters)
• Use a regularisation parameter
• Artificially add noise to your model
• Only use linear models or make sure that the non-linearity in your model is closer to a linear function
• Combine multiple models that each over-fit in their own way into what is called an ensemble

5.4 Extrapolation

Now let's extend the range of the optimal plot you achieved from -4 to 10. What do you see? Does it look like a sine function?

As you can see, the extrapolation results for non-linear regression are even worse than for those of linear regression. This is because models only work well in the input space they have been trained in.

A possible way to be able to extrapolate and to use a non-linear method is to use forecasting techniques. This part is optional for those interested and going through the tutorial quite fast. Otherwise continue to the final part on classification in exercise 7.

6. Classification

In classification the purpose is to separate 2 classes. As an example we will use the double spiral. It is a very common toy example in machine learning and allows you to visually show what is going on.

As shown in the graph below the purpose is to separate the blue from the red dots.

In a colored image this is easy to do, but when you remove the color it becomes much harder. Can you do the classification in the image below?

In black the samples from the train set are shown and in yellow the samples from the validation set.

As you can see classifying is very hard to do when you don't get the answer even if you saw the solution earlier. But you will see that machine learning algorithms can solve this quite well if they can learn from examples.

6.1 Linear classifier

Let's try to do this with a linear classifier.

A linear classifier is basically a form of linear regression where the output is set to 1 for all the data points of class 1 and to 0 for all the data points of class 0.

Now let's plot the result.

As you can see a linear classifier returns a linear decision boundary.

6.2 Non-linear classification

Now let's do this better with a non-linear classifier using polynomials. Play with the degree of the polynomial expansion and look for the effect of the RidgeCV() and LassoCV() models. What gives you the best results?

If everything went well you should get a validation/test accuracy very close to 0.8.

6.3 Random Forests

An often used technique in machine learning are random forests. Basically they are decision trees, or in programmers terms, if-then-else structures, like the one shown below.

Decision trees are know to over-fit a lot because they just learn the train set by heart and store it. Random forests on the other hand combine multiple different (randomly initialized) decision trees that all over-fit in their own way. But by combining their output using a voting mechanism, they tend to cancel out eachothers mistakes. This approach is called an ensemble and can be used for any combination of machine learning techniques. A schematical representation of how such a random forest works is shown below.

Now let's try to use a random forest to solve the double spiral problem. (see sklearn.ensemble.RandomForestClassifier())

As you can see they are quite powerful right out of the box without any parameter tuning. But we can get the results even beter with some fine tuning.

Try changing the min_samples_leaf parameter for values between 0 and 1.

The min_samples_leaf parameter sets the number of datapoints that can create a new branch/leaf in the tree. So in practice it limits the depth of the decision tree. The bigger this parameter is, the less deep the tree will be and less likely each tree will overfit.

For this parameter you can set integer numbers to set the specific number of samples, or you can use values between 0 and 1 to express a percentage of the size of the dataset. Since you might exeperiment with a smaller dataset to roughly tune your paremeters, it is best to use values between 0 and 1 so that the value you chose is not as dependant on the size of the dataset you are working with.

Now that you have found the optimal min_samples_leaf run the code again with the same parameter. Do you get the same result? Why not?

Another parameter to play with is the n_estimators parameter. Play with only this parameter to see what happens.

As you can see increasing the number of estimators improves the model and reduces over-fitting. This parameter actually sets the number of trees in the random forest. The more trees there are in the forest the better the result is. But obviously it requires more computing power so that is the limiting factor here.

This is the basic idea behind ensembles: if you combine more tools you get a good result on average.

Now try combining the n_estimators and min_samples_leaf parameter below.

As you have noticed by now it seems that random forests are less powerful than linear regression with polynomial feature extraction. This is because these polynomials are ideally suited for this task. This also means that you could get a better result if you would also apply polynomial expansion for random forests. Try that below.

As you have noticed it is still not possible to get the same result as linear regression. This illustrates that linear techniques are very powerful and often underrated. But in some situations they are not powerful enough and you need something stronger like a random forest or even neural networks.

There is one neat trick that can be used for random forests. If you set the n_jobs it will use more than 1 core to compute. Set it to -1 to use all the cores (including hyperthreading cores). But don't do that during this tutorial because that would block the machine you are all working on.

To avoid over-fitting you have set the max_depth parameter for random forests which sets the maximum depth of the tree. Instead you can also set the min_samples_split parameter which determines while building the tree how many data points you need at least before you create another split (this is an additional if-else structure). Or the min_samples_leaf that sets the minimum amount of data points you have in each leaf. All 3 parameters are dependent on the number of data points in your dataset especially the last 2 so don't forget to adapt them if you have been playing around with a small subset of the data. (A good trick to solve this might be to use a range similar to [0.0001, 0.001, 0.01, 0.1] * len(x_train). Feel free to extend the range in any direction. It is generally good practice to construct them using a log scale like in the example, or better like this: 10.0**np.arange(-5, 0, 0.5) * len(x_train).) In my experience min_samples_split or min_samples_leaf give slightly better results and it ususally doesn't make sense to combine more than 1 of these parameters.

In the previous exercises we have done a lot of the optimizations on the test set. This should of course be avoided. What you should do instead is to optimize and select your model using a validation set and of course you should automate this proces as shown in one of the earlier exercises. One thing to take into account here is that you should use multiple initialisation of a random forest because the decision trees is randomly generated.

7. Forecasting (Optional)

For the forecasting we are going to use page views data, very similar to the data used in the anomaly detection section. It is also page view data and contains 1 sample per hour.

In the graph above you can clearly see that there is a rising trend in the data.

7.1 One-step ahead prediction

This forecasting section will describe the one-step ahead prediction. This means in this case that we will only predict the next data point which is in this case the number of pageviews in the next hour.

Now let's first build a model that tries to predict the next data point from the previous one.

As you can see from the score above, the model is not perfect but it seems to get a relatively high score. Now let's make a prediction into the future and plot this.

To predict the datapoint after that we will use the predicted data to make a new prediction. The code below shows how this works for this data set using the linear model you used earlier. Don't forget to fill out the missing code.

As you can see from the image above the model doesn't quite seem to fit the data well. Let's see how we can improve this.

7.2 Multiple features

If your model is not smart enough there is a simple trick in machine learning to make your model more intelligent (but also more complex). This is by adding more features.

To make our model better we will use more than 1 sample from the past. To make your life easier there is a simple function below that will create a data set for you. The width parameter sets the number of hours in the past that will be used.

As you can see from the print above both x_train and y_train contains 303 datapoints. For x_train you see that there are now 5 features which contain the pageviews from the 5 past hours.

So let's have a look what the increase from 1 to 5 features results to.

Now change the width parameter to see if you can get a better score.

7.3 Over-fitting

Now execute the code below to see the prediction of this model.

As you can see in the image above the prediction is not what you would expect from a perfect model. What happened is that the model learned the training data by heart without 'understanding' what the data is really about. This fenomenon is called over-fitting and will always occur if you make your model more complex.

Now play with the width variable below to see if you can find a more sensible width.

As you will have noticed by now is that it is better to have a non-perfect score which will give you a much better outcome. Now try the same thing for the following models:

• sklearn.linear_model.RidgeCV()
• sklearn.linear_model.LassoCV()
• sklearn.ensemble.RandomForestRegressor()

The first 2 models also estimate the noise that is present in the data to avoid overfitting. RidgeCV() will keep the weights that are found small, but it won't put them to zero. LassoCV() on the other hand will put several weights to 0. Execute model.coef_ to see the actual coefficients that have been found.

RandomForestRegressor() is the regression variant of the RandomForestClassifier() and is therefore thus a non-linear method. This makes this method a lot more complex and therefore it will be able to represent more complex shapes than the linear method. This also means that it is much more capable to learn the data by heart (and thus to over-fit). In many cases however this additional complexity allows to better understand the data given the correct parameter settings (try a couple of times width = 25 (since it is random) and see what the results are; set the n_estimators parameter to a higher number to get a more stable results).

7.4 Automation

What we have done up to now is manually selecting the best outcome based on the test result. This can be considered cheating because you have just created a self-fulfilling prophecy. Additionally it is not only cheating it is also hard to find the exact width that gives the best result by just visually inspecting it. So we need a more objective approach to solve this.

To automate this process you can use a validation set. In this case we will use the last 48 hours of the training set to validate the score and select the best parameter value. This means that we will have to use a subset of the training set to fit the model.

If everything is correct the LassoCV methods was selected.

Now we are going to train this best model on all the data. In this way we use all the available data to build a model.

Altough the optimal result found here might not be the best visually, it is a far better result than the one you selected manually just because there was no cheating involved ;-).

Some additional info:

• This noise level of RidgeCV() and LassoCV() is estimated by automatically performing train and validation within the method itself. This will make them much more robust against over-fitting. The actual method used is Cross-validation) which is a better approach of what we do here because it repeats the training and validation multiple times for different training and validation sets. The parameter that is set for these methods is often called the regularization parameter in literature and is well suited to avoid over-fitting.

8. Main take home messages

Because we can't cover everything, we listed all the basics of what you should take home before working on your own machine learning project below.

8.1 The basic rules of machine learning

Any good club has its own set of rules. The rules for machine learning club are the following:

• First rule of ML is: Over-fitting is a real problem and try anything to avoid it
• Second rule of ML is: You are probably over-fitting. Are you sure you are not fitting on your test data?
• Third rule of ML is: You think over-fitting will not happen to you, but it is happening right now!
• Fourth rule of ML is: Talk about it with your peers because over-fitting is a real issue.

8.2 My winning strategy

Although I'd like to claim it as mine, it is a general (non-written) consensus amongst data scientists to use the following approach. Even experts should not skip any of the steps below.

1. Create a train set and a test set

• Rescale your train set to zero-mean-unit-variance (most methods assume gaussian distributed data)
• Don't look at the test set
• Implement a cross-validation framework
• Try linear regression with regularisation for regression and classification (RidgeCV or LassoCV)
• Try techniques to avoid over-fitting
• Check the validation score
• If the results are not optimal and there is no overfitting going on try adding features else go to step 17
• Rescale your features to zero-mean-unit-variance (most methods assume gaussian distributed data) or select those features that have this property
• Try techniques to avoid over-fitting (including removing features for more info see feature selection techniques)
• Check the validation score
• If the results are not optimal and there is no overfitting going on try random forrests else go to step 17
• Try techniques to avoid over-fitting (such as feature selection, to rank the features you can use this approach)
• Check the validation score
• If the results are not optimal and there is no overfitting going on try neural networks or deep learning else go to step 17
• Try techniques to avoid over-fitting
• Only in the end check the score on the test set and make sure it is similar to the validation score. Otherwise you have been overfitting and you need to take a couple steps back.
• Make an ensemble of your best (but significantly different) methods
• Finally build the model using all the data available and run it in production

You can try other machine learning techniques, but usually the difference is quite small. So don't waste too much time on getting to know them because they all have their own quirks and specific ways of over-fitting. Besides maybe most important of all, Kaggle competitions are usually won with one of these techniques.

If you do want to dive into other methods or if you want more details on the methods discussed here the sklearn website is a goot starting point.

8.3 How to avoid over-fitting

As you should know by now over-fitting is one of the biggest issues in machine learning. So pay attention for it.

Below you can find some of the most common techniques to avoid over-fitting:

• Use more data
• Artificially generate more data based on the orriginal data
• Use a smaller model (with less parameters)
• Use less features (and thus less parameters)
• Use a regularisation parameter
• Artificially add noise to your model (can be random noise or can be on/of noise in neural networks so that you get dropout)
• Only use linear models (or in neural networks make sure that the non-linearity in your model is closer to a linear function)
• Combine multiple models that each over-fit in their own peculiar way into what is called an ensemble

8.4 Most common features

Although there is no general rule to which features you should use, there are a couple of features that come back regularly:

• Log: Take the log of the data to make it more Gaussian. This works best for data that is exponentially or log-normally distributed
• Polynomials: The square is quite common but higher orders are often used as well
• Differentials: The first and sometimes the second derivative are used (see numpy.diff())
• Integrals (use numpy.sum() for example to implement it)
• Mean: Often used to smooth the data
• Median: Same as the mean but this ignores outliers
• Standard deviation or variance
• Skewness and kurtosis: These are rarely used but sometimes they contain valuable information
• Fourier transform: If your data contains a frequency spectrum. Typically used when processing speech and sound. (see numpy.fft.fft())
• Frequency filtering: Similar to the fourier transform (see scipy.signal.butter())
• Spatial filters: Are often used for images. Edge detectors for example
• Any other feature that seems to make sense regarding your data

Feedback

If you have any feedback regarding this tutorial, feel free to share it with us. You can mail to pieter.buteneers@gmail.com.