Lesson 1/2 Code Along

Replicating https://github.com/fastai/fastai/blob/e1df9c167918f1b349e1cb43517666773fc70c22/courses/dl1/lesson1.ipynb

Importing the libraries we need:

If not already: wget http://files.fast.ai/data/dogscats.zip

A first look at cat pictures

This is what the raw data looks like:

First model: Quick Start

Using a pretrained ResNet34 model.

For Computer Vision, ResNet34 is usually what you want to start with. It's fast and accurate. Start with 1 epoch, and if that's not enough to produce a good answer, run more.

Analyzing results: looking at pictures

As well as looking at the overall metrics, it's also a good idea to look at examples each of:

1. A few correct labels at random
2. A few incorrect labels at random
3. The most correct labels of each class (ie those with highest probability that are correct)
4. The most incorrect labels of each class (ie those with highest probability that are incorrect)
5. The most uncertain labels (ie those with probability closest to 0.5)

Choosing a learning rate

The method learn.lr_find() helps you find an optimal learning. It uses the technique dveloepd in the 2015 paper Cyclical Learning Rates for Training Neural Networks, where we simply keep increasing the learning rate from a very small value, until the loss starts decreasing. We can plot the learning rate across batches to see what this looks like.

We first create a new learner, since we want to know how to set the learning rate for a new (untrained) model.

Above, it stopped at 79% because the loss had clearly begun to get worse, so the finder stopped increasing the LR.

Our learn object contains an attribute sched that contains our learning rate scheduler, and has some convenient plotting functionality including this one:

The LR Finder has gone through the entire dataset, a mini-batch at a time, gradually increasing the learning rate at each mini-batch.

Note that in the previous plot iteration is one iteration (or minibatch) of SGD. In one epoch there are (num_train_samples / num_iterations) of SGD.

We can see the plot of loss versus learning rate to see where our loss stops decreasing:

The loss is still clearly imporoving at lr=1e-2 (0.01), so that's what we use.NOTE that the optimal learning rate can chance as we're training the model, so you may want to re-run this function from time to time. -- Cool makes me think of an automated pipeline.

We want to pick the largest learning rate where the loss is clearly improving.

WHAT YOU DO: Call learn.lr_find() and then you call learn.sched.plot() on the learning rate scheduler, find the point at which the loss is still clearly going down, and pick that.

Improving our model

Data Augmentation

If you try training for more epochs, you'll notice that we start to overfit, meaning our model is learning to recognize the specific images in the training set, rather than generalizing such that we also get good results on the validation set. One way to fix this is to effectively create more data, via data augmentation. This refers to randomly changing the images in ways that shouldn't impact their interpretation, such as horizontal flipping, zooming, and rotating.

We can do this by passing aug_tfms (augmentation transforms) to tfms_from_model, with a list of functions to appply that randomly change the image however we wish. For photos that are largely taken from teh side (eg: most photos of dogs & cats, as opposed to top-down photos like satellite imagery) we can use the pre-defined list of functions transforms_side_on. We can also specify random zooming of images up to specified scale by adding the max_zoom parameter.

Data Augmentation is one of the most important things you can do to make a better model.

Let's createa a new data object that includes this augmentation in the transforms.

precompute=True is saving all up to the penultimate layer activations. We can very quickly train a linear model atop those. But Data Augmentation won't work since activations are precomputed.

We can set learn.precompute=False and then run a few more epochs to use Data Augmentation.

By default, when we create a learner, it sets all by the last layer to frozen. That means that it's still only updating the weights in the last layer when we call fit.

Above: the accuracy isn't really improving, but we're not overfitting because the training loss is worse than the validation loss. Generally: if your training loss is much better than your validation loss: your model's not generalizing.

cycle_len=.. enables SGDR:

We're using a technique called Stochastic Gradient Descent with Restarts (SGDR), a variant of Learning Rate Annealing - which graudally deceases the learning rate as training progresses. This is helpful bc as we get closer to the optimal weights, we want to take smaller steps.

However, we may find ourselves in a fragile part of the weight space -- where small changes to the weights may result in large changes to the loss. To encourage our model to find parts of the weight space that are both accurate and stable, we increase the learning rate from time to time (restarts), to force it to jump to a different part of the weight space if the current area is "spikey".

(From paper: Snapshot Ensembles

The cycle_len parameter is the number of epochs between resetting the learning rate, and the number of times this happesn is the number of cycles, and is what we're actually passing as the 2nd paramter to fit(). So here's what our actual learning rate looks like:

Instead of a hacky manual decrease by an order of magnitude (common in academia & industry) method to LR annealing; it's better to choose a functional form. A great one turns out to be half a Cosine curve. (~ backwards 'lazy' S). That's: Cosine Annealing.

To do Snapshot Ensembling where the min-LR weights are saved & averaged, specify cycle save name along with cycle_len.

Above we're doing Cosine Annealing, then jumping back up, then repeat. So we wan't our LR to be aggressive enough to break out of unstable zones. The changing of the LR every minibatch turns out to be critical, and is not something done yet in academia/industry.

Our validation loss isn't improving much, so there's probably no point further training the last layer on its own.

Since we've got a pretty good model at this point, we might want to save it so we can load it again later without training from scratch.

Fine-tuning and Differential Learning Rate Annealing

Now that we have a good final layer trained, we can try fine-tuning the other layers. To tell the learner that we want to unfreeze the remaining layers, just call unfreeze().

This is how we setup to change the convolutional filters themselves. .unfreeze() unfreezes all layers.

NOTE that the other layers have already been trained to recognize ImageNet photos (whereas our final layers were randomly initialized), so we want to be careful of not destroying the carefully tuned weights that are already there.

Generally speaking, the earlier layers (as we've seen) have more general-purpose features. Therefore we'd expect them to need less fine-tuning for new datasets. For this reason we'll use Differential Learning Rates for different layers: the first few layers will be at 1e-4, the middle layers at 1e-3, and our FC layers we'll leave at 1e-2 as before. We refer to this as differential learning rates, although there's no standard name for this technique in the literature that we're aware of.

This array of LRs lets us tune the amount of learning/change to each layer/set of layers.

This ability to unfreeze and then use differential learning rates is, I've found, the secret taking a pretty good model and turning it into an awesome model -- J.Howard

This will assign the LRs to groups of layers.

you can do learn.unfreeze_to(.) and pass in a layer number, as well -- this is rarely needed thanks to the power of Dftnl LRS. This unfreezes all layers from the index onwards.

oh yeah, little note: the more layers you unfreeze the more memory (and compute?) intensive training becomes.

If cycle_len=2 and n_cycles=3, then the learner would train 3 cycles each of 2 epochs. But the below will train 7 epochs, not 6, because of cycle_mult=2, which multiplies the number of iterations by a factor (scaling the LR-Annealing linearly w/ time). This serves to get the learner to jump out of spikey loss spaces quickly at first, and once it's begun to find a good space, do more and more exploring as time goes on.

In practice I find this: 3 cycles, startng at 1, cycle_mult=2 works v.very often to get a pretty decent model. If not I'll do 3 cycles of length 2 with no cycle_mult. -- J.Howard

Another trick is adding the cycle_mult parameter.

Note that what's being plotted above is the learning rate of the final layers. The learning rates of the earlier layers are fixed at the same multiples of the final layer rates as we initially requested (ie: the first layers have 100x smaller, and middle layers 10x smaller LRs, since we set lr=np.array([1e-4,1e-3,1e-2]).

There's something else we can do with Data Augmentation: use it at inference time (aka test time). This is Test Time Augmentation, TTA.

TTA makes predictions on the images in your validation set and on a number of randomly augmented versions of them (by default: original image + 4 augmented vsns). It then takes the average prediction from them and uses that. To use TTA on the validation set, we can use the learner's TTA() method.

This dataset generally gets a 10-20% reduction in error when using TTA at this point.

8. Analyzing Results

Confusion Matrix

A common way to analyze the result of a classification model is to use a Confusion Matrix. Scikit-learn has a convenient function we cna use for this:

We can just print the matrix, or show a graphica view (mainly used for dependents w/ a large num of categories)

Looking at Pictures Again

9. Review: easy steps to train a World-Class Image Classifier

1. Enable data augmentation, and precompute=True
2. Use lr_find() to find the highest leanring rate where loss is stil clearly improving
3. Train last layer from precomputed activations for 1-2 epochs
4. Train last layer with data augmentation (ie precompute=False) for 2-3 epochs w/ cycle_len=1
5. Unfreeze all layers
6. Set earlier layers to 3x-10x lower learning rate than next higher layer
7. Use lr_find() again
8. Train full network with cycle_mult=2 until over-fitting

NOTE if you call lr_find() having set differential LRs, it'll print out the LR of the last layer group.

10. Understaind the Code for our First Model

Looking at the Dogs v Cats code line by line.

tfms stands for transformations. tfms_from_model takes care of resizing, image cropping, initial normalization (creating data w/ (mean, stdev) of (0,1)), and more.

We need a path that points to the dataset. In this path we'll also store temprorary data and final results. ImageClassifierData.from_paths reads data from a provided path and creates a dataset ready for training.

ConvLearner.pretrained builds a learner that contains a pre-trained model. The last layer of the model needs to be replaced w/ the layer of the right dimensions. The pretrained model was trained for 1000 classes, therefore the final layer predicts a vector of 1000 probabilities. The model for cats and dogs needs to output a 2D vector. The diagram below shows an example of how this was done in one of the earliest successful CNNs. The layer "FC8" here would be replaced w/ a new layer w/ 2 outputs.

original image

Parameters are learned by fitting a model to the data. Hyperparameters are another kind of parameter, that can't be directly learned from the regular training process. These parameters express 'higher-level' propoerties of the mdoel such as its complexity or how fast it should learn. Two examples of hyperpars are the learning rate and the number of epochs.

During the iterative training of a neural network, a batch or mini-batch is a subset of training samples used in one iterations of SGD. An epoch is a single pass thru the entire training set which consists of multiple iters of SGD

We can now fit the model - that is, use Gradient Descent to find the best pars for the fully connected layer we added, that can separate cat pictures from dog pictures. We need to pass 2 hyperpars: the learning rate (generally 1e-2, or 1e-3 is a good startin gpoint) and the number of epochs (you can pass in a higher number and just stop training when you see it's no longer improving, then re-run it with the number of epochs you found works well).

Analyzing results: Loss and Accuracy

When we run learn.fit we print 3 performance vals (above). The 1st is loss in the training set, the 2nd is loss in the validation set, and the 3rd is validaiton accuracy.

Accuracy is the ratio of correct predictions to the total number of predictions.

In Machine Learning, the loss function or cost function represents the price paid for inaccuracy of predictions.

The loss associated with one example in binary classification is given by: -(y * log(p) + (1-y) * log(1-p)) where y is the true label of x and p is the probability predicted by our model that the label is 1.

NOTE that in our toy exmaple above our accuracy is 100% and our loss is 0.16. Compare that to a loss of 0.03 that we're getting while predicting cats and dogs. Exercise: play w/ preds to get a lower loss for this example.

Example: Here's an example on how to compute the loss for 1 exmpl of a binary clsfn problem. Suppose for an image x w/ label 1, your model gives it a prediction of 0.9. For this case the loss should be small bc our model is predicting a lebel 1 w/ high probability.

loss = -log(0.9) = 0.10

Now suppose x has a label 0 but our model is predicted 0.9, in this case our lossshould be much larger:

loss = -log(1-0.9) = 2.30

• Exc: look at the other cases and convince yourself that this makes sense.
• Exc: how would you rewrite binary_loss using if instead of * and +?

Why not just maximuze accuracy? The binary classification loss is an easier function to optimize.