Transfer Learning on Fashion MNIST


Import Libraries


In [1]:
import os
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

In [2]:
import torch
import torchvision

In [3]:
from fastai.imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

Global Configs


In [4]:
size = 224
DATAPATH = "../data/FashionMNIST/"

In [5]:
is_sample = True
if is_sample: PATH = DATAPATH+'sample'
else: PATH = DATAPATH

In [8]:
torch.cuda.is_available()


Out[8]:
False

In [7]:
torch.backends.cudnn.enabled


Out[7]:
True

Load Dataset


In [8]:
data = torchvision.datasets.FashionMNIST(DATAPATH,
                                         download=True)

In [9]:
test = torchvision.datasets.FashionMNIST(DATAPATH,
                                         train=False)

Data Exploration

Label Description
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot

Let's look at the class distribution...


In [10]:
sns.distplot(data.train_labels,kde=False)


Out[10]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fc55c2de550>

We find an equal distribution for all classes i.e. 6000 for all 10 fashion classes

Let's look at the images...


In [11]:
def plot_images(imgs, figsize=(12,6), titles=None, maintitle=None):
    fig = plt.figure(figsize=figsize)
    imgs = np.array(imgs)
    if maintitle is not None:
        plt.suptitle(maintitle, fontsize=16)
    imgs_len = len(imgs)
    for idx in range(imgs_len):
        subplt = fig.add_subplot(1, imgs_len, idx+1)
        subplt.axis('Off')
        if titles is not None:
            subplt.set_title(titles[idx], fontsize=16)
        plt.imshow(imgs[idx], cmap='gray')

In [12]:
plot_images(data.train_data[:5],
            titles=data.train_labels[:5],
            maintitle='FashionMNIST Samples')



In [13]:
data.train_data.shape


Out[13]:
torch.Size([60000, 28, 28])

Hence, we have 60,000 images of dimensions 28 by 28


In [10]:
labels_dict = {0: 'Top', 1: 'Trouser', 2: 'Pullover', 
               3: 'Dress', 4: 'Coat', 5: 'Sandal',
               6: 'Shirt', 7: 'Sneaker', 8: 'Bag', 9: 'Boot'}

Data Preprocessing

Here we split the dataset into sample_train, sample_valid, train and valid sets.

Sample dataset is for testing on local machine before training on the cloud. It is set by the global config SAMPLE.


In [15]:
# sample_train = data.train_data[:1000]
# sample_train_labels = data.train_labels[:1000]
# sample_valid = data.train_data[1000:1100]
# sample_valid_labels = data.train_labels[1000:1100]

In [16]:
train = data.train_data[:50000]
train_labels = data.train_labels[:50000]
valid = data.train_data[50000:60000]
valid_labels = data.train_labels[50000:60000]

Create data directory structure


In [17]:
os.mkdir(DATAPATH+'train')
os.mkdir(DATAPATH+'valid')


---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-17-cc7a918cd0be> in <module>()
----> 1 os.mkdir(DATAPATH+'train')
      2 os.mkdir(DATAPATH+'valid')

FileExistsError: [Errno 17] File exists: '../data/FashionMNIST/train'

In [18]:
# a sample of the dataset
# os.mkdir(DATAPATH+'sample/train')
# os.mkdir(DATAPATH+'sample/valid')

Populate data directories


In [19]:
def imwrite_dir(images, labels, path):
    # make directory structure for labels or classes
    for label in np.unique(labels):
        os.mkdir(path+str(label))
    # put the images in the proper label directories
    for i in range(images.shape[0]):
        image = images[i]
        filepath = path+str(labels[i])+'/'+str(i)+'.jpg'
        torchvision.utils.save_image(image,
                                     filepath)

In [20]:
# imwrite_dir(sample_train, sample_train_labels, DATAPATH+'sample/train/')
# imwrite_dir(sample_valid, sample_valid_labels, DATAPATH+'sample/valid/')

In [21]:
imwrite_dir(train, train_labels, DATAPATH+'train/')
imwrite_dir(valid, valid_labels, DATAPATH+'valid/')


---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-21-0dd338b26ac5> in <module>()
----> 1 imwrite_dir(train, train_labels, DATAPATH+'train/')
      2 imwrite_dir(valid, valid_labels, DATAPATH+'valid/')

<ipython-input-19-4d79eefab7e5> in imwrite_dir(images, labels, path)
      2     # make directory structure for labels or classes
      3     for label in np.unique(labels):
----> 4         os.mkdir(path+str(label))
      5     # put the images in the proper label directories
      6     for i in range(images.shape[0]):

FileExistsError: [Errno 17] File exists: '../data/FashionMNIST/train/0'

Sanity check for data distribution


In [22]:
! ls ../data/FashionMNIST/train/0 | wc -l
! ls ../data/FashionMNIST/train/1 | wc -l
! ls ../data/FashionMNIST/train/5 | wc -l
! ls ../data/FashionMNIST/train/9 | wc -l


4977
5012
5004
4979

In [23]:
! ls ../data/FashionMNIST/valid/0 | wc -l
! ls ../data/FashionMNIST/valid/1 | wc -l
! ls ../data/FashionMNIST/valid/5 | wc -l
! ls ../data/FashionMNIST/valid/9 | wc -l


1023
988
996
1021

Training Model [resnet34]


In [19]:
arch = resnet34
data = ImageClassifierData.from_paths(PATH,tfms=tfms_from_model(arch, size))

In [20]:
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(0.01, 2)


epoch      trn_loss   val_loss   accuracy   
    0      0.59579    0.459351   0.830414  
    1      0.535873   0.425754   0.844944  

Out[20]:
[0.42575356, 0.8449442675159236]

In [23]:
learn.save('resnet34-train_valid')

Model Analysis


In [24]:
learn.load('resnet34-sample')

In [25]:
log_preds=learn.predict()
log_preds.shape


Out[25]:
(10000, 10)

In [26]:
log_preds[:5]


Out[26]:
array([[ -0.3408 ,  -8.96755,  -5.81225,  -2.67534,  -4.84085, -11.03217,  -1.56664, -11.63359,  -9.01889,
        -11.67291],
       [ -0.02209,  -7.26183,  -4.97118,  -4.8595 ,  -8.38599, -12.07985,  -5.1748 , -13.61431,  -7.4938 ,
        -11.41757],
       [ -0.01932, -11.85231,  -7.59891,  -4.70157,  -8.99029,  -8.4608 ,  -4.69603, -12.3272 ,  -9.60526,
        -11.60536],
       [ -0.00894, -10.84166,  -8.04645,  -6.77204,  -8.92337, -11.85935,  -4.92604, -13.90546, -11.24127,
        -13.27043],
       [ -0.00164, -12.69415,  -7.22549,  -7.82598, -11.4448 , -14.22698,  -7.66379, -15.51375, -10.36539,
        -16.76541]], dtype=float32)

In [27]:
preds = np.argmax(log_preds, axis=1)
preds[-5:]


Out[27]:
array([0, 9, 9, 9, 9])

In [28]:
all_probs = {}
for key, label in labels_dict.items():
    all_probs[key] = np.exp(log_preds[:,key])

Analysis Utilities


In [11]:
def rand_by_mask(mask):
    return np.random.choice(np.where(mask)[0], 4, replace=False)
def rand_by_correct(is_correct):
    return rand_by_mask((preds == data.val_y)==is_correct)

def most_by_mask(mask):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    return most_by_mask(((preds == data.val_y)==is_correct)
                        & (data.val_y == y))

Plot Utilities


In [12]:
def plots(ims, figsize=(12,6), rows=1, titles=None, main_title=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
    plt.suptitle(main_title, fontsize=24)

In [13]:
def load_img_id(ds, idx):
    return np.array(PIL.Image.open(PATH+'/'+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_probs = [(data.val_y[x], preds[x]) for x in idxs]
    title=title+'\n(True Label, Predicted Label)'
    return plots(imgs, rows=1, titles=title_probs,
                 figsize=(16,8), main_title=title)

A few correct labels at random


In [33]:
correct_idx = rand_by_correct(is_correct=True)
probs = all_probs[0]
plot_val_with_title(correct_idx, "Random Correct")


A few incorrect labels at random


In [34]:
incorrect_idx = rand_by_correct(is_correct=False)
probs = all_probs[0]
plot_val_with_title(incorrect_idx, "Random Incorrect")


The most correct labels of each class


In [35]:
for key, label in labels_dict.items():
    probs=all_probs[key]
    plot_val_with_title(most_by_correct(key, True),
                        "Most Correct {}".format(label))


The most incorrect labels of each class


In [36]:
for key, label in labels_dict.items():
    probs=all_probs[0]
    plot_val_with_title(most_by_correct(key, False),
                        "Most Incorrect {}".format(label))


The most uncertain labels for each class


In [37]:
for key, label in labels_dict.items():
    probs = all_probs[key]
    most_uncertain = np.argsort(np.abs(probs-0.5))[:4]
    plot_val_with_title(most_uncertain,
                        "Most uncertain {}".format(label))


Tuning the Learning Rate


In [36]:
learn = ConvLearner.pretrained(arch, data, precompute=True)

Finding the best learning rate by increasing it till the loss does not decrease any further


In [37]:
learn.lr_find()


 83%|████████▎ | 648/782 [00:30<00:06, 21.59it/s, loss=3.28] 

In [38]:
learn.sched.plot_lr()



In [39]:
learn.sched.plot()


We see that the loss stabilises at around 0.1. However, it is still improving at around 0.01.

Learning Rate Experiment: 0.05, 0.001


In [48]:
lr = 0.05

In [42]:
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(lr, 2)


 48%|████▊     | 376/782 [00:08<00:09, 42.69it/s, loss=0.623]
 50%|█████     | 391/782 [00:09<00:09, 42.96it/s, loss=0.625]
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/home/amanthevinci/anaconda3/envs/fastai/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/amanthevinci/anaconda3/envs/fastai/lib/python3.6/site-packages/tqdm/_tqdm.py", line 144, in run
    for instance in self.tqdm_cls._instances:
  File "/home/amanthevinci/anaconda3/envs/fastai/lib/python3.6/_weakrefset.py", line 60, in __iter__
    for itemref in self.data:
RuntimeError: Set changed size during iteration

epoch      trn_loss   val_loss   accuracy                    
    0      0.561684   0.447371   0.834793  
    1      0.509062   0.4164     0.848627                    

Out[42]:
[0.41640002, 0.8486265923566879]

The accuracy didn't change much. However, we find that the loss was still decreasing. Maybe more epochs are required for it to converge


In [44]:
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(lr, 3)


epoch      trn_loss   val_loss   accuracy                    
    0      0.533264   0.456031   0.832205  
    1      0.511457   0.41858    0.848826                    
    2      0.476418   0.401341   0.852309                    

Out[44]:
[0.40134063, 0.8523089171974523]

We see an improvement of 1% in the accuracy with a new epoch. Maybe we can try more.


In [49]:
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(lr, 4)


epoch      trn_loss   val_loss   accuracy                    
    0      0.563212   0.44973    0.83758   
    1      0.496929   0.415596   0.84992                     
    2      0.472383   0.409247   0.852607                    
    3      0.440639   0.387531   0.854399                    

Out[49]:
[0.38753062, 0.8543988853503185]

We see no significant improvement by adding another epoch of training.


In [46]:
lr = 0.001

In [47]:
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(lr, 4)


epoch      trn_loss   val_loss   accuracy                    
    0      0.767021   0.589015   0.798069  
    1      0.659202   0.526268   0.811803                    
    2      0.623955   0.496948   0.821756                    
    3      0.575266   0.481673   0.827926                    

Out[47]:
[0.48167282, 0.8279259554140127]

The model accuracy worsens on lowering the learning rate. Hence, 0.05 seems to be the optimal learning rate.

Report

Most Optimal Learning Rate: 0.05 [4 Epochs]
Accuracy: 85.4%


Boosting Model

1. Data Augmentation


In [15]:
arch = resnet34

In [16]:
tfms_side = tfms_from_model(arch, size, aug_tfms=transforms_side_on)

In [17]:
def get_augs():
    data=ImageClassifierData.from_paths(DATAPATH, bs=2, tfms=tfms_side)
    x, _ = next(iter(data.aug_dl))
    return data.trn_ds.denorm(x)[1]

In [18]:
sample_imgs = np.stack([get_augs() for i in range(6)])
plots(sample_imgs, rows=2, main_title='Augmented Data')