MDR, June 2017, personal study.

This notebook records a personal learning project, conducted during the Fast.AI course, in which a convnet is taught to ID distracted drivers. The dataset comes from an old Kaggle comp. This is the second experiment, in which I build a CNN from scratch, following the statefarm notebook.

Planning

This notebook assumes that the data has already been downloaded and organised

setup packages, paths
review the task
set up a simple model
- with batchnorm (not dropout?)
train the network
review the val_acc
assess errors
try additional techniques for improving the accuracy, reducing overfitting
- data augmentation
- ensembling
submit to Kaggle

Setup (every session)

Import packages (every new session)

N.B. You may need to ensure that the files...

'utils.py' (by fast.ai),
'vgg16.py' (adapted by fast.ai?), and
'vgg16bn' </strong> ...are findable by python - e.g. present in the root of the current directory.



In [1]:

    
%matplotlib inline
import numpy as np
import os



In [2]:

    
import utils
from utils import *
#from imp import reload  # fixes a P2-P3 incompatibility
#reload(utils)









    



WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 1070 (CNMeM is disabled, cuDNN 5110)
Using Theano backend.

A function for checking for a folder name and creating one if needed.



In [3]:

    
import errno

def make_sure_path_exists(path):
    try:
        os.makedirs(path)
    except OSError as exception:
        if exception.errno != errno.EEXIST:
            raise

Set up the path variables

DO re-run these cells, too...



In [5]:

    
current_dir = os.getcwd()
HOME_DIR = current_dir
DATA_HOME_DIR = current_dir + '/data'
print(' ', HOME_DIR, '\n ', DATA_HOME_DIR)









    



  /home/mark/Study/dl.fast.ai/deeplearning1/_MDR/2_distracted_driver 
  /home/mark/Study/dl.fast.ai/deeplearning1/_MDR/2_distracted_driver/data

This relative approach to folder referencing also works...



In [6]:

    
all_data_path = "data/"
sampled_path = all_data_path + "sample"
test_path = all_data_path + "test"

# The following allows for easy switching...
data_path = all_data_path   
#data_path = sampled_path      

%ls $all_data_path









    



driver_imgs_list.csv      results/                   test/      valid/
driver_imgs_list.csv.zip  sample/                    train/
imgs.zip                  sample_submission.csv.zip  unzipped/

Review the task and data

The task

Quoted from the Kaggle data page:

In this competition you are given driver images, each taken in a car with a driver doing something in the car (texting, eating, talking on the phone, makeup, reaching behind, etc). Your goal is to predict the likelihood of what the driver is doing in each picture.

The 10 classes to predict are:

c0: safe driving
c1: texting - right
c2: talking on the phone - right
c3: texting - left
c4: talking on the phone - left
c5: operating the radio
c6: drinking
c7: reaching behind
c8: hair and makeup
c9: talking to passenger

To ensure that this is a computer vision problem, we have removed metadata such as creation dates. The train and test data are split on the drivers, such that one driver can only appear on either train or test set.

To discourage hand labeling, we have supplemented the test dataset with some images that are resized. These processed images are ignored and don't count towards your score. </blockquote> File descriptions

imgs.zip - zipped folder of all (train/test) images
sample_submission.csv - a sample submission file in the correct format
driver_imgs_list.csv - a list of training images, their subject (driver) id, and class id



In [73]:

    
%ls -l $all_data_path









    



total 4203272
-rw-r--r--  1 mark mark     491359 Apr  7  2016 driver_imgs_list.csv
-rw-rw-r--  1 mark mark      95118 May  6 09:41 driver_imgs_list.csv.zip
-rw-rw-r--  1 mark mark 4292669227 May  6 09:41 imgs.zip
drwxrwxr-x  4 mark mark       4096 May 14 22:08 results/
drwxrwxr-x  5 mark mark       4096 May  9 21:09 sample/
-rw-rw-r--  1 mark mark     211199 May  6 09:18 sample_submission.csv.zip
drwxr-xr-x  3 mark mark   10620928 May 14 21:32 test/
drwxr-xr-x 12 mark mark       4096 May  7 22:04 train/
drwxrwxr-x  2 mark mark       4096 May 13 09:23 unzipped/
drwxrwxr-x 12 mark mark       4096 May 11 22:08 valid/

Import and inspect the class data (as a data frame)



In [6]:

    
import pandas as pd
images_df = pd.read_csv(class_file_path, sep=',')









    



---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-e59066ee3358> in <module>()
      1 import pandas as pd
----> 2 images_df = pd.read_csv(class_file_path, sep=',')

NameError: name 'class_file_path' is not defined



In [10]:

    
print(images_df.shape)
print(images_df.columns)









    



(22424, 3)
Index(['subject', 'classname', 'img'], dtype='object')

List unique values - classes.



In [11]:

    
#List unique values in the df['name'] column
images_df.classname.unique()









    Out[11]:





array(['c0', 'c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8', 'c9'], dtype=object)

... and drivers ...



In [12]:

    
l = len(images_df.subject.unique())          ; print(l)
s = sorted(list(images_df.subject.unique())) ; print(s)









    



26
['p002', 'p012', 'p014', 'p015', 'p016', 'p021', 'p022', 'p024', 'p026', 'p035', 'p039', 'p041', 'p042', 'p045', 'p047', 'p049', 'p050', 'p051', 'p052', 'p056', 'p061', 'p064', 'p066', 'p072', 'p075', 'p081']

This suggests that - to achieve an 80/20 split between test and validation - we'd have to select 5 drivers to move into the 'valid' set. We'll store this list of drivers for use when we build the valid directory.



In [13]:

    
v = int(l * .2) ; print(v)
valid_subj = s[-v:]
print(valid_subj)









    



5
['p064', 'p066', 'p072', 'p075', 'p081']

Check the directory structure

Let's check that against the directory structure of the unzipped images (imgs).



In [14]:

    
!tree -d $all_data_path/unzipped









    



data/unzipped

0 directories

Q: How many training images are there, across all 10 classes? A: 22,424 images



In [16]:

    
src = all_data_path + '/train/**/*.jpg'
print(src)
g = glob(src, recursive=True)
train_set_size = len(g)
print(train_set_size)









    



data/train/**/*.jpg
22424






    Out[16]:





['data/train/c8/img_22971.jpg',
 'data/train/c8/img_76367.jpg',
 'data/train/c8/img_25239.jpg',
 'data/train/c8/img_23496.jpg',
 'data/train/c8/img_44057.jpg',
 'data/train/c8/img_70207.jpg',
 'data/train/c8/img_47484.jpg',
 'data/train/c8/img_72616.jpg',
 'data/train/c8/img_63477.jpg',
 'data/train/c8/img_93136.jpg',
 'data/train/c8/img_37289.jpg',
 'data/train/c8/img_98495.jpg',
 'data/train/c8/img_5355.jpg',
 'data/train/c8/img_26813.jpg',
 'data/train/c8/img_56050.jpg',
 'data/train/c8/img_8631.jpg',
 'data/train/c8/img_18910.jpg',
 'data/train/c8/img_14072.jpg',
 'data/train/c8/img_56207.jpg',
 'data/train/c8/img_74098.jpg',
 'data/train/c8/img_101737.jpg',
 'data/train/c8/img_100480.jpg',
 'data/train/c8/img_43109.jpg',
 'data/train/c8/img_76691.jpg',
 'data/train/c8/img_88573.jpg',
 'data/train/c8/img_88803.jpg',
 'data/train/c8/img_17506.jpg',
 'data/train/c8/img_28833.jpg',
 'data/train/c8/img_37131.jpg',
 'data/train/c8/img_55033.jpg',
 'data/train/c8/img_27663.jpg',
 'data/train/c8/img_101814.jpg',
 'data/train/c8/img_85299.jpg',
 'data/train/c8/img_20212.jpg',
 'data/train/c8/img_84399.jpg',
 'data/train/c8/img_29907.jpg',
 'data/train/c8/img_28517.jpg',
 'data/train/c8/img_4232.jpg',
 'data/train/c8/img_86145.jpg',
 'data/train/c8/img_43730.jpg',
 'data/train/c8/img_31409.jpg',
 'data/train/c8/img_66162.jpg',
 'data/train/c8/img_22941.jpg',
 'data/train/c8/img_1540.jpg',
 'data/train/c8/img_45868.jpg',
 'data/train/c8/img_63216.jpg',
 'data/train/c8/img_15331.jpg',
 'data/train/c8/img_67093.jpg',
 'data/train/c8/img_77515.jpg',
 'data/train/c8/img_77015.jpg',
 'data/train/c8/img_56484.jpg',
 'data/train/c8/img_1973.jpg',
 'data/train/c8/img_48174.jpg',
 'data/train/c8/img_86447.jpg',
 'data/train/c8/img_52914.jpg',
 'data/train/c8/img_101751.jpg',
 'data/train/c8/img_39081.jpg',
 'data/train/c8/img_15441.jpg',
 'data/train/c8/img_20708.jpg',
 'data/train/c8/img_27671.jpg',
 'data/train/c8/img_67095.jpg',
 'data/train/c8/img_87515.jpg',
 'data/train/c8/img_92985.jpg',
 'data/train/c8/img_5474.jpg',
 'data/train/c8/img_35863.jpg',
 'data/train/c8/img_51140.jpg',
 'data/train/c8/img_76450.jpg',
 'data/train/c8/img_75090.jpg',
 'data/train/c8/img_75413.jpg',
 'data/train/c8/img_49335.jpg',
 'data/train/c8/img_68187.jpg',
 'data/train/c8/img_6628.jpg',
 'data/train/c8/img_96734.jpg',
 'data/train/c8/img_23854.jpg',
 'data/train/c8/img_21258.jpg',
 'data/train/c8/img_62210.jpg',
 'data/train/c8/img_10352.jpg',
 'data/train/c8/img_44946.jpg',
 'data/train/c8/img_9464.jpg',
 'data/train/c8/img_4823.jpg',
 'data/train/c8/img_25530.jpg',
 'data/train/c8/img_24889.jpg',
 'data/train/c8/img_7325.jpg',
 'data/train/c8/img_6289.jpg',
 'data/train/c8/img_65842.jpg',
 'data/train/c8/img_66733.jpg',
 'data/train/c8/img_2464.jpg',
 'data/train/c8/img_51981.jpg',
 'data/train/c8/img_23655.jpg',
 'data/train/c8/img_29028.jpg',
 'data/train/c8/img_78496.jpg',
 'data/train/c8/img_64415.jpg',
 'data/train/c8/img_92534.jpg',
 'data/train/c8/img_86836.jpg',
 'data/train/c8/img_29432.jpg',
 'data/train/c8/img_35831.jpg',
 'data/train/c8/img_671.jpg',
 'data/train/c8/img_35086.jpg',
 'data/train/c8/img_82784.jpg',
 'data/train/c8/img_21070.jpg',
 'data/train/c8/img_50401.jpg',
 'data/train/c8/img_82587.jpg',
 'data/train/c8/img_76964.jpg',
 'data/train/c8/img_69119.jpg',
 'data/train/c8/img_68147.jpg',
 'data/train/c8/img_77051.jpg',
 'data/train/c8/img_95038.jpg',
 'data/train/c8/img_54710.jpg',
 'data/train/c8/img_25156.jpg',
 'data/train/c8/img_58070.jpg',
 'data/train/c8/img_31139.jpg',
 'data/train/c8/img_90109.jpg',
 'data/train/c8/img_31319.jpg',
 'data/train/c8/img_35035.jpg',
 'data/train/c8/img_10361.jpg',
 'data/train/c8/img_74145.jpg',
 'data/train/c8/img_28755.jpg',
 'data/train/c8/img_53436.jpg',
 'data/train/c8/img_97477.jpg',
 'data/train/c8/img_15160.jpg',
 'data/train/c8/img_69278.jpg',
 'data/train/c8/img_4409.jpg',
 'data/train/c8/img_66705.jpg',
 'data/train/c8/img_96763.jpg',
 'data/train/c8/img_86546.jpg',
 'data/train/c8/img_80725.jpg',
 'data/train/c8/img_78081.jpg',
 'data/train/c8/img_25010.jpg',
 'data/train/c8/img_46580.jpg',
 'data/train/c8/img_91555.jpg',
 'data/train/c8/img_64028.jpg',
 'data/train/c8/img_86353.jpg',
 'data/train/c8/img_89839.jpg',
 'data/train/c8/img_80930.jpg',
 'data/train/c8/img_39640.jpg',
 'data/train/c8/img_45480.jpg',
 'data/train/c8/img_98292.jpg',
 'data/train/c8/img_87262.jpg',
 'data/train/c8/img_38649.jpg',
 'data/train/c8/img_24684.jpg',
 'data/train/c8/img_48387.jpg',
 'data/train/c8/img_86490.jpg',
 'data/train/c8/img_85692.jpg',
 'data/train/c8/img_95399.jpg',
 'data/train/c8/img_68431.jpg',
 'data/train/c8/img_22999.jpg',
 'data/train/c8/img_14814.jpg',
 'data/train/c8/img_99693.jpg',
 'data/train/c8/img_64191.jpg',
 'data/train/c8/img_16874.jpg',
 'data/train/c8/img_59760.jpg',
 'data/train/c8/img_7887.jpg',
 'data/train/c8/img_85605.jpg',
 'data/train/c8/img_57611.jpg',
 'data/train/c8/img_29031.jpg',
 'data/train/c8/img_32976.jpg',
 'data/train/c8/img_13647.jpg',
 'data/train/c8/img_56767.jpg',
 'data/train/c8/img_77709.jpg',
 'data/train/c8/img_7139.jpg',
 'data/train/c8/img_83024.jpg',
 'data/train/c8/img_51986.jpg',
 'data/train/c8/img_94489.jpg',
 'data/train/c8/img_59301.jpg',
 'data/train/c8/img_48712.jpg',
 'data/train/c8/img_36908.jpg',
 'data/train/c8/img_70488.jpg',
 'data/train/c8/img_75893.jpg',
 'data/train/c8/img_92151.jpg',
 'data/train/c8/img_63759.jpg',
 'data/train/c8/img_45725.jpg',
 'data/train/c8/img_20639.jpg',
 'data/train/c8/img_68838.jpg',
 'data/train/c8/img_91893.jpg',
 'data/train/c8/img_50461.jpg',
 'data/train/c8/img_82951.jpg',
 'data/train/c8/img_83277.jpg',
 'data/train/c8/img_35250.jpg',
 'data/train/c8/img_7514.jpg',
 'data/train/c8/img_89208.jpg',
 'data/train/c8/img_42085.jpg',
 'data/train/c8/img_17661.jpg',
 'data/train/c8/img_86015.jpg',
 'data/train/c8/img_94081.jpg',
 'data/train/c8/img_9128.jpg',
 'data/train/c8/img_56891.jpg',
 'data/train/c8/img_33794.jpg',
 'data/train/c8/img_50684.jpg',
 'data/train/c8/img_1808.jpg',
 'data/train/c8/img_24354.jpg',
 'data/train/c8/img_20614.jpg',
 'data/train/c8/img_37864.jpg',
 'data/train/c8/img_36813.jpg',
 'data/train/c8/img_18454.jpg',
 'data/train/c8/img_33934.jpg',
 'data/train/c8/img_35337.jpg',
 'data/train/c8/img_4853.jpg',
 'data/train/c8/img_31338.jpg',
 'data/train/c8/img_25738.jpg',
 'data/train/c8/img_80919.jpg',
 'data/train/c8/img_41659.jpg',
 'data/train/c8/img_6680.jpg',
 'data/train/c8/img_93469.jpg',
 'data/train/c8/img_22463.jpg',
 'data/train/c8/img_97587.jpg',
 'data/train/c8/img_68491.jpg',
 'data/train/c8/img_100787.jpg',
 'data/train/c8/img_57135.jpg',
 'data/train/c8/img_5377.jpg',
 'data/train/c8/img_44430.jpg',
 'data/train/c8/img_7935.jpg',
 'data/train/c8/img_17167.jpg',
 'data/train/c8/img_69290.jpg',
 'data/train/c8/img_55008.jpg',
 'data/train/c8/img_13868.jpg',
 'data/train/c8/img_32199.jpg',
 'data/train/c8/img_16321.jpg',
 'data/train/c8/img_97774.jpg',
 'data/train/c8/img_26801.jpg',
 'data/train/c8/img_34916.jpg',
 'data/train/c8/img_4809.jpg',
 'data/train/c8/img_22684.jpg',
 'data/train/c8/img_19399.jpg',
 'data/train/c8/img_50700.jpg',
 'data/train/c8/img_71576.jpg',
 'data/train/c8/img_95880.jpg',
 'data/train/c8/img_70752.jpg',
 'data/train/c8/img_34235.jpg',
 'data/train/c8/img_19616.jpg',
 'data/train/c8/img_99596.jpg',
 'data/train/c8/img_94544.jpg',
 'data/train/c8/img_87552.jpg',
 'data/train/c8/img_84837.jpg',
 'data/train/c8/img_11472.jpg',
 'data/train/c8/img_56078.jpg',
 'data/train/c8/img_49411.jpg',
 'data/train/c8/img_89591.jpg',
 'data/train/c8/img_40632.jpg',
 'data/train/c8/img_49353.jpg',
 'data/train/c8/img_17897.jpg',
 'data/train/c8/img_61587.jpg',
 'data/train/c8/img_17273.jpg',
 'data/train/c8/img_101096.jpg',
 'data/train/c8/img_26115.jpg',
 'data/train/c8/img_20475.jpg',
 'data/train/c8/img_52404.jpg',
 'data/train/c8/img_94215.jpg',
 'data/train/c8/img_97904.jpg',
 'data/train/c8/img_41440.jpg',
 'data/train/c8/img_34328.jpg',
 'data/train/c8/img_77481.jpg',
 'data/train/c8/img_15203.jpg',
 'data/train/c8/img_46117.jpg',
 'data/train/c8/img_23002.jpg',
 'data/train/c8/img_77225.jpg',
 'data/train/c8/img_69730.jpg',
 'data/train/c8/img_59098.jpg',
 'data/train/c8/img_87593.jpg',
 'data/train/c8/img_11357.jpg',
 'data/train/c8/img_62221.jpg',
 'data/train/c8/img_3471.jpg',
 'data/train/c8/img_31795.jpg',
 'data/train/c8/img_2009.jpg',
 'data/train/c8/img_85931.jpg',
 'data/train/c8/img_8860.jpg',
 'data/train/c8/img_91046.jpg',
 'data/train/c8/img_40249.jpg',
 'data/train/c8/img_22421.jpg',
 'data/train/c8/img_76212.jpg',
 'data/train/c8/img_63251.jpg',
 'data/train/c8/img_58983.jpg',
 'data/train/c8/img_87249.jpg',
 'data/train/c8/img_32509.jpg',
 'data/train/c8/img_34788.jpg',
 'data/train/c8/img_68820.jpg',
 'data/train/c8/img_74520.jpg',
 'data/train/c8/img_54016.jpg',
 'data/train/c8/img_19665.jpg',
 'data/train/c8/img_23269.jpg',
 'data/train/c8/img_88398.jpg',
 'data/train/c8/img_2378.jpg',
 'data/train/c8/img_25969.jpg',
 'data/train/c8/img_12535.jpg',
 'data/train/c8/img_7167.jpg',
 'data/train/c8/img_73618.jpg',
 'data/train/c8/img_100814.jpg',
 'data/train/c8/img_46264.jpg',
 'data/train/c8/img_79203.jpg',
 'data/train/c8/img_54657.jpg',
 'data/train/c8/img_20041.jpg',
 'data/train/c8/img_18354.jpg',
 'data/train/c8/img_93848.jpg',
 'data/train/c8/img_87156.jpg',
 'data/train/c8/img_23948.jpg',
 'data/train/c8/img_17280.jpg',
 'data/train/c8/img_99709.jpg',
 'data/train/c8/img_100235.jpg',
 'data/train/c8/img_89462.jpg',
 'data/train/c8/img_49179.jpg',
 'data/train/c8/img_46871.jpg',
 'data/train/c8/img_10650.jpg',
 'data/train/c8/img_50150.jpg',
 'data/train/c8/img_88749.jpg',
 'data/train/c8/img_91700.jpg',
 'data/train/c8/img_11324.jpg',
 'data/train/c8/img_14159.jpg',
 'data/train/c8/img_78925.jpg',
 'data/train/c8/img_4520.jpg',
 'data/train/c8/img_26819.jpg',
 'data/train/c8/img_70836.jpg',
 'data/train/c8/img_51189.jpg',
 'data/train/c8/img_7757.jpg',
 'data/train/c8/img_40069.jpg',
 'data/train/c8/img_5003.jpg',
 'data/train/c8/img_5365.jpg',
 'data/train/c8/img_7905.jpg',
 'data/train/c8/img_68365.jpg',
 'data/train/c8/img_28206.jpg',
 'data/train/c8/img_50660.jpg',
 'data/train/c8/img_45021.jpg',
 'data/train/c8/img_58793.jpg',
 'data/train/c8/img_48752.jpg',
 'data/train/c8/img_99262.jpg',
 'data/train/c8/img_61128.jpg',
 'data/train/c8/img_35212.jpg',
 'data/train/c8/img_60684.jpg',
 'data/train/c8/img_20265.jpg',
 'data/train/c8/img_10147.jpg',
 'data/train/c8/img_89787.jpg',
 'data/train/c8/img_95215.jpg',
 'data/train/c8/img_82621.jpg',
 'data/train/c8/img_247.jpg',
 'data/train/c8/img_51103.jpg',
 'data/train/c8/img_9282.jpg',
 'data/train/c8/img_36126.jpg',
 'data/train/c8/img_11702.jpg',
 'data/train/c8/img_62076.jpg',
 'data/train/c8/img_50664.jpg',
 'data/train/c8/img_8652.jpg',
 'data/train/c8/img_35293.jpg',
 'data/train/c8/img_59923.jpg',
 'data/train/c8/img_56372.jpg',
 'data/train/c8/img_35367.jpg',
 'data/train/c8/img_95148.jpg',
 'data/train/c8/img_75475.jpg',
 'data/train/c8/img_82142.jpg',
 'data/train/c8/img_25470.jpg',
 'data/train/c8/img_43680.jpg',
 'data/train/c8/img_86842.jpg',
 'data/train/c8/img_41153.jpg',
 'data/train/c8/img_71835.jpg',
 'data/train/c8/img_66741.jpg',
 'data/train/c8/img_83468.jpg',
 'data/train/c8/img_52003.jpg',
 'data/train/c8/img_59353.jpg',
 'data/train/c8/img_62133.jpg',
 'data/train/c8/img_57685.jpg',
 'data/train/c8/img_57804.jpg',
 'data/train/c8/img_54379.jpg',
 'data/train/c8/img_28169.jpg',
 'data/train/c8/img_4384.jpg',
 'data/train/c8/img_99344.jpg',
 'data/train/c8/img_24378.jpg',
 'data/train/c8/img_2362.jpg',
 'data/train/c8/img_19078.jpg',
 'data/train/c8/img_84719.jpg',
 'data/train/c8/img_28131.jpg',
 'data/train/c8/img_48403.jpg',
 'data/train/c8/img_59999.jpg',
 'data/train/c8/img_90703.jpg',
 'data/train/c8/img_96623.jpg',
 'data/train/c8/img_26378.jpg',
 'data/train/c8/img_44870.jpg',
 'data/train/c8/img_9596.jpg',
 'data/train/c8/img_69406.jpg',
 'data/train/c8/img_59882.jpg',
 'data/train/c8/img_79889.jpg',
 'data/train/c8/img_88994.jpg',
 'data/train/c8/img_68389.jpg',
 'data/train/c8/img_15914.jpg',
 'data/train/c8/img_102106.jpg',
 'data/train/c8/img_58226.jpg',
 'data/train/c8/img_8219.jpg',
 'data/train/c8/img_76640.jpg',
 'data/train/c8/img_28549.jpg',
 'data/train/c8/img_51605.jpg',
 'data/train/c8/img_41812.jpg',
 'data/train/c8/img_75188.jpg',
 'data/train/c8/img_83330.jpg',
 'data/train/c8/img_100846.jpg',
 'data/train/c8/img_85744.jpg',
 'data/train/c8/img_46154.jpg',
 'data/train/c8/img_17480.jpg',
 'data/train/c8/img_86786.jpg',
 'data/train/c8/img_101973.jpg',
 'data/train/c8/img_83931.jpg',
 'data/train/c8/img_86286.jpg',
 'data/train/c8/img_91124.jpg',
 'data/train/c8/img_81097.jpg',
 'data/train/c8/img_798.jpg',
 'data/train/c8/img_10359.jpg',
 'data/train/c8/img_12025.jpg',
 'data/train/c8/img_50220.jpg',
 'data/train/c8/img_5921.jpg',
 'data/train/c8/img_49247.jpg',
 'data/train/c8/img_39261.jpg',
 'data/train/c8/img_52383.jpg',
 'data/train/c8/img_84337.jpg',
 'data/train/c8/img_4734.jpg',
 'data/train/c8/img_7999.jpg',
 'data/train/c8/img_34329.jpg',
 'data/train/c8/img_48430.jpg',
 'data/train/c8/img_59176.jpg',
 'data/train/c8/img_40911.jpg',
 'data/train/c8/img_33277.jpg',
 'data/train/c8/img_74406.jpg',
 'data/train/c8/img_67544.jpg',
 'data/train/c8/img_48976.jpg',
 'data/train/c8/img_95446.jpg',
 'data/train/c8/img_88675.jpg',
 'data/train/c8/img_79537.jpg',
 'data/train/c8/img_72317.jpg',
 'data/train/c8/img_101034.jpg',
 'data/train/c8/img_32172.jpg',
 'data/train/c8/img_97558.jpg',
 'data/train/c8/img_83892.jpg',
 'data/train/c8/img_12635.jpg',
 'data/train/c8/img_35130.jpg',
 'data/train/c8/img_70748.jpg',
 'data/train/c8/img_9649.jpg',
 'data/train/c8/img_48710.jpg',
 'data/train/c8/img_79177.jpg',
 'data/train/c8/img_27337.jpg',
 'data/train/c8/img_75335.jpg',
 'data/train/c8/img_2494.jpg',
 'data/train/c8/img_74193.jpg',
 'data/train/c8/img_19798.jpg',
 'data/train/c8/img_58640.jpg',
 'data/train/c8/img_20962.jpg',
 'data/train/c8/img_7181.jpg',
 'data/train/c8/img_63083.jpg',
 'data/train/c8/img_24927.jpg',
 'data/train/c8/img_81077.jpg',
 'data/train/c8/img_67177.jpg',
 'data/train/c8/img_72387.jpg',
 'data/train/c8/img_78677.jpg',
 'data/train/c8/img_28593.jpg',
 'data/train/c8/img_99160.jpg',
 'data/train/c8/img_76882.jpg',
 'data/train/c8/img_96702.jpg',
 'data/train/c8/img_74831.jpg',
 'data/train/c8/img_82701.jpg',
 'data/train/c8/img_28079.jpg',
 'data/train/c8/img_70997.jpg',
 'data/train/c8/img_93006.jpg',
 'data/train/c8/img_29095.jpg',
 'data/train/c8/img_30621.jpg',
 'data/train/c8/img_88508.jpg',
 'data/train/c8/img_1842.jpg',
 'data/train/c8/img_100850.jpg',
 'data/train/c8/img_76073.jpg',
 'data/train/c8/img_42327.jpg',
 'data/train/c8/img_19909.jpg',
 'data/train/c8/img_52950.jpg',
 'data/train/c8/img_74065.jpg',
 'data/train/c8/img_79385.jpg',
 'data/train/c8/img_95770.jpg',
 'data/train/c8/img_89028.jpg',
 'data/train/c8/img_36367.jpg',
 'data/train/c8/img_86819.jpg',
 'data/train/c8/img_4207.jpg',
 'data/train/c8/img_27683.jpg',
 'data/train/c8/img_96547.jpg',
 'data/train/c8/img_92546.jpg',
 'data/train/c8/img_13387.jpg',
 'data/train/c8/img_59666.jpg',
 'data/train/c8/img_5160.jpg',
 'data/train/c8/img_8735.jpg',
 'data/train/c8/img_28747.jpg',
 'data/train/c8/img_80432.jpg',
 'data/train/c8/img_12041.jpg',
 'data/train/c8/img_51051.jpg',
 'data/train/c8/img_52484.jpg',
 'data/train/c8/img_43496.jpg',
 'data/train/c8/img_57724.jpg',
 'data/train/c8/img_52140.jpg',
 'data/train/c8/img_71059.jpg',
 'data/train/c8/img_34889.jpg',
 'data/train/c8/img_3371.jpg',
 'data/train/c8/img_32789.jpg',
 'data/train/c8/img_10706.jpg',
 'data/train/c8/img_68094.jpg',
 'data/train/c8/img_57030.jpg',
 'data/train/c8/img_100976.jpg',
 'data/train/c8/img_895.jpg',
 'data/train/c8/img_34370.jpg',
 'data/train/c8/img_18790.jpg',
 'data/train/c8/img_59400.jpg',
 'data/train/c8/img_100791.jpg',
 'data/train/c8/img_52997.jpg',
 'data/train/c8/img_5538.jpg',
 'data/train/c8/img_91029.jpg',
 'data/train/c8/img_36526.jpg',
 'data/train/c8/img_58956.jpg',
 'data/train/c8/img_82366.jpg',
 'data/train/c8/img_7188.jpg',
 'data/train/c8/img_40554.jpg',
 'data/train/c8/img_47790.jpg',
 'data/train/c8/img_95436.jpg',
 'data/train/c8/img_29908.jpg',
 'data/train/c8/img_10199.jpg',
 'data/train/c8/img_70664.jpg',
 'data/train/c8/img_48231.jpg',
 'data/train/c8/img_98882.jpg',
 'data/train/c8/img_65519.jpg',
 'data/train/c8/img_92263.jpg',
 'data/train/c8/img_32089.jpg',
 'data/train/c8/img_41149.jpg',
 'data/train/c8/img_77385.jpg',
 'data/train/c8/img_36658.jpg',
 'data/train/c8/img_78465.jpg',
 'data/train/c8/img_93499.jpg',
 'data/train/c8/img_63867.jpg',
 'data/train/c8/img_20770.jpg',
 'data/train/c8/img_57232.jpg',
 'data/train/c8/img_65976.jpg',
 'data/train/c8/img_8759.jpg',
 'data/train/c8/img_86493.jpg',
 'data/train/c8/img_43230.jpg',
 'data/train/c8/img_91228.jpg',
 'data/train/c8/img_30425.jpg',
 'data/train/c8/img_11182.jpg',
 'data/train/c8/img_100735.jpg',
 'data/train/c8/img_49772.jpg',
 'data/train/c8/img_47271.jpg',
 'data/train/c8/img_39068.jpg',
 'data/train/c8/img_94630.jpg',
 'data/train/c8/img_36109.jpg',
 'data/train/c8/img_8658.jpg',
 'data/train/c8/img_40874.jpg',
 'data/train/c8/img_86037.jpg',
 'data/train/c8/img_30638.jpg',
 'data/train/c8/img_58218.jpg',
 'data/train/c8/img_44416.jpg',
 'data/train/c8/img_16728.jpg',
 'data/train/c8/img_31101.jpg',
 'data/train/c8/img_1062.jpg',
 'data/train/c8/img_95290.jpg',
 'data/train/c8/img_102095.jpg',
 'data/train/c8/img_34783.jpg',
 'data/train/c8/img_6117.jpg',
 'data/train/c8/img_53839.jpg',
 'data/train/c8/img_59130.jpg',
 'data/train/c8/img_19450.jpg',
 'data/train/c8/img_42704.jpg',
 'data/train/c8/img_56991.jpg',
 'data/train/c8/img_88458.jpg',
 'data/train/c8/img_85742.jpg',
 'data/train/c8/img_67820.jpg',
 'data/train/c8/img_50224.jpg',
 'data/train/c8/img_22805.jpg',
 'data/train/c8/img_65562.jpg',
 'data/train/c8/img_90873.jpg',
 'data/train/c8/img_32914.jpg',
 'data/train/c8/img_75654.jpg',
 'data/train/c8/img_16820.jpg',
 'data/train/c8/img_98810.jpg',
 'data/train/c8/img_60467.jpg',
 'data/train/c8/img_8649.jpg',
 'data/train/c8/img_65016.jpg',
 'data/train/c8/img_19530.jpg',
 'data/train/c8/img_98568.jpg',
 'data/train/c8/img_69584.jpg',
 'data/train/c8/img_12485.jpg',
 'data/train/c8/img_57199.jpg',
 'data/train/c8/img_75351.jpg',
 'data/train/c8/img_36371.jpg',
 'data/train/c8/img_61610.jpg',
 'data/train/c8/img_100368.jpg',
 'data/train/c8/img_75545.jpg',
 'data/train/c8/img_22133.jpg',
 'data/train/c8/img_47469.jpg',
 'data/train/c8/img_72630.jpg',
 'data/train/c8/img_62126.jpg',
 'data/train/c8/img_15381.jpg',
 'data/train/c8/img_90242.jpg',
 'data/train/c8/img_39467.jpg',
 'data/train/c8/img_90621.jpg',
 'data/train/c8/img_34176.jpg',
 'data/train/c8/img_77901.jpg',
 'data/train/c8/img_45869.jpg',
 'data/train/c8/img_55795.jpg',
 'data/train/c8/img_28537.jpg',
 'data/train/c8/img_48517.jpg',
 'data/train/c8/img_79514.jpg',
 'data/train/c8/img_6916.jpg',
 'data/train/c8/img_72072.jpg',
 'data/train/c8/img_75572.jpg',
 'data/train/c8/img_1338.jpg',
 'data/train/c8/img_41546.jpg',
 'data/train/c8/img_98681.jpg',
 'data/train/c8/img_54303.jpg',
 'data/train/c8/img_41462.jpg',
 'data/train/c8/img_51467.jpg',
 'data/train/c8/img_82047.jpg',
 'data/train/c8/img_71404.jpg',
 'data/train/c8/img_75915.jpg',
 'data/train/c8/img_7056.jpg',
 'data/train/c8/img_27314.jpg',
 'data/train/c8/img_91565.jpg',
 'data/train/c8/img_23721.jpg',
 'data/train/c8/img_73931.jpg',
 'data/train/c8/img_82020.jpg',
 'data/train/c8/img_24481.jpg',
 'data/train/c8/img_47167.jpg',
 'data/train/c8/img_11769.jpg',
 'data/train/c8/img_12575.jpg',
 'data/train/c8/img_23365.jpg',
 'data/train/c8/img_65563.jpg',
 'data/train/c8/img_24647.jpg',
 'data/train/c8/img_50668.jpg',
 'data/train/c8/img_68849.jpg',
 'data/train/c8/img_74133.jpg',
 'data/train/c8/img_62702.jpg',
 'data/train/c8/img_73702.jpg',
 'data/train/c8/img_85516.jpg',
 'data/train/c8/img_72641.jpg',
 'data/train/c8/img_75428.jpg',
 'data/train/c8/img_8547.jpg',
 'data/train/c8/img_92543.jpg',
 'data/train/c8/img_58162.jpg',
 'data/train/c8/img_11339.jpg',
 'data/train/c8/img_67682.jpg',
 'data/train/c8/img_23737.jpg',
 'data/train/c8/img_36411.jpg',
 'data/train/c8/img_7974.jpg',
 'data/train/c8/img_57619.jpg',
 'data/train/c8/img_44369.jpg',
 'data/train/c8/img_64091.jpg',
 'data/train/c8/img_18970.jpg',
 'data/train/c8/img_27544.jpg',
 'data/train/c8/img_34728.jpg',
 'data/train/c8/img_75161.jpg',
 'data/train/c8/img_84414.jpg',
 'data/train/c8/img_26265.jpg',
 'data/train/c8/img_69333.jpg',
 'data/train/c8/img_26400.jpg',
 'data/train/c8/img_26772.jpg',
 'data/train/c8/img_76845.jpg',
 'data/train/c8/img_13213.jpg',
 'data/train/c8/img_90151.jpg',
 'data/train/c8/img_96566.jpg',
 'data/train/c8/img_47986.jpg',
 'data/train/c8/img_85990.jpg',
 'data/train/c8/img_13505.jpg',
 'data/train/c8/img_80048.jpg',
 'data/train/c8/img_52947.jpg',
 'data/train/c8/img_70956.jpg',
 'data/train/c8/img_76082.jpg',
 'data/train/c8/img_18817.jpg',
 'data/train/c8/img_11924.jpg',
 'data/train/c8/img_32349.jpg',
 'data/train/c8/img_57525.jpg',
 'data/train/c8/img_94297.jpg',
 'data/train/c8/img_92021.jpg',
 'data/train/c8/img_43521.jpg',
 'data/train/c8/img_51539.jpg',
 'data/train/c8/img_10273.jpg',
 'data/train/c8/img_97733.jpg',
 'data/train/c8/img_57737.jpg',
 'data/train/c8/img_44267.jpg',
 'data/train/c8/img_59260.jpg',
 'data/train/c8/img_34109.jpg',
 'data/train/c8/img_37479.jpg',
 'data/train/c8/img_7932.jpg',
 'data/train/c8/img_49457.jpg',
 'data/train/c8/img_29553.jpg',
 'data/train/c8/img_90126.jpg',
 'data/train/c8/img_90786.jpg',
 'data/train/c8/img_13480.jpg',
 'data/train/c8/img_23521.jpg',
 'data/train/c8/img_41757.jpg',
 'data/train/c8/img_85996.jpg',
 'data/train/c8/img_7128.jpg',
 'data/train/c8/img_91096.jpg',
 'data/train/c8/img_10080.jpg',
 'data/train/c8/img_43569.jpg',
 'data/train/c8/img_64367.jpg',
 'data/train/c8/img_95736.jpg',
 'data/train/c8/img_11508.jpg',
 'data/train/c8/img_9581.jpg',
 'data/train/c8/img_11417.jpg',
 'data/train/c8/img_64987.jpg',
 'data/train/c8/img_26130.jpg',
 'data/train/c8/img_6374.jpg',
 'data/train/c8/img_88202.jpg',
 'data/train/c8/img_83361.jpg',
 'data/train/c8/img_94413.jpg',
 'data/train/c8/img_23776.jpg',
 'data/train/c8/img_72814.jpg',
 'data/train/c8/img_6399.jpg',
 'data/train/c8/img_92372.jpg',
 'data/train/c8/img_78749.jpg',
 'data/train/c8/img_50325.jpg',
 'data/train/c8/img_5978.jpg',
 'data/train/c8/img_21182.jpg',
 'data/train/c8/img_59872.jpg',
 'data/train/c8/img_25438.jpg',
 'data/train/c8/img_12358.jpg',
 'data/train/c8/img_78799.jpg',
 'data/train/c8/img_8431.jpg',
 'data/train/c8/img_39929.jpg',
 'data/train/c8/img_39194.jpg',
 'data/train/c8/img_96819.jpg',
 'data/train/c8/img_75735.jpg',
 'data/train/c8/img_46259.jpg',
 'data/train/c8/img_43411.jpg',
 'data/train/c8/img_90536.jpg',
 'data/train/c8/img_54199.jpg',
 'data/train/c8/img_29381.jpg',
 'data/train/c8/img_72684.jpg',
 'data/train/c8/img_67207.jpg',
 'data/train/c8/img_90567.jpg',
 'data/train/c8/img_38109.jpg',
 'data/train/c8/img_85625.jpg',
 'data/train/c8/img_15408.jpg',
 'data/train/c8/img_11798.jpg',
 'data/train/c8/img_15244.jpg',
 'data/train/c8/img_78438.jpg',
 'data/train/c8/img_3131.jpg',
 'data/train/c8/img_63579.jpg',
 'data/train/c8/img_37003.jpg',
 'data/train/c8/img_28231.jpg',
 'data/train/c8/img_31951.jpg',
 'data/train/c8/img_47709.jpg',
 'data/train/c8/img_24834.jpg',
 'data/train/c8/img_12704.jpg',
 'data/train/c8/img_99577.jpg',
 'data/train/c8/img_86528.jpg',
 'data/train/c8/img_81850.jpg',
 'data/train/c8/img_35237.jpg',
 'data/train/c8/img_80083.jpg',
 'data/train/c8/img_90239.jpg',
 'data/train/c8/img_54567.jpg',
 'data/train/c8/img_31158.jpg',
 'data/train/c8/img_67527.jpg',
 'data/train/c8/img_75427.jpg',
 'data/train/c8/img_102047.jpg',
 'data/train/c8/img_57014.jpg',
 'data/train/c8/img_48621.jpg',
 'data/train/c8/img_2902.jpg',
 'data/train/c8/img_39422.jpg',
 'data/train/c8/img_2409.jpg',
 'data/train/c8/img_77599.jpg',
 'data/train/c8/img_91792.jpg',
 'data/train/c8/img_46663.jpg',
 'data/train/c8/img_89648.jpg',
 'data/train/c8/img_75937.jpg',
 'data/train/c8/img_48897.jpg',
 'data/train/c8/img_81314.jpg',
 'data/train/c8/img_69491.jpg',
 'data/train/c8/img_96683.jpg',
 'data/train/c8/img_91289.jpg',
 'data/train/c8/img_99982.jpg',
 'data/train/c8/img_79148.jpg',
 'data/train/c8/img_48213.jpg',
 'data/train/c8/img_32768.jpg',
 'data/train/c8/img_11990.jpg',
 'data/train/c8/img_86899.jpg',
 'data/train/c8/img_3149.jpg',
 'data/train/c8/img_21087.jpg',
 'data/train/c8/img_28340.jpg',
 'data/train/c8/img_64291.jpg',
 'data/train/c8/img_50348.jpg',
 'data/train/c8/img_28224.jpg',
 'data/train/c8/img_74910.jpg',
 'data/train/c8/img_66070.jpg',
 'data/train/c8/img_3967.jpg',
 'data/train/c8/img_58907.jpg',
 'data/train/c8/img_99438.jpg',
 'data/train/c8/img_20780.jpg',
 'data/train/c8/img_81710.jpg',
 'data/train/c8/img_76717.jpg',
 'data/train/c8/img_75770.jpg',
 'data/train/c8/img_24040.jpg',
 'data/train/c8/img_50033.jpg',
 'data/train/c8/img_25660.jpg',
 'data/train/c8/img_35959.jpg',
 'data/train/c8/img_79307.jpg',
 'data/train/c8/img_57400.jpg',
 'data/train/c8/img_90746.jpg',
 'data/train/c8/img_83156.jpg',
 'data/train/c8/img_10867.jpg',
 'data/train/c8/img_93942.jpg',
 'data/train/c8/img_92391.jpg',
 'data/train/c8/img_51222.jpg',
 'data/train/c8/img_2832.jpg',
 'data/train/c8/img_49471.jpg',
 'data/train/c8/img_93212.jpg',
 'data/train/c8/img_88266.jpg',
 'data/train/c8/img_13807.jpg',
 'data/train/c8/img_35144.jpg',
 'data/train/c8/img_48605.jpg',
 'data/train/c8/img_6970.jpg',
 'data/train/c8/img_88877.jpg',
 'data/train/c8/img_52847.jpg',
 'data/train/c8/img_46758.jpg',
 'data/train/c8/img_64759.jpg',
 'data/train/c8/img_26763.jpg',
 'data/train/c8/img_48859.jpg',
 'data/train/c8/img_85352.jpg',
 'data/train/c8/img_96586.jpg',
 'data/train/c8/img_81654.jpg',
 'data/train/c8/img_44407.jpg',
 'data/train/c8/img_95193.jpg',
 'data/train/c8/img_13465.jpg',
 'data/train/c8/img_16283.jpg',
 'data/train/c8/img_27991.jpg',
 'data/train/c8/img_101984.jpg',
 'data/train/c8/img_77484.jpg',
 'data/train/c8/img_89464.jpg',
 'data/train/c8/img_52386.jpg',
 'data/train/c8/img_70560.jpg',
 'data/train/c8/img_26585.jpg',
 'data/train/c8/img_44088.jpg',
 'data/train/c8/img_30760.jpg',
 'data/train/c8/img_66306.jpg',
 'data/train/c8/img_70050.jpg',
 'data/train/c8/img_29002.jpg',
 'data/train/c8/img_41152.jpg',
 'data/train/c8/img_19204.jpg',
 'data/train/c8/img_63453.jpg',
 'data/train/c8/img_35460.jpg',
 'data/train/c8/img_81916.jpg',
 'data/train/c8/img_23011.jpg',
 'data/train/c8/img_18476.jpg',
 'data/train/c8/img_97010.jpg',
 'data/train/c8/img_25081.jpg',
 'data/train/c8/img_12294.jpg',
 'data/train/c8/img_29627.jpg',
 'data/train/c8/img_85945.jpg',
 'data/train/c8/img_16143.jpg',
 'data/train/c8/img_46910.jpg',
 'data/train/c8/img_5430.jpg',
 'data/train/c8/img_20676.jpg',
 'data/train/c8/img_102005.jpg',
 'data/train/c8/img_43861.jpg',
 'data/train/c8/img_16114.jpg',
 'data/train/c8/img_92690.jpg',
 'data/train/c8/img_19585.jpg',
 'data/train/c8/img_100816.jpg',
 'data/train/c8/img_13438.jpg',
 'data/train/c8/img_72778.jpg',
 'data/train/c8/img_28664.jpg',
 'data/train/c8/img_21946.jpg',
 'data/train/c8/img_92054.jpg',
 'data/train/c8/img_101367.jpg',
 'data/train/c8/img_86824.jpg',
 'data/train/c8/img_7287.jpg',
 'data/train/c8/img_93552.jpg',
 'data/train/c8/img_99869.jpg',
 'data/train/c8/img_67067.jpg',
 'data/train/c8/img_30792.jpg',
 'data/train/c8/img_97952.jpg',
 'data/train/c8/img_15437.jpg',
 'data/train/c8/img_72229.jpg',
 'data/train/c8/img_70539.jpg',
 'data/train/c8/img_101229.jpg',
 'data/train/c8/img_47652.jpg',
 'data/train/c8/img_39039.jpg',
 'data/train/c8/img_64914.jpg',
 'data/train/c8/img_89739.jpg',
 'data/train/c8/img_87738.jpg',
 'data/train/c8/img_6600.jpg',
 'data/train/c8/img_47890.jpg',
 'data/train/c8/img_19224.jpg',
 'data/train/c8/img_41461.jpg',
 'data/train/c8/img_15918.jpg',
 'data/train/c8/img_3632.jpg',
 'data/train/c8/img_3948.jpg',
 'data/train/c8/img_11529.jpg',
 'data/train/c8/img_97910.jpg',
 'data/train/c8/img_52162.jpg',
 'data/train/c8/img_20728.jpg',
 'data/train/c8/img_96028.jpg',
 'data/train/c8/img_9449.jpg',
 'data/train/c8/img_56616.jpg',
 'data/train/c8/img_46712.jpg',
 'data/train/c8/img_76663.jpg',
 'data/train/c8/img_76714.jpg',
 'data/train/c8/img_58700.jpg',
 'data/train/c8/img_52567.jpg',
 'data/train/c8/img_16529.jpg',
 'data/train/c8/img_75474.jpg',
 'data/train/c8/img_32672.jpg',
 'data/train/c8/img_21663.jpg',
 'data/train/c8/img_15878.jpg',
 'data/train/c8/img_53812.jpg',
 'data/train/c8/img_94016.jpg',
 'data/train/c8/img_67360.jpg',
 'data/train/c8/img_51392.jpg',
 'data/train/c8/img_26896.jpg',
 'data/train/c8/img_44652.jpg',
 'data/train/c8/img_36898.jpg',
 'data/train/c8/img_18737.jpg',
 'data/train/c8/img_26454.jpg',
 'data/train/c8/img_4238.jpg',
 'data/train/c8/img_33071.jpg',
 'data/train/c8/img_63438.jpg',
 'data/train/c8/img_26672.jpg',
 'data/train/c8/img_73629.jpg',
 'data/train/c8/img_27146.jpg',
 'data/train/c8/img_23170.jpg',
 'data/train/c8/img_90894.jpg',
 'data/train/c8/img_76358.jpg',
 'data/train/c8/img_93860.jpg',
 'data/train/c8/img_85043.jpg',
 'data/train/c8/img_54660.jpg',
 'data/train/c8/img_33479.jpg',
 'data/train/c8/img_88674.jpg',
 'data/train/c8/img_50196.jpg',
 'data/train/c8/img_12069.jpg',
 'data/train/c8/img_47390.jpg',
 'data/train/c8/img_96565.jpg',
 'data/train/c8/img_42899.jpg',
 'data/train/c8/img_11896.jpg',
 'data/train/c8/img_69590.jpg',
 'data/train/c8/img_50865.jpg',
 'data/train/c8/img_14402.jpg',
 'data/train/c8/img_100715.jpg',
 'data/train/c8/img_36833.jpg',
 'data/train/c8/img_92262.jpg',
 'data/train/c8/img_46830.jpg',
 'data/train/c8/img_67447.jpg',
 'data/train/c8/img_44710.jpg',
 'data/train/c8/img_47801.jpg',
 'data/train/c8/img_81770.jpg',
 'data/train/c8/img_26143.jpg',
 'data/train/c8/img_46716.jpg',
 'data/train/c8/img_50462.jpg',
 'data/train/c8/img_89556.jpg',
 'data/train/c8/img_41429.jpg',
 'data/train/c8/img_60384.jpg',
 'data/train/c8/img_82536.jpg',
 'data/train/c8/img_38121.jpg',
 'data/train/c8/img_37519.jpg',
 'data/train/c8/img_61550.jpg',
 'data/train/c8/img_50570.jpg',
 'data/train/c8/img_65947.jpg',
 'data/train/c8/img_88106.jpg',
 'data/train/c8/img_26381.jpg',
 'data/train/c8/img_48627.jpg',
 'data/train/c8/img_75673.jpg',
 'data/train/c8/img_90486.jpg',
 'data/train/c8/img_1493.jpg',
 'data/train/c8/img_58137.jpg',
 'data/train/c8/img_77428.jpg',
 'data/train/c8/img_33499.jpg',
 'data/train/c8/img_9499.jpg',
 'data/train/c8/img_61599.jpg',
 'data/train/c8/img_73484.jpg',
 'data/train/c8/img_47117.jpg',
 'data/train/c8/img_9212.jpg',
 'data/train/c8/img_84602.jpg',
 'data/train/c8/img_36882.jpg',
 'data/train/c8/img_97731.jpg',
 'data/train/c8/img_70019.jpg',
 'data/train/c8/img_9104.jpg',
 'data/train/c8/img_28854.jpg',
 'data/train/c8/img_7559.jpg',
 'data/train/c8/img_46920.jpg',
 'data/train/c8/img_84449.jpg',
 'data/train/c8/img_24529.jpg',
 'data/train/c8/img_68387.jpg',
 'data/train/c8/img_72343.jpg',
 'data/train/c8/img_32145.jpg',
 'data/train/c8/img_56543.jpg',
 'data/train/c8/img_80528.jpg',
 'data/train/c8/img_6462.jpg',
 'data/train/c8/img_40284.jpg',
 'data/train/c8/img_97122.jpg',
 'data/train/c8/img_24754.jpg',
 'data/train/c8/img_89588.jpg',
 'data/train/c8/img_22194.jpg',
 'data/train/c8/img_35435.jpg',
 'data/train/c8/img_97624.jpg',
 'data/train/c8/img_93577.jpg',
 'data/train/c8/img_19323.jpg',
 'data/train/c8/img_87982.jpg',
 'data/train/c8/img_87278.jpg',
 'data/train/c8/img_90349.jpg',
 'data/train/c8/img_11557.jpg',
 'data/train/c8/img_43441.jpg',
 'data/train/c8/img_64193.jpg',
 'data/train/c8/img_62613.jpg',
 'data/train/c8/img_83735.jpg',
 'data/train/c8/img_9792.jpg',
 'data/train/c8/img_98501.jpg',
 'data/train/c8/img_40824.jpg',
 'data/train/c8/img_63365.jpg',
 ...]

Set up a simple model

Set up the batches



In [7]:

    
# Grab a few images at a time for training and validation.
# NB: They must be in subdirectories named based on their category
print(data_path+'train')
batch_size = 64
batches = get_batches(data_path+'train', batch_size = batch_size)
val_batches = get_batches(data_path+'valid', batch_size = batch_size*2, shuffle=False)









    



data/train
Found 18587 images belonging to 10 classes.
Found 3837 images belonging to 10 classes.



In [8]:

    
(val_classes, trn_classes, val_labels, trn_labels, val_filenames, filenames, test_filenames) = get_classes(data_path)









    



Found 18587 images belonging to 10 classes.
Found 3837 images belonging to 10 classes.
Found 79726 images belonging to 1 classes.

Inspecting the imported class data

Let's make some sense of what that last command achieved. Looking at the vgg (fast.ai) code from which it comes, it extracts information from the json file that comes with imagenet.



In [65]:

    
print('val_classes:', len(val_classes))
print(val_classes[454:458])
print(val_labels[454:458])
print(val_filenames[454:458])
print('trn_classes:', len(trn_classes))
print(trn_classes[2031:2034])
print(trn_labels[2031:2034])
print('filenames:', len(filenames))
print(filenames[2031:2034])
print('test_filenames:', len(test_filenames))
print(test_filenames[2031:2034])









    



val_classes: 3837
[0 0 1 1]
[[ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]]
['c0/img_75810.jpg', 'c0/img_25783.jpg', 'c1/img_59482.jpg', 'c1/img_86904.jpg']
trn_classes: 18587
[0 0 1]
[[ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
 [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]]
filenames: 18587
['c0/img_57235.jpg', 'c0/img_88177.jpg', 'c1/img_80626.jpg']
test_filenames: 79726
['unknown/img_74200.jpg', 'unknown/img_49022.jpg', 'unknown/img_39851.jpg']



In [ ]:

Importing the data

The lines below need only be run once (they take 4 minutes) - on following sessions, one can simply load the arrays (see the cell below it).



In [67]:

    
trn = get_data(data_path + 'train')
val = get_data(data_path + 'valid')









    



Found 18587 images belonging to 10 classes.
Found 3837 images belonging to 10 classes.

Save the data arrays, for faster reloading.



In [68]:

    
save_array(data_path + 'results/val.dat', val)
save_array(data_path + 'results/trn.dat', trn)



In [9]:

    
val = load_array(data_path + 'results/val.dat')
trn = load_array(data_path + 'results/trn.dat')

Experiments

A single conv layer



In [7]:

    
def conv1(batches):
    model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,224,224)),
            Convolution2D(32,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Convolution2D(64,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Flatten(),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dense(10, activation='softmax')
        ])

    model.compile(Adam(lr=1e-4), loss='categorical_crossentropy', metrics=['accuracy'])
    model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                     nb_val_samples=val_batches.nb_sample)
    model.optimizer.lr = 0.001
    model.fit_generator(batches, batches.nb_sample, nb_epoch=4, validation_data=val_batches, 
                     nb_val_samples=val_batches.nb_sample)
    return model

Train the model



In [11]:

    
model = conv1(batches)









    



Epoch 1/2
18587/18587 [==============================] - 202s - loss: 0.2188 - acc: 0.9461 - val_loss: 4.1314 - val_acc: 0.1337
Epoch 2/2
18587/18587 [==============================] - 196s - loss: 0.0137 - acc: 0.9985 - val_loss: 2.8235 - val_acc: 0.2374
Epoch 1/4
18587/18587 [==============================] - 214s - loss: 0.0063 - acc: 0.9995 - val_loss: 2.9924 - val_acc: 0.2637
Epoch 2/4
18587/18587 [==============================] - 207s - loss: 0.0028 - acc: 0.9999 - val_loss: 2.9473 - val_acc: 0.2353
Epoch 3/4
18587/18587 [==============================] - 193s - loss: 0.0018 - acc: 0.9999 - val_loss: 3.0685 - val_acc: 0.2531
Epoch 4/4
18587/18587 [==============================] - 198s - loss: 0.0016 - acc: 0.9998 - val_loss: 3.2383 - val_acc: 0.2651

Discussion

I'm really puzzled by this, on two counts (first two points are same issue?).

The model took almost 2x as long as the fast.ai notebook, when it usually runs much faster.
The temperature profiles were the opposite of normal - this time it was the CPU that ran hot. The GPU barely hit 37deg.
More importantly, the val_acc was far lower than Howerd's (shown below).



In [17]:

    
On further reflection, I suspect that the simplicity of the model explains the temperature differences.









    



  File "<ipython-input-17-fc08a237f549>", line 1
    On further reflection, I suspect that the simplicity of the model explains the temperature differences.
             ^
SyntaxError: invalid syntax

## HOWARD's results! Epoch 1/2 18946/18946 [==============================] - 114s - loss: 0.2273 - acc: 0.9405 - val_loss: 2.4946 - val_acc: 0.2826 Epoch 2/2 18946/18946 [==============================] - 114s - loss: 0.0120 - acc: 0.9990 - val_loss: 1.5872 - val_acc: 0.5253 Epoch 1/4 18946/18946 [==============================] - 114s - loss: 0.0093 - acc: 0.9992 - val_loss: 1.4836 - val_acc: 0.5825 Epoch 2/4 18946/18946 [==============================] - 114s - loss: 0.0032 - acc: 1.0000 - val_loss: 1.3142 - val_acc: 0.6162 Epoch 3/4 18946/18946 [==============================] - 114s - loss: 0.0035 - acc: 0.9996 - val_loss: 1.5061 - val_acc: 0.5771 Epoch 4/4 18946/18946 [==============================] - 114s - loss: 0.0036 - acc: 0.9997 - val_loss: 1.4528 - val_acc: 0.5808

One possibility is that Howard used a different approach to setting up the validation set. Hypothesis: I ensured that the same drivers could not be found in both sets, but he didn't? One way to test this would be to see what I get against the Kaggle test set.

Augment the data

Data augmentation is really a way to ensure that the model doesn't overfit on some fairly simple characteristics. It's built in to Keras.



In [9]:

    
batch_size = 64
gen_t = image.ImageDataGenerator(channel_shift_range=20, height_shift_range=0.05, 
                                 rotation_range=15, shear_range=0.1, width_shift_range=0.1)  
batches = get_batches(data_path+'train', gen_t, batch_size=batch_size)









    



Found 18587 images belonging to 10 classes.



In [13]:

    
model = conv1(batches)









    



Epoch 1/2
18587/18587 [==============================] - 211s - loss: 1.2281 - acc: 0.6060 - val_loss: 2.3168 - val_acc: 0.3378
Epoch 2/2
18587/18587 [==============================] - 201s - loss: 0.5687 - acc: 0.8308 - val_loss: 2.1443 - val_acc: 0.3909
Epoch 1/4
18587/18587 [==============================] - 206s - loss: 0.3892 - acc: 0.8892 - val_loss: 2.0790 - val_acc: 0.4217
Epoch 2/4
18587/18587 [==============================] - 202s - loss: 0.2957 - acc: 0.9190 - val_loss: 1.8599 - val_acc: 0.4793
Epoch 3/4
18587/18587 [==============================] - 201s - loss: 0.2398 - acc: 0.9375 - val_loss: 2.2379 - val_acc: 0.4321
Epoch 4/4
18587/18587 [==============================] - 199s - loss: 0.2005 - acc: 0.9475 - val_loss: 2.3212 - val_acc: 0.4384

Discussion

The weird temps haven't changed, but the amount of time that an epoch takes has gone up. This makes sense. Not sure whether raising the batch size might increase overfitting, though?
By Epoch 2-4, train-acc is lower and val_acc is higher, so overfitting has been reduced. Good.

At this point, JH reduces the training rate and re-runs the same model and data. Perhaps it would be smarter to reduce the LR after the second epoch in the second set?

Employ JH's model with an adapted training process



In [14]:

    
def conv2(batches):
    model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,224,224)),
            Convolution2D(32,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Convolution2D(64,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D((3,3)),
            Flatten(),
            Dense(200, activation='relu'),
            BatchNormalization(),
            Dense(10, activation='softmax')
        ])

    model.compile(Adam(lr=1e-4), loss='categorical_crossentropy', metrics=['accuracy'])
    model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                     nb_val_samples=val_batches.nb_sample)
    model.optimizer.lr = 0.001
    # 2, not 4, epochs here - MDR
    model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                     nb_val_samples=val_batches.nb_sample)
    model.optimizer.lr = 0.0001
    # added - MDR
    model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                     nb_val_samples=val_batches.nb_sample)
    return model



In [16]:

    
batch_size = 128
model = conv2(batches)









    



Epoch 1/2
18587/18587 [==============================] - 207s - loss: 1.1820 - acc: 0.6210 - val_loss: 2.0422 - val_acc: 0.3193
Epoch 2/2
18587/18587 [==============================] - 203s - loss: 0.5783 - acc: 0.8243 - val_loss: 2.5763 - val_acc: 0.3683
Epoch 1/2
18587/18587 [==============================] - 207s - loss: 0.4063 - acc: 0.8797 - val_loss: 2.0120 - val_acc: 0.4613
Epoch 2/2
18587/18587 [==============================] - 200s - loss: 0.3102 - acc: 0.9133 - val_loss: 2.0487 - val_acc: 0.4746
Epoch 1/2
18587/18587 [==============================] - 207s - loss: 0.2486 - acc: 0.9335 - val_loss: 1.9054 - val_acc: 0.4730
Epoch 2/2
18587/18587 [==============================] - 202s - loss: 0.2082 - acc: 0.9432 - val_loss: 1.9443 - val_acc: 0.4848

One other change - I experimented with turning the GPU fan back to automatic settings. And then turned it back to manual (75%)

Discussion

val_acc increased monotonically throughout this experiment - GOOD.
acc hit a fairly high point, though (.9432), so we're still overfitting.
Re the fan: temps did increase, when I turned it to auto. They rose by 15deg (35-50deg), but this took five minutes. This steady rise is still very much slower than the profile for VGG, but it's reassuring to see that it did happen. Putting the fan back on dropped the temps again - but not quite to the previous baseline (at least, not within 12 mins).
JH notes that his results were very 'unstable' - the val_acc was jumping around (unlike mine).

A deeper architecture: four conv-pooling pairs, plus dropout

Three things are changed, here (plus one repeal of a change I made above, and now doubt the reason of):

the architecture has changed: it has more conv and pool layers, and it has dropout
the data augmentation params have been adjusted, compared to those used above
I've switched the batch_size back to 64, to match that used in setting up the batches at the top of the notebook - shouldn't they match?



In [18]:

    
gen_t = image.ImageDataGenerator(channel_shift_range=20, height_shift_range=0.05, 
                                 rotation_range=15, shear_range=0.1, width_shift_range=0.1)  
batches = get_batches(data_path+'train', gen_t, batch_size=batch_size)









    



Found 18587 images belonging to 10 classes.

Note - this time, we haven't wrapped the model in a function (that returns a model). Both ways work.



In [19]:

    
model = Sequential([
        BatchNormalization(axis=1, input_shape=(3,224,224)),
        Convolution2D(32,3,3, activation='relu'),
        BatchNormalization(axis=1),
        MaxPooling2D(),                               # before, he defined the size (3,3)
        Convolution2D(64,3,3, activation='relu'),
        BatchNormalization(axis=1),
        MaxPooling2D(),
        Convolution2D(128,3,3, activation='relu'),    # new
        BatchNormalization(axis=1),                   # new
        MaxPooling2D(),                               # new
        Flatten(),              
        Dense(200, activation='relu'),                # new
        BatchNormalization(),                         # new
        Dropout(0.5),                                 # new
        Dense(200, activation='relu'),
        BatchNormalization(),
        Dropout(0.5),                                 # new
        Dense(10, activation='softmax')
    ])

JH's LR is really low, here.



In [20]:

    
model.compile(Adam(lr=0.00001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                    nb_val_samples=val_batches.nb_sample)









    



Epoch 1/2
18587/18587 [==============================] - 215s - loss: 3.1516 - acc: 0.1628 - val_loss: 2.4575 - val_acc: 0.1168
Epoch 2/2
18587/18587 [==============================] - 202s - loss: 2.6143 - acc: 0.2531 - val_loss: 2.5248 - val_acc: 0.1267






    Out[20]:





<keras.callbacks.History at 0x7fb59bbf3b00>

Surely way too slow!



In [21]:

    
model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                    nb_val_samples=val_batches.nb_sample)









    



Epoch 1/2
18587/18587 [==============================] - 216s - loss: 2.1611 - acc: 0.3356 - val_loss: 2.4113 - val_acc: 0.2695
Epoch 2/2
18587/18587 [==============================] - 203s - loss: 1.1126 - acc: 0.6193 - val_loss: 1.9400 - val_acc: 0.4368






    Out[21]:





<keras.callbacks.History at 0x7fb599af08d0>



In [22]:

    
model.optimizer.lr = 0.0001
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                    nb_val_samples=val_batches.nb_sample)









    



Epoch 1/2
18587/18587 [==============================] - 216s - loss: 0.6640 - acc: 0.7776 - val_loss: 2.0587 - val_acc: 0.4545
Epoch 2/2
18587/18587 [==============================] - 204s - loss: 0.4434 - acc: 0.8553 - val_loss: 2.4554 - val_acc: 0.3563






    Out[22]:





<keras.callbacks.History at 0x7fb598d6f208>

I wonder if it's worth keeping going?



In [23]:

    
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                    nb_val_samples=val_batches.nb_sample)









    



Epoch 1/2
18587/18587 [==============================] - 216s - loss: 0.3341 - acc: 0.8901 - val_loss: 1.9272 - val_acc: 0.5728
Epoch 2/2
18587/18587 [==============================] - 202s - loss: 0.2536 - acc: 0.9181 - val_loss: 1.5902 - val_acc: 0.5575






    Out[23]:





<keras.callbacks.History at 0x7fb598d6feb8>

Perhaps with a lower learning rate at this point...



In [24]:

    
model.optimizer.lr = 0.00001
model.fit_generator(batches, batches.nb_sample, nb_epoch=2, validation_data=val_batches, 
                    nb_val_samples=val_batches.nb_sample)









    



Epoch 1/2
18587/18587 [==============================] - 208s - loss: 0.2041 - acc: 0.9355 - val_loss: 1.9901 - val_acc: 0.5262
Epoch 2/2
18587/18587 [==============================] - 193s - loss: 0.1754 - acc: 0.9477 - val_loss: 1.4121 - val_acc: 0.6375






    Out[24]:





<keras.callbacks.History at 0x7fb599af0a20>

Discussion

I'm stopping this one here. It worked ok - 63% acc isn't bad, but we're overfitting still. It's likely that I'll have more luck applying the same techniques in fine-tuning VGG.



In [ ]: