Credits: Forked from TensorFlow by Google
Refer to the setup instructions.
The objective of this exercise is to learn about simple data curation practices, and familiarize you with some of the data we'll be reusing later.
This notebook uses the notMNIST dataset to be used with python experiments. This dataset is designed to look like the classic MNIST dataset, while looking a little more like real data: it's a harder task, and the data is a lot less 'clean' than MNIST.
In [1]:
print 'xxxx'
In [1]:
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
import matplotlib.pyplot as plt
import numpy as np
import os
import tarfile
import urllib
from IPython.display import display, Image
from scipy import ndimage
from sklearn.linear_model import LogisticRegression
import cPickle as pickle
First, we'll download the dataset to our local machine. The data consists of characters rendered in a variety of fonts on a 28x28 image. The labels are limited to 'A' through 'J' (10 classes). The training set has about 500k and the testset 19000 labelled examples. Given these sizes, it should be possible to train models quickly on any machine.
In [3]:
url = 'http://yaroslavvb.com/upload/notMNIST/'
def maybe_download(filename, expected_bytes):
"""Download a file if not present, and make sure it's the right size."""
if not os.path.exists(filename):
filename, _ = urllib.urlretrieve(url + filename, filename)
statinfo = os.stat(filename)
if statinfo.st_size == expected_bytes:
print 'Found and verified', filename
else:
raise Exception(
'Failed to verify' + filename + '. Can you get to it with a browser?')
return filename
train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)
Extract the dataset from the compressed .tar.gz file. This should give you a set of directories, labelled A through J.
In [4]:
num_classes = 10
def extract(filename):
# tar = tarfile.open(filename)
# tar.extractall()
# tar.close()
root = os.path.splitext(os.path.splitext(filename)[0])[0] # remove .tar.gz
data_folders = [os.path.join(root, d) for d in sorted(os.listdir(root))]
if len(data_folders) != num_classes:
raise Exception(
'Expected %d folders, one per class. Found %d instead.' % (
num_folders, len(data_folders)))
print data_folders
return data_folders
In [5]:
train_folders = extract(train_filename)
print train_folders
In [6]:
test_folders = extract(test_filename)
print test_folders
Now let's load the data in a more manageable format.
We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road. The labels will be stored into a separate array of integers 0 through 9.
A few images might not be readable, we'll just skip them.
In [7]:
parent_folder = './'
In [8]:
all_train_filles = []
for i in range(10):
folder = train_folders[i]
flist =os.listdir(parent_folder+folder)
for fname in flist :
all_train_filles.append((i,parent_folder+folder+'/'+fname))
In [9]:
from random import shuffle
print all_train_filles[:10]
shuffle(all_train_filles)
print all_train_filles[:10]
In [10]:
def getNext():
import numpy as np
s = 100
batch_szie = 100
image_size = 28
partial_data = all_train_filles[s:s+batch_szie]
lables = np.ndarray(shape=(batch_szie), dtype=np.int32)
dataset = np.ndarray(
shape=(batch_szie, image_size, image_size), dtype=np.float32)
image_index = 0
for label,fname in partial_data:
image_data = (ndimage.imread(fname).astype(float) -
128 / 2) / 128
# print image_data
if image_data.shape != (image_size, image_size):
raise Exception('Unexpected image shape: %s' % str(image_data.shape))
dataset[image_index,:,:] = image_data
lables[image_index] = label
image_index += 1
return lables,dataset
In [11]:
image_size = 28 # Pixel width and height.
pixel_depth = 255.0 # Number of levels per pixel.
def load(data_folders, min_num_images, max_num_images):
dataset = np.ndarray(
shape=(max_num_images, image_size, image_size), dtype=np.float32)
labels = np.ndarray(shape=(max_num_images), dtype=np.int32)
label_index = 0
image_index = 0
for folder in data_folders:
print folder
for image in os.listdir(folder):
if image_index >= max_num_images:
# raise Exception('More images than expected: %d >= %d' % (
# num_images, max_num_images))
print "Too Many Images",folder
break
image_file = os.path.join(folder, image)
try:
image_data = (ndimage.imread(image_file).astype(float) -
pixel_depth / 2) / pixel_depth
if image_data.shape != (image_size, image_size):
raise Exception('Unexpected image shape: %s' % str(image_data.shape))
dataset[image_index, :, :] = image_data
labels[image_index] = label_index
image_index += 1
except IOError as e:
print 'Could not read:', image_file, ':', e, '- it\'s ok, skipping.'
label_index += 1
num_images = image_index
dataset = dataset[0:num_images, :, :]
labels = labels[0:num_images]
if num_images < min_num_images:
raise Exception('Many fewer images than expected: %d < %d' % (
num_images, min_num_images))
print 'Full dataset tensor:', dataset.shape
print 'Mean:', np.mean(dataset)
print 'Standard deviation:', np.std(dataset)
print 'Labels:', labels.shape
return dataset, labels
train_dataset, train_labels = load(train_folders, 450000, 550000)
In [31]:
test_dataset, test_labels = load(test_folders, 18000, 20000)
Next, we'll randomize the data. It's important to have the labels well shuffled for the training and test distributions to match.
In [32]:
np.random.seed(133)
def randomize(dataset, labels):
permutation = np.random.permutation(labels.shape[0])
shuffled_dataset = dataset[permutation,:,:]
shuffled_labels = labels[permutation]
return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)
In [33]:
train_labels[:100]
Out[33]:
In [34]:
print 'train_dataset'
print train_dataset.shape
print train_dataset.mean()
print train_dataset.std()
print 'test_dataset'
print test_dataset.shape
print test_dataset.mean()
print test_dataset.std()
In [35]:
t_lable_count =[0]*10
for i in train_labels:
t_lable_count[i] +=1
In [36]:
t_lable_count
Out[36]:
In [37]:
test_lable_count =[0]*10
for i in test_labels:
test_lable_count[i] +=1
test_lable_count
Out[37]:
Prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed.
Also create a validation dataset for hyperparameter tuning.
In [38]:
train_size = 200000
valid_size = 10000
valid_dataset = train_dataset[:valid_size,:,:]
valid_labels = train_labels[:valid_size]
train_dataset = train_dataset[valid_size:valid_size+train_size,:,:]
train_labels = train_labels[valid_size:valid_size+train_size]
print 'Training', train_dataset.shape, train_labels.shape
print 'Validation', valid_dataset.shape, valid_labels.shape
Finally, let's save the data for later reuse:
In [39]:
pickle_file = 'notMNIST.pickle'
try:
f = open(pickle_file, 'wb')
save = {
'train_dataset': train_dataset,
'train_labels': train_labels,
'valid_dataset': valid_dataset,
'valid_labels': valid_labels,
'test_dataset': test_dataset,
'test_labels': test_labels,
}
pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
f.close()
except Exception as e:
print 'Unable to save data to', pickle_file, ':', e
raise
In [40]:
statinfo = os.stat(pickle_file)
print 'Compressed pickle size:', statinfo.st_size
In [2]:
pickle_file = 'notMNIST.pickle'
with open(pickle_file, 'rb') as f:
save = pickle.load(f)
train_dataset = save['train_dataset']
train_labels = save['train_labels']
valid_dataset = save['valid_dataset']
valid_labels = save['valid_labels']
test_dataset = save['test_dataset']
test_labels = save['test_labels']
del save # hint to help gc free up memory
print 'Training set', train_dataset.shape, train_labels.shape
print 'Validation set', valid_dataset.shape, valid_labels.shape
print 'Test set', test_dataset.shape, test_labels.shape
By construction, this dataset might contain a lot of overlapping samples, including training data that's also contained in the validation and test set! Overlap between training and test can skew the results if you expect to use your model in an environment where there is never an overlap, but are actually ok if you expect to see training samples recur when you use it. Measure how much overlap there is between training, validation and test samples. Optional questions:
Let's get an idea of what an off-the-shelf classifier can give you on this data. It's always good to check that there is something to learn, and that it's a problem that is not so trivial that a canned solution solves it.
Train a simple model on this data using 50, 100, 1000 and 5000 training samples. Hint: you can use the LogisticRegression model from sklearn.linear_model.
Optional question: train an off-the-shelf model on all the data!
In [3]:
import tensorflow as tf
In [4]:
x = tf.placeholder(tf.float32,shape=[None,28*28],name='Input_X')
y = tf.placeholder(tf.float32,shape=[None,10],name='Input_Y')
In [5]:
W = tf.Variable(tf.truncated_normal(shape=[28*28,10]))
b = tf.Variable(tf.truncated_normal(shape=[10]))
In [6]:
xw = tf.matmul(x,W)
r = xw + b
cost = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(r,y))
a = tf.nn.softmax(r)
# cost_array = y*tf.log(a)
# cost = -1*tf.reduce_sum(cost_array )/1000
In [7]:
op = tf.train.GradientDescentOptimizer(0.5).minimize(cost)
In [8]:
init = tf.initialize_all_variables()
In [17]:
session = tf.Session()
In [18]:
session.run(init)
In [19]:
batch_size =128
batch_list = []
for i in range(0,train_labels[:1024].shape[0]/batch_size):
# print (i*batch_size,i*batch_size+batch_size)
batch_list.append((i*batch_size,(i+1)*batch_size))
# batch_list.append(((i+1)*batch_size,train_labels.shape[0]-1))
In [20]:
# for start,end in batch_list:
start,end = batch_list[0]
print start,end
input_images = np.reshape(train_dataset[start:end],[batch_size,28*28])
output_labes = np.zeros([batch_size,10],dtype=np.int)
output_labes[100][9]
Out[20]:
In [21]:
for epcho in range(10000):
for start,end in batch_list:
for index,value in enumerate(train_labels[start:end]):
output_labes[index][value] = 1
session.run(op,feed_dict={x:input_images,y:output_labes})
if epcho % 100 == 0 :
print "epcho",epcho,
print session.run(cost,feed_dict={x:input_images,y:output_labes})/batch_size,
test_image = np.reshape(train_dataset[:1024],[-1,28*28])
prediction = tf.arg_max(a,1)
pr = session.run(prediction,feed_dict={x:test_image,})
print np.sum(np.equal(pr,train_labels[:1024])*1)*1./test_image.shape[0]
In [41]:
test_image = np.reshape(train_dataset,[-1,28*28])
prediction = tf.arg_max(a,1)
pr = session.run(prediction,feed_dict={x:test_image,})
np.sum(np.equal(pr,train_labels)*1)*1./test_image.shape[0]
Out[41]:
In [162]:
import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib
%matplotlib inline
import numpy
In [167]:
tmp2 = train_dataset[2222].reshape((28,28))
plt.imshow(tmp2, cmap = cm.Greys)
plt.show()
In [ ]: