Just like programming has "Hello World", machine learning has MNIST. MNIST is a simple computer vision dataset. It consists of images of handwritten digits, as well as labels for each image telling us which digit it is. In this tutorial, we're going to train a model to look at MNIST images and predict what digits they are.
This tutorial is taken, with slight modification and different annotations, from TensorFlow's official documentation.
This tutorial is intended for readers who are new to both machine learning and TensorFlow.
The MNIST dataset is hosted on Yann LeCun's website.
The following two lines of code will take care of downloading and reading in the data automatically.
mnist is a lightweight class which stores the training, validation, and testing sets as NumPy arrays, as well as providing a function for iterating through data minibatches.
In [1]:
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
First of all, we import the TensorFlow library.
In [2]:
import tensorflow as tf
We assign a shape of [None, 784], where 784 is the dimensionality of a single flattened 28 by 28 pixel MNIST image, and None indicates that the first dimension, corresponding to the batch size, can be of any size.
The target output classes y_ will also consist of a 2d tensor, where each row is a one-hot 10-dimensional vector (a one-hot vector is a vector which is 0 in most dimensions and 1 in a single dimension) indicating which digit class (zero to nine) the corresponding MNIST image belongs to.
Let's create some symbolic variables using TensorFlow's placeholder() operation. The result will not be a specific value; it will be a placeholder, a value that we'll input when we ask TensorFlow to run a computation.
In [3]:
x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
The input images x consists of a 2D tensor of floating point numbers. Here we assign it a shape of [None, 784], where 784 is the dimensionality of a single flattened 28 by 28 pixel MNIST image, and None indicates that the first dimension, corresponding to the batch size, can be of any size. The target output classes y_ also consists of a 2D tensor, where each row is a one-hot 10-dimensional vector indicating which digit class (zero through nine) the corresponding MNIST image belongs to.
Flattening the data throws away information about the 2D structure of the image. This is bad! We are going to use Convolutional Neural Networks (CNN), which is great in exploiting the 2D structure of an image. So, let's reshape x into a 28 x 28 matrix (the final dimension corresponding to the number of color channels).
In [4]:
x_image = tf.reshape(x, [-1,28,28,1])
To create this model, we're going to need to create a lot of weights and biases.
One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients.
Instead of doing this repeatedly while we build the model, let's create two handy functions to do it for us.
The resulting weights and biases will all be Variables. A Variable is a modifiable tensor that lives in TensorFlow's graph of interacting operations. It can be used and even modified by the computation. For machine learning applications, one generally has the model parameters be Variables. We create these Variables by giving tf.Variable() the initial value of the Variable
In [5]:
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
Our convolutions uses a stride of one and are zero padded so that the output is the same size as the input.
Our pooling is plain old max pooling over 2x2 blocks.
To keep our code cleaner, let's also abstract those operations into functions.
In [6]:
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
Build the Deep Learning architecture.
In [7]:
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
We know that every image in MNIST is of a handwritten digit between zero and nine, so we know there are only ten possible values that a given image can be.
We want to be able to look at an image and give the probabilities for it being each digit. For example, our model might look at a picture of a nine and be 80% sure it's a nine, but give a 5% chance to it being an eight (because of the top loop) and a bit of probability to all the others because it isn't 100% sure.
If you want to assign probabilities to an object being one of several different things, you want to use SoftMax: SoftMax gives us a list of values between 0 and 1 that add up to 1. A softmax regression has two steps:
For this reason, we use SoftMax as our last layer.
In [8]:
y_conv = tf.nn.softmax(tf.add(tf.matmul(h_fc1_drop, W_fc2), b_fc2))
Our loss function is the cross-entropy between the target and the softmax activation function applied to the model's prediction. In some rough sense, the cross-entropy is measuring how inefficient our predictions are for describing the truth, how large is the discrepancy between two probability distributions.
In [9]:
cross_entropy = - tf.reduce_sum(y_ * tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
How well does our model do? We have to figure out where we predicted the correct label. tf.argmax() is an extremely useful function which gives you the index of the highest entry in a tensor along some axis. tf.argmax(y,1) is the label our model thinks is most likely for each input, while tf.argmax(y_,1) is the correct label. We can use tf.equal to check if our prediction matches the truth.
In [10]:
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
That gives us a list of booleans. To determine what fraction are correct, we cast to floating point numbers and then take the mean. For example, [True, False, True, True] would become [1,0,1,1] which would become 0.75.
In [ ]:
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
Let's launch the graph!
Each step of the inner loop, we get a "batch" of one hundred random data points from our training set. We run train_step feeding in the batches data to replace the placeholders. Ideally, we'd like to use all our data for every step of training because that would give us a better sense of what we should be doing, but that's expensive. So, instead, we use a different subset every time. Doing this is cheap and has much of the same benefit.
In [ ]:
init = tf.initialize_all_variables()
sess = tf.Session()
with sess.as_default():
sess.run(init)
for i in range(20000):
batch = mnist.train.next_batch(50)
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
print("step %d, training accuracy %g" % (i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print("test accuracy %g"%accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))