A Network Tour of Data Science

Xavier Bresson, Winter 2016/17

Exercise 4 : Introduction to TensorFlow



In [ ]:

    
# Import libraries
import tensorflow as tf
import numpy as np
import time
import collections
import os



In [ ]:

    
# Import MNIST data with TensorFlow
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets(os.path.join('datasets', 'mnist'), one_hot=True) # load data in local folder

train_data = mnist.train.images.astype(np.float32)
train_labels = mnist.train.labels

test_data = mnist.test.images.astype(np.float32)
test_labels = mnist.test.labels

print(train_data.shape)
print(train_labels.shape)
print(test_data.shape)
print(test_labels.shape)

1st Step: Construct Computational Graph

Question 1: Prepare the input variables (x,y_label) of the computational graph

Hint: You may use the function tf.placeholder()



In [ ]:

    
# computational graph inputs
batch_size = 100
d = train_data.shape[1]
nc = 10
x = tf.placeholder(tf.float32,[batch_size,d]); print('x=',x,x.get_shape())
y_label = YOUR CODE HERE

Question 2: Prepare the variables (W,b) of the computational graph

Hint: You may use the function tf.Variable(), tf.truncated_normal()



In [ ]:

    
# computational graph variables
initial = tf.truncated_normal([d,nc], stddev=0.1); W = tf.Variable(initial); print('W=',W.get_shape())
b = YOUR CODE HERE

Question 3: Compute the classifier such that

$$ y=softmax(Wx +b) $$

Hint: You may use the function tf.matmul(), tf.nn.softmax()



In [ ]:

    
# Construct CG / output value
y = YOUR CODE HERE; print('y1=',y,y.get_shape())
y += b; print('y2=',y,y.get_shape())
y = YOUR CODE HERE; print('y3=',y,y.get_shape())

Question 4: Construct the loss of the computational graph such that

$$ loss = cross\ entropy(y_{label},y) = mean_{all\ data} \ \sum_{all\ classes} -\ y_{label}.\log(y) $$

Hint: You may use the function tf.Variable(), tf.truncated_normal()



In [ ]:

    
# Loss
cross_entropy = tf.reduce_mean(-tf.reduce_sum(YOUR CODE HERE * tf.log(YOUR CODE HERE), 1))

Question 5: Construct the L2 regularization of (W,b) to the computational graph such that

$$ R(W) = \|W\|_2^2\\ R(b) = \|b\|_2^2 $$

Hint: You may use the function tf.nn.l2_loss()



In [ ]:

    
reg_loss = YOUR CODE HERE

Question 6: Form the total loss

$$ total\ loss = cross\ entropy(y_{label},y) + reg\_par* (R(W) + R(b)) $$



In [ ]:

    
reg_par = 1e-3
total_loss = YOUR CODE HERE

Question 7: Perform optimization of the total loss for learning weight variables of the computational graph

Hint: You may use the function tf.train.GradientDescentOptimizer(learning_rate).minimize(total_loss)



In [ ]:

    
# Update CG variables / backward pass
train_step = YOUR CODE HERE

Question 8: Evaluate the accuracy

Hint: You may use the function tf.equal(tf.argmax(y,1), tf.argmax(y_label,1)) and tf.reduce_mean()



In [ ]:

    
# Accuracy
correct_prediction = YOUR CODE HERE
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

2nd Step: Run the Computational Graph with batches of training data

Check out the accuracy of test set



In [ ]:

    
# Create test set 
idx = np.random.permutation(test_data.shape[0]) # rand permutation
idx = idx[:batch_size]
test_x, test_y = test_data[idx,:], test_labels[idx]



In [ ]:

    
n = train_data.shape[0]
indices = collections.deque()

# Running Computational Graph
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(50):
    
    # Batch extraction
    if len(indices) < batch_size:
        indices.extend(np.random.permutation(n)) # rand permutation
    idx = [indices.popleft() for i in range(batch_size)] # extract n_batch data
    batch_x, batch_y = train_data[idx,:], train_labels[idx]
    
    # Run CG for variable training
    _,acc_train,total_loss_o = sess.run([train_step,accuracy,total_loss], feed_dict={x: batch_x, y_label: batch_y})
    print('\nIteration i=',i,', train accuracy=',acc_train,', loss=',total_loss_o)
    
    # Run CG for testset
    acc_test = sess.run(accuracy, feed_dict={x: test_x, y_label: test_y})
    print('test accuracy=',acc_test)