notebook.community

Edit and run

A Network Tour of Data Science

      Xavier Bresson, Winter 2016/17

Exercise 4 : Introduction to TensorFlow





In [1]:



    


# Import libraries
import tensorflow as tf
import numpy as np
import time
import collections
import os



    











In [2]:



    


# Import MNIST data with TensorFlow
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets(os.path.join('datasets', 'mnist'), one_hot=True) # load data in local folder

train_data = mnist.train.images.astype(np.float32)
train_labels = mnist.train.labels

test_data = mnist.test.images.astype(np.float32)
test_labels = mnist.test.labels

print(train_data.shape)
print(train_labels.shape)
print(test_data.shape)
print(test_labels.shape)



    


















    






Extracting datasets/mnist/train-images-idx3-ubyte.gz
Extracting datasets/mnist/train-labels-idx1-ubyte.gz
Extracting datasets/mnist/t10k-images-idx3-ubyte.gz
Extracting datasets/mnist/t10k-labels-idx1-ubyte.gz
(55000, 784)
(55000, 10)
(10000, 784)
(10000, 10)

1st Step: Construct Computational Graph

 Question 1: Prepare the input variables (x,y_label) of the computational graph

Hint: You may use the function tf.placeholder()





In [3]:



    


# computational graph inputs
batch_size = 100
d = train_data.shape[1]
lbls = train_labels.shape[1]
nc = 10
x = tf.placeholder(tf.float32,[batch_size,d]); print('x=',x,x.get_shape())
y_label = tf.placeholder(tf.float32,[batch_size,lbls]); print('y_label=',y_label,y_label.get_shape())



    


















    






x= Tensor("Placeholder:0", shape=(100, 784), dtype=float32) (100, 784)
y_label= Tensor("Placeholder_1:0", shape=(100, 10), dtype=float32) (100, 10)

 Question 2: Prepare the variables (W,b) of the computational graph

Hint: You may use the function tf.Variable(), tf.truncated_normal()





In [33]:



    


# computational graph variables
initial = tf.truncated_normal([d,nc], stddev=0.1); W = tf.Variable(initial); print('W=',W.get_shape())
b = tf.Variable(tf.zeros(10)); print('b=',b.get_shape())



    


















    






W= (784, 10)
b= (10,)

 Question 3: Compute the classifier such that

$$ y=softmax(Wx +b) $$

Hint: You may use the function tf.matmul(), tf.nn.softmax()





In [36]:



    


# Construct CG / output value
y =tf.matmul(x,W); print('y1=',y,y.get_shape())
y += b; print('y2=',y,y.get_shape())
y = tf.nn.softmax(y, name=None); print('y3=',y,y.get_shape())



    


















    






y1= Tensor("MatMul_6:0", shape=(100, 10), dtype=float32) (100, 10)
y2= Tensor("add_6:0", shape=(100, 10), dtype=float32) (100, 10)
y3= Tensor("Softmax:0", shape=(100, 10), dtype=float32) (100, 10)

 Question 4: Construct the loss of the computational graph such that

$$ loss = cross\ entropy(y_{label},y) = mean_{all\ data} \ \sum_{all\ classes} -\ y_{label}.\log(y) $$ 




In [38]:



    


# Loss
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_label * tf.log(y), 1))



    


















    
Out[38]:








<tf.Tensor 'Mean_1:0' shape=() dtype=float32>

 Question 5: Construct the L2 regularization of (W,b) to the computational graph such that

$$ R(W) = \|W\|_2^2\\ R(b) = \|b\|_2^2 $$

Hint: You may use the function tf.nn.l2_loss()





In [39]:



    


reg_loss = 2*(tf.nn.l2_loss(W)+tf.nn.l2_loss(b))

 Question 6: Form the total loss

$$ total\ loss = cross\ entropy(y_{label},y) + reg\_par* (R(W) + R(b)) $$ 




In [40]:



    


reg_par = 1e-3
total_loss = cross_entropy + reg_par*reg_loss

 Question 7: Perform optimization of the total loss for learning weight variables of the computational graph

Hint: You may use the function tf.train.GradientDescentOptimizer(learning_rate).minimize(total_loss)





In [45]:



    


# Update CG variables / backward pass
learning_rate = 0.1#tf.placeholder(tf.float32, shape=[])
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(total_loss)

 Question 8: Evaluate the accuracy

Hint: You may use the function tf.equal(tf.argmax(y,1), tf.argmax(y_label,1))





In [42]:



    


# Accuracy
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_label,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

 2nd Step: Run the Computational Graph with batches of training data

Check out the accuracy of test set





In [43]:



    


# Create test set 
idx = np.random.permutation(test_data.shape[0]) # rand permutation
idx = idx[:batch_size]
test_x, test_y = test_data[idx,:], test_labels[idx]



    











In [46]:



    


n = train_data.shape[0]
indices = collections.deque()

# Running Computational Graph
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(50):
    
    # Batch extraction
    if len(indices) < batch_size:
        indices.extend(np.random.permutation(n)) # rand permutation
    idx = [indices.popleft() for i in range(batch_size)] # extract n_batch data
    batch_x, batch_y = train_data[idx,:], train_labels[idx]
    
    # Run CG for variable training
    _,acc_train,total_loss_o = sess.run([train_step,accuracy,total_loss], feed_dict={x: batch_x, y_label: batch_y})
    print('\nIteration i=',i,', train accuracy=',acc_train,', loss=',total_loss_o)
    
    # Run CG for testset
    acc_test = sess.run(accuracy, feed_dict={x: test_x, y_label: test_y})
    print('test accuracy=',acc_test)



    


















    






Iteration i= 0 , train accuracy= 0.11 , loss= 2.62957
test accuracy= 0.16

Iteration i= 1 , train accuracy= 0.23 , loss= 2.3591
test accuracy= 0.23

Iteration i= 2 , train accuracy= 0.24 , loss= 2.28646
test accuracy= 0.25

Iteration i= 3 , train accuracy= 0.23 , loss= 2.20308
test accuracy= 0.28

Iteration i= 4 , train accuracy= 0.19 , loss= 2.15221
test accuracy= 0.32

Iteration i= 5 , train accuracy= 0.33 , loss= 2.10529
test accuracy= 0.33

Iteration i= 6 , train accuracy= 0.39 , loss= 2.00862
test accuracy= 0.38

Iteration i= 7 , train accuracy= 0.38 , loss= 1.93917
test accuracy= 0.39

Iteration i= 8 , train accuracy= 0.42 , loss= 1.80285
test accuracy= 0.44

Iteration i= 9 , train accuracy= 0.47 , loss= 1.74397
test accuracy= 0.52

Iteration i= 10 , train accuracy= 0.53 , loss= 1.68617
test accuracy= 0.52

Iteration i= 11 , train accuracy= 0.53 , loss= 1.63493
test accuracy= 0.54

Iteration i= 12 , train accuracy= 0.55 , loss= 1.59938
test accuracy= 0.57

Iteration i= 13 , train accuracy= 0.59 , loss= 1.554
test accuracy= 0.59

Iteration i= 14 , train accuracy= 0.6 , loss= 1.51773
test accuracy= 0.61

Iteration i= 15 , train accuracy= 0.63 , loss= 1.51477
test accuracy= 0.64

Iteration i= 16 , train accuracy= 0.61 , loss= 1.47688
test accuracy= 0.63

Iteration i= 17 , train accuracy= 0.68 , loss= 1.39394
test accuracy= 0.6

Iteration i= 18 , train accuracy= 0.65 , loss= 1.43529
test accuracy= 0.64

Iteration i= 19 , train accuracy= 0.58 , loss= 1.41979
test accuracy= 0.64

Iteration i= 20 , train accuracy= 0.59 , loss= 1.42947
test accuracy= 0.65

Iteration i= 21 , train accuracy= 0.61 , loss= 1.38538
test accuracy= 0.67

Iteration i= 22 , train accuracy= 0.6 , loss= 1.3769
test accuracy= 0.69

Iteration i= 23 , train accuracy= 0.63 , loss= 1.39994
test accuracy= 0.7

Iteration i= 24 , train accuracy= 0.68 , loss= 1.25396
test accuracy= 0.69

Iteration i= 25 , train accuracy= 0.65 , loss= 1.30319
test accuracy= 0.72

Iteration i= 26 , train accuracy= 0.69 , loss= 1.33592
test accuracy= 0.72

Iteration i= 27 , train accuracy= 0.68 , loss= 1.23335
test accuracy= 0.74

Iteration i= 28 , train accuracy= 0.77 , loss= 1.14386
test accuracy= 0.73

Iteration i= 29 , train accuracy= 0.72 , loss= 1.13069
test accuracy= 0.71

Iteration i= 30 , train accuracy= 0.78 , loss= 1.15764
test accuracy= 0.72

Iteration i= 31 , train accuracy= 0.73 , loss= 1.15789
test accuracy= 0.76

Iteration i= 32 , train accuracy= 0.7 , loss= 1.14777
test accuracy= 0.75

Iteration i= 33 , train accuracy= 0.74 , loss= 1.11072
test accuracy= 0.74

Iteration i= 34 , train accuracy= 0.79 , loss= 1.10595
test accuracy= 0.77

Iteration i= 35 , train accuracy= 0.72 , loss= 1.1696
test accuracy= 0.78

Iteration i= 36 , train accuracy= 0.75 , loss= 1.0789
test accuracy= 0.77

Iteration i= 37 , train accuracy= 0.79 , loss= 1.01109
test accuracy= 0.77

Iteration i= 38 , train accuracy= 0.89 , loss= 0.819416
test accuracy= 0.78

Iteration i= 39 , train accuracy= 0.7 , loss= 1.08737
test accuracy= 0.77

Iteration i= 40 , train accuracy= 0.83 , loss= 0.915718
test accuracy= 0.77

Iteration i= 41 , train accuracy= 0.83 , loss= 0.937208
test accuracy= 0.77

Iteration i= 42 , train accuracy= 0.8 , loss= 1.03092
test accuracy= 0.76

Iteration i= 43 , train accuracy= 0.83 , loss= 0.941506
test accuracy= 0.76

Iteration i= 44 , train accuracy= 0.74 , loss= 1.01248
test accuracy= 0.79

Iteration i= 45 , train accuracy= 0.85 , loss= 0.819709
test accuracy= 0.79

Iteration i= 46 , train accuracy= 0.82 , loss= 0.841836
test accuracy= 0.79

Iteration i= 47 , train accuracy= 0.76 , loss= 0.968287
test accuracy= 0.79

Iteration i= 48 , train accuracy= 0.84 , loss= 0.942129
test accuracy= 0.78

Iteration i= 49 , train accuracy= 0.82 , loss= 0.884058
test accuracy= 0.78

















In [ ]:

Content source: malogrisard/NTDScourse

Similar notebooks:

notebook.community | gallery | about