Try not to peek at the solutions when you go through the exercises. ;-)
First let's make sure this notebook works well in both Python 2 and Python 3:
In [ ]:
from __future__ import absolute_import, division, print_function, unicode_literals
In [ ]:
import tensorflow as tf
tf.__version__
From previous notebooks
In [ ]:
learning_rate = 0.01
momentum = 0.8
In this exercise, you will learn to use TensorBoard. It is a great visualization tool that comes with TensorFlow. It works by parsing special TensorFlow logs, called summaries, and displaying them nicely.
6.1) Starting the TensorBoard server. Open a Terminal and type the following commands.
Move to the tensorflow-safari-course directory:
~$ cd tensorflow-safari-course
Create the tf_logs directory that will hold the TensorFlow data that we will want TensorBoard to display:
~/tensorflow-safari-course$ mkdir tf_logs
Activate the virtual environment:
~/tensorflow-safari-course$ source env/bin/activate
Start the TensorBoard server:
(env) ~/tensorflow-safari-course$ tensorboard --logdir=tf_logs
Starting TensorBoard b'41' on port 6006
(You can navigate to http://127.0.1.1:6006 )
Now visit the URL given by TensorBoard. You should see the TensorBoard interface.
In [ ]:
In [ ]:
In [ ]:
6.2) Now create a tf.summary.FileWriter, with the parameters: logdir="tf_logs/run_number_1/" and graph=graph where graph is the one we built just before this exercise. This will automatically:
run_number_1 directory inside the tf_logs directory,events.out.tfevents.* file in that subdirectory that will contain the data that TensorBoard will display,Next, try refreshing the TensorBoard page in your browser (you may need to wait a couple minutes for it to detect the change, or else you can just restart the TensorBoard server). Visit the Graph tab: you should be able to visualize the graph.
In [ ]:
In [ ]:
In [ ]:
6.3) As you can see, the graph looks really messy in TensorBoard. We need to organize it a bit. For this, name scopes come in handy. An operation can be placed inside a name scope in one of two ways:
a = tf.constant(0.0, name="my_name_scope/a")
tf.name_scope() block, for example:with tf.name_scope("my_name_scope"):
a = tf.constant(0.0, name="a")
Add name scopes to the following graph, then write it to TensorBoard (using a different run number for the log directory name) and see how much better it looks, and how much easier it is to explore.
In [ ]:
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
reader = tf.TextLineReader(skip_header_lines=1)
filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
record_id, record = reader.read(filename_queue)
record_defaults = [[''], [0.0], [0.0]]
country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
b = tf.Variable(0.0, name="b")
w = tf.Variable(tf.zeros([1, 1]), name="w")
y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred") # X @ w + b
mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
global_step = tf.Variable(0, trainable=False, name='global_step')
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
training_op = optimizer.minimize(mse, global_step=global_step)
init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
saver = tf.train.Saver()
In [ ]:
In [ ]:
6.4) Print out the name of a few operations. Notice how the names now have the scope as a prefix.
In [ ]:
In [ ]:
In [ ]:
6.5) TensorBoard is capable of displaying data from multiple TensorFlow runs (for example multiple training sessions). For this, we need to place the data from each run in a different subdirectory of the tf_logs directory. We can name these subdirectories however we want, but a simple option is to name them using a timestamp. The following logdir() function returns the path of a subdirectory whose name is based on the current date and time:
In [ ]:
from datetime import datetime
def logdir():
root_logdir = "tf_logs"
now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
return "{}/run_{}/".format(root_logdir, now)
In [ ]:
logdir()
Create a few different graphs and instantiate a different FileWriter for each one, using a different log directory every time (with the help of the logdir() function). Refresh TensorBoard and notice that you can browse any graph you want by selecting the appropriate run.
In [ ]:
In [ ]:
In [ ]:
6.6) Now we will use TensorBoard to visualize the learning curve, that is the evolution of the cost function during training.
tf.summary.scalar("MSE", mse).FileWriter's add_summary() method (also specifying the training step). For performance reasons, you probably want to do this only every 10 training iterations or so.
In [ ]:
In [ ]:
In [ ]:
Try not to peek at the solution below before you have done the exercise! :)
6.2)
In [ ]:
summary_writer = tf.summary.FileWriter("tf_logs/run_number_1_solution/", graph=graph)
6.3)
In [ ]:
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
with tf.name_scope("reader"):
reader = tf.TextLineReader(skip_header_lines=1)
filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
record_id, record = reader.read(filename_queue)
record_defaults = [[''], [0.0], [0.0]]
country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
with tf.name_scope("linear_model"):
X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
b = tf.Variable(0.0, name="b")
w = tf.Variable(tf.zeros([1, 1]), name="w")
y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred") # X @ w + b
with tf.name_scope("train"):
mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
global_step = tf.Variable(0, trainable=False, name='global_step')
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
training_op = optimizer.minimize(mse, global_step=global_step)
init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
saver = tf.train.Saver()
In [ ]:
summary_writer = tf.summary.FileWriter("tf_logs/run_number_2_solution/", graph=graph)
6.4)
In [ ]:
country.name, gdp_per_capita.name, X_batch.name, y_batch.name
In [ ]:
X.name, y.name, b.name, w.name, y_pred.name
In [ ]:
mse.name, global_step.name, training_op.name
6.5)
In [ ]:
graph1 = tf.Graph()
with graph1.as_default():
a = tf.constant(1.0)
In [ ]:
summary_writer = tf.summary.FileWriter(logdir(), graph=graph)
In [ ]:
graph2 = tf.Graph()
with graph2.as_default():
a = tf.constant(1.0, name="a")
b = tf.Variable(2.0, name="b")
c = a * b
If we run logdir() twice within the same second, we will get the same directory name twice. To avoid this, let's wait a bit over 1 second here. In real life, this is quite unlikely to happen since training a model typically takes much longer than 1 second.
In [ ]:
import time
time.sleep(1.1)
In [ ]:
summary_writer = tf.summary.FileWriter(logdir(), graph=graph)
In [ ]:
time.sleep(1.1)
6.6)
In [ ]:
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
with tf.name_scope("reader"):
reader = tf.TextLineReader(skip_header_lines=1)
filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
record_id, record = reader.read(filename_queue)
record_defaults = [[''], [0.0], [0.0]]
country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
with tf.name_scope("linear_model"):
X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
b = tf.Variable(0.0, name="b")
w = tf.Variable(tf.zeros([1, 1]), name="w")
y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")
with tf.name_scope("train"):
mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
mse_summary = tf.summary.scalar('MSE', mse) # <= ADDED
global_step = tf.Variable(0, trainable=False, name='global_step')
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
training_op = optimizer.minimize(mse, global_step=global_step)
init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
saver = tf.train.Saver()
In [ ]:
summary_writer = tf.summary.FileWriter(logdir(), graph)
In [ ]:
with tf.Session(graph=graph) as sess:
init.run()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
try:
while not coord.should_stop():
_, mse_summary_val, global_step_val = sess.run([training_op, mse_summary, global_step])
if global_step_val % 10 == 0:
summary_writer.add_summary(mse_summary_val, global_step_val)
except tf.errors.OutOfRangeError:
print("End of training")
coord.request_stop()
coord.join(threads)
saver.save(sess, "./my_life_satisfaction_model")