Google TensorFlow

Overview

TensorFlow is a programming system in which you represent computations as graphs. Nodes in the graph are called ops (short for operations). An op takes zero or more Tensors, performs some computation, and produces zero or more Tensors. A Tensor is a typed multi-dimensional array. For example, you can represent a mini-batch of images as a 4-D array of floating point numbers with dimensions [batch, height, width, channels].

A TensorFlow graph is a description of computations. To compute anything, a graph must be launched in a Session. A Session places the graph ops onto Devices, such as CPUs or GPUs, and provides methods to execute them. These methods return tensors produced by ops as numpy ndarray objects in Python, and as tensorflow::Tensor instances in C and C++.

Installation

  • Yet no support for GPU on OS X
  • Can be hardly installed on Windows

In [5]:
%%capture
%%bash
# Ubuntu/Linux 64-bit, CPU only:
# pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl
# Ubuntu/Linux 64-bit, GPU enabled:
# pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl
# Mac OS X, CPU only:
easy_install --upgrade six
pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.6.0-py2-none-any.whl

Basic usage

Launch the default graph

To run the matmul op we call the session 'run()' method, passing 'product' which represents the output of the matmul op. This indicates to the call that we want to get the output of the matmul op back.

All inputs needed by the op are run automatically by the session. They typically are run in parallel.

The call 'run(product)' thus causes the execution of threes ops in the graph: the two constants and matmul.

The output of the op is returned in 'result' as a numpy ndarray object.


In [67]:
import tensorflow as tf

with tf.Graph().as_default():
    # Create a Constant op that produces a 1x2 matrix.  The op is
    # added as a node to the default graph.
    #
    # The value returned by the constructor represents the output
    # of the Constant op.
    matrix1 = tf.constant([[3., 3.]])

    # Create another Constant that produces a 2x1 matrix.
    matrix2 = tf.constant([[2.],[2.]])

    # Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.
    # The returned value, 'product', represents the result of the matrix
    # multiplication.
    product = tf.matmul(matrix1, matrix2)
    with tf.Session() as sess:
        result = sess.run([product])
        print(result)


[array([[ 12.]], dtype=float32)]

There is an opportunity to explicitly select computational device for session:


In [68]:
with tf.Graph().as_default():
    with tf.Session() as sess:
        with tf.device("/cpu:0"):
            matrix1 = tf.constant([[3., 3.]])
            matrix2 = tf.constant([[2.],[2.]])
            product = tf.matmul(matrix1, matrix2)
            result = sess.run([product])
            print(result)


[array([[ 12.]], dtype=float32)]

And we can do it in a parallel way:


In [ ]:
with tf.Graph().as_default():
    c = []
    for d in ['/gpu:2', '/gpu:3']:
        with tf.device(d):
            a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
            b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
            c.append(tf.matmul(a, b))
    with tf.device('/cpu:0'):
        sum = tf.add_n(c)
    # Creates a session with log_device_placement set to True.
    sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
    # Runs the op.
    print(sess.run(sum))

Variables

Variables maintain state across executions of the graph. The following example shows a variable serving as a simple counter.


In [70]:
with tf.Graph().as_default():
    # Create a Variable, that will be initialized to the scalar value 0.
    state = tf.Variable(0, name="counter")

    # Create an Op to add one to `state`.

    one = tf.constant(1)
    new_value = tf.add(state, one)
    update = tf.assign(state, new_value)

    # Variables must be initialized by running an `init` Op after having
    # launched the graph.  We first have to add the `init` Op to the graph.
    init_op = tf.initialize_all_variables()

    # Launch the graph and run the ops.
    with tf.Session() as sess:
        sess.run(init_op)
        print(sess.run(state))
        for _ in range(3):
            sess.run(update)
            print(sess.run(state))


0
1
2
3

The assign() operation in this code is a part of the expression graph just like the add() operation, so it does not actually perform the assignment until run() executes the expression.

You typically represent the parameters of a statistical model as a set of Variables. For example, you would store the weights for a neural network as a tensor in a Variable. During training you update this tensor by running a training graph repeatedly.

Explore linear equation


In [31]:
import tensorflow as tf
import numpy as np

# Create 100 phony x, y data points in NumPy, y = x * 0.1 + 0.3
x_data = np.random.rand(10000).astype("float32")
y_data = x_data * 0.1 + 0.3

# Try to find values for W and b that compute y_data = W * x_data + b
# (We know that W should be 0.1 and b 0.3, but Tensorflow will
# figure that out for us.)
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))
b = tf.Variable(tf.zeros([1]))
y = W * x_data + b

# Minimize the mean squared errors.
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.6)
train = optimizer.minimize(loss)

# Before starting, initialize the variables.  We will 'run' this first.
init = tf.initialize_all_variables()

# Launch the graph.
sess = tf.Session()
sess.run(init)

# Fit the line.
for step in range(201):
    sess.run(train)
    if step % 20 == 0:
        print(step, sess.run(W), sess.run(b))


0 [-0.26455051] [ 0.90916252]
20 [-0.00258814] [ 0.35509714]
40 [ 0.08039634] [ 0.3105284]
60 [ 0.09625393] [ 0.30201188]
80 [ 0.09928416] [ 0.30038446]
100 [ 0.0998632] [ 0.30007347]
120 [ 0.09997385] [ 0.30001405]
140 [ 0.09999499] [ 0.30000269]
160 [ 0.09999905] [ 0.30000052]
180 [ 0.09999983] [ 0.3000001]
200 [ 0.0999999] [ 0.30000007]

Tensorboard

There is an opportunity to control and visualize your computational graph:


In [32]:
tf.train.SummaryWriter('./tfdata/', sess.graph_def);

In [33]:
%%bash
tensorboard --logdir ./tfdata/


Process is interrupted.

MNIST example


In [46]:
import math
import tensorflow.python.platform
import tensorflow as tf
NUM_CLASSES = 10
IMAGE_SIZE = 28
IMAGE_PIXELS = IMAGE_SIZE * IMAGE_SIZE

def inference(images, hidden1_units, hidden2_units):
  # Hidden 1
  with tf.name_scope('hidden1'):
    weights = tf.Variable(
        tf.truncated_normal([IMAGE_PIXELS, hidden1_units],
                            stddev=1.0 / math.sqrt(float(IMAGE_PIXELS))),
        name='weights')
    biases = tf.Variable(tf.zeros([hidden1_units]),
                         name='biases')
    hidden1 = tf.nn.relu(tf.matmul(images, weights) + biases)
  # Hidden 2
  with tf.name_scope('hidden2'):
    weights = tf.Variable(
        tf.truncated_normal([hidden1_units, hidden2_units],
                            stddev=1.0 / math.sqrt(float(hidden1_units))),
        name='weights')
    biases = tf.Variable(tf.zeros([hidden2_units]),
                         name='biases')
    hidden2 = tf.nn.relu(tf.matmul(hidden1, weights) + biases)
  # Linear
  with tf.name_scope('softmax_linear'):
    weights = tf.Variable(
        tf.truncated_normal([hidden2_units, NUM_CLASSES],
                            stddev=1.0 / math.sqrt(float(hidden2_units))),
        name='weights')
    biases = tf.Variable(tf.zeros([NUM_CLASSES]),
                         name='biases')
    logits = tf.matmul(hidden2, weights) + biases
  return logits

def loss_function(logits, labels):
  batch_size = tf.size(labels)
  labels = tf.expand_dims(labels, 1)
  indices = tf.expand_dims(tf.range(0, batch_size), 1)
  concated = tf.concat(1, [indices, labels])
  onehot_labels = tf.sparse_to_dense(
      concated, tf.pack([batch_size, NUM_CLASSES]), 1.0, 0.0)
  cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,
                                                          onehot_labels,
                                                          name='xentropy')
  loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
  return loss

def training(loss, learning_rate):
  tf.scalar_summary(loss.op.name, loss)
  optimizer = tf.train.GradientDescentOptimizer(learning_rate)
  global_step = tf.Variable(0, name='global_step', trainable=False)
  train_op = optimizer.minimize(loss, global_step=global_step)
  return train_op

def evaluation(logits, labels):
  correct = tf.nn.in_top_k(logits, labels, 1)
  return tf.reduce_sum(tf.cast(correct, tf.int32))

In [71]:
import os.path
import time
import tensorflow.python.platform
import numpy
from six.moves import xrange  # pylint: disable=redefined-builtin
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
learning_rate = 0.01
max_steps = 10000
hidden1 = 1024
hidden2 = 512
batch_size = 200
train_dir = 'tfdata/mnist'
fake_data = False

def placeholder_inputs(batch_size):
  images_placeholder = tf.placeholder(tf.float32, shape=(batch_size,
                                                         IMAGE_PIXELS))
  labels_placeholder = tf.placeholder(tf.int32, shape=(batch_size))
  return images_placeholder, labels_placeholder

def fill_feed_dict(data_set, images_pl, labels_pl):
  images_feed, labels_feed = data_set.next_batch(batch_size,
                                                 fake_data)
  feed_dict = {
      images_pl: images_feed,
      labels_pl: labels_feed,
  }
  return feed_dict

def do_eval(sess,
            eval_correct,
            images_placeholder,
            labels_placeholder,
            data_set):
  true_count = 0
  steps_per_epoch = data_set.num_examples // batch_size
  num_examples = steps_per_epoch * batch_size
  for step in xrange(steps_per_epoch):
    feed_dict = fill_feed_dict(data_set,
                               images_placeholder,
                               labels_placeholder)
    true_count += sess.run(eval_correct, feed_dict=feed_dict)
  precision = true_count / num_examples
  print('  Num examples: %d  Num correct: %d  Precision @ 1: %0.04f' %
        (num_examples, true_count, precision))
    
def run_training():
  data_sets = input_data.read_data_sets(train_dir, fake_data)
  with tf.Graph().as_default():
    # Generate placeholders for the images and labels.
    images_placeholder, labels_placeholder = placeholder_inputs(batch_size)
    logits = inference(images_placeholder,
                             hidden1,
                             hidden2)
    loss = loss_function(logits, labels_placeholder)
    train_op = training(loss, learning_rate)
    eval_correct = evaluation(logits, labels_placeholder)
    summary_op = tf.merge_all_summaries()
    saver = tf.train.Saver()
    sess = tf.Session()
    init = tf.initialize_all_variables()
    sess.run(init)
    summary_writer = tf.train.SummaryWriter(train_dir,
                                            graph_def=sess.graph_def)
    # And then after everything is built, start the training loop.
    for step in xrange(max_steps):
      start_time = time.time()
      feed_dict = fill_feed_dict(data_sets.train,
                                 images_placeholder,
                                 labels_placeholder)
      _, loss_value = sess.run([train_op, loss],
                               feed_dict=feed_dict)
      duration = time.time() - start_time
      if step % 100 == 0:
        print('Step %d: loss = %.2f (%.3f sec)' % (step, loss_value, duration))
        summary_str = sess.run(summary_op, feed_dict=feed_dict)
        summary_writer.add_summary(summary_str, step)
      if (step + 1) % 1000 == 0 or (step + 1) == max_steps:
        saver.save(sess, train_dir, global_step=step)
        print('Training Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.train)
        print('Validation Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.validation)
        print('Test Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.test)
run_training()


Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting tfdata/mnist/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting tfdata/mnist/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting tfdata/mnist/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting tfdata/mnist/t10k-labels-idx1-ubyte.gz
Step 0: loss = 2.31 (0.054 sec)
Step 100: loss = 1.96 (0.023 sec)
Step 200: loss = 1.56 (0.022 sec)
Step 300: loss = 1.10 (0.023 sec)
Step 400: loss = 0.81 (0.023 sec)
Step 500: loss = 0.74 (0.023 sec)
Step 600: loss = 0.66 (0.022 sec)
Step 700: loss = 0.57 (0.028 sec)
Step 800: loss = 0.53 (0.024 sec)
Step 900: loss = 0.43 (0.030 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 48744  Precision @ 1: 0.8863
Validation Data Eval:
  Num examples: 5000  Num correct: 4465  Precision @ 1: 0.8930
Test Data Eval:
  Num examples: 10000  Num correct: 8931  Precision @ 1: 0.8931
Step 1000: loss = 0.37 (0.039 sec)
Step 1100: loss = 0.33 (0.133 sec)
Step 1200: loss = 0.34 (0.038 sec)
Step 1300: loss = 0.35 (0.027 sec)
Step 1400: loss = 0.38 (0.026 sec)
Step 1500: loss = 0.38 (0.026 sec)
Step 1600: loss = 0.32 (0.029 sec)
Step 1700: loss = 0.25 (0.025 sec)
Step 1800: loss = 0.39 (0.026 sec)
Step 1900: loss = 0.36 (0.043 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 49940  Precision @ 1: 0.9080
Validation Data Eval:
  Num examples: 5000  Num correct: 4577  Precision @ 1: 0.9154
Test Data Eval:
  Num examples: 10000  Num correct: 9145  Precision @ 1: 0.9145
Step 2000: loss = 0.38 (0.045 sec)
Step 2100: loss = 0.29 (0.026 sec)
Step 2200: loss = 0.38 (0.132 sec)
Step 2300: loss = 0.41 (0.028 sec)
Step 2400: loss = 0.32 (0.026 sec)
Step 2500: loss = 0.36 (0.026 sec)
Step 2600: loss = 0.27 (0.026 sec)
Step 2700: loss = 0.27 (0.026 sec)
Step 2800: loss = 0.33 (0.026 sec)
Step 2900: loss = 0.25 (0.026 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 50588  Precision @ 1: 0.9198
Validation Data Eval:
  Num examples: 5000  Num correct: 4625  Precision @ 1: 0.9250
Test Data Eval:
  Num examples: 10000  Num correct: 9233  Precision @ 1: 0.9233
Step 3000: loss = 0.32 (0.042 sec)
Step 3100: loss = 0.35 (0.027 sec)
Step 3200: loss = 0.27 (0.027 sec)
Step 3300: loss = 0.31 (0.142 sec)
Step 3400: loss = 0.25 (0.030 sec)
Step 3500: loss = 0.29 (0.027 sec)
Step 3600: loss = 0.30 (0.024 sec)
Step 3700: loss = 0.24 (0.026 sec)
Step 3800: loss = 0.22 (0.029 sec)
Step 3900: loss = 0.20 (0.027 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 50990  Precision @ 1: 0.9271
Validation Data Eval:
  Num examples: 5000  Num correct: 4660  Precision @ 1: 0.9320
Test Data Eval:
  Num examples: 10000  Num correct: 9310  Precision @ 1: 0.9310
Step 4000: loss = 0.28 (0.041 sec)
Step 4100: loss = 0.29 (0.026 sec)
Step 4200: loss = 0.24 (0.032 sec)
Step 4300: loss = 0.37 (0.026 sec)
Step 4400: loss = 0.25 (0.135 sec)
Step 4500: loss = 0.31 (0.026 sec)
Step 4600: loss = 0.25 (0.025 sec)
Step 4700: loss = 0.28 (0.025 sec)
Step 4800: loss = 0.24 (0.025 sec)
Step 4900: loss = 0.21 (0.026 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 51372  Precision @ 1: 0.9340
Validation Data Eval:
  Num examples: 5000  Num correct: 4695  Precision @ 1: 0.9390
Test Data Eval:
  Num examples: 10000  Num correct: 9376  Precision @ 1: 0.9376
Step 5000: loss = 0.26 (0.043 sec)
Step 5100: loss = 0.19 (0.026 sec)
Step 5200: loss = 0.21 (0.025 sec)
Step 5300: loss = 0.24 (0.028 sec)
Step 5400: loss = 0.34 (0.028 sec)
Step 5500: loss = 0.22 (0.136 sec)
Step 5600: loss = 0.22 (0.027 sec)
Step 5700: loss = 0.20 (0.025 sec)
Step 5800: loss = 0.31 (0.025 sec)
Step 5900: loss = 0.29 (0.025 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 51730  Precision @ 1: 0.9405
Validation Data Eval:
  Num examples: 5000  Num correct: 4724  Precision @ 1: 0.9448
Test Data Eval:
  Num examples: 10000  Num correct: 9417  Precision @ 1: 0.9417
Step 6000: loss = 0.22 (0.043 sec)
Step 6100: loss = 0.29 (0.025 sec)
Step 6200: loss = 0.21 (0.026 sec)
Step 6300: loss = 0.17 (0.027 sec)
Step 6400: loss = 0.18 (0.027 sec)
Step 6500: loss = 0.16 (0.028 sec)
Step 6600: loss = 0.19 (0.142 sec)
Step 6700: loss = 0.24 (0.031 sec)
Step 6800: loss = 0.18 (0.025 sec)
Step 6900: loss = 0.18 (0.025 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 51962  Precision @ 1: 0.9448
Validation Data Eval:
  Num examples: 5000  Num correct: 4747  Precision @ 1: 0.9494
Test Data Eval:
  Num examples: 10000  Num correct: 9454  Precision @ 1: 0.9454
Step 7000: loss = 0.20 (0.040 sec)
Step 7100: loss = 0.23 (0.026 sec)
Step 7200: loss = 0.30 (0.026 sec)
Step 7300: loss = 0.21 (0.026 sec)
Step 7400: loss = 0.21 (0.027 sec)
Step 7500: loss = 0.15 (0.027 sec)
Step 7600: loss = 0.11 (0.027 sec)
Step 7700: loss = 0.21 (0.137 sec)
Step 7800: loss = 0.14 (0.025 sec)
Step 7900: loss = 0.14 (0.026 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 52170  Precision @ 1: 0.9485
Validation Data Eval:
  Num examples: 5000  Num correct: 4760  Precision @ 1: 0.9520
Test Data Eval:
  Num examples: 10000  Num correct: 9483  Precision @ 1: 0.9483
Step 8000: loss = 0.20 (0.041 sec)
Step 8100: loss = 0.15 (0.026 sec)
Step 8200: loss = 0.23 (0.026 sec)
Step 8300: loss = 0.16 (0.025 sec)
Step 8400: loss = 0.14 (0.026 sec)
Step 8500: loss = 0.14 (0.026 sec)
Step 8600: loss = 0.17 (0.026 sec)
Step 8700: loss = 0.25 (0.026 sec)
Step 8800: loss = 0.16 (0.135 sec)
Step 8900: loss = 0.24 (0.030 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 52480  Precision @ 1: 0.9542
Validation Data Eval:
  Num examples: 5000  Num correct: 4774  Precision @ 1: 0.9548
Test Data Eval:
  Num examples: 10000  Num correct: 9507  Precision @ 1: 0.9507
Step 9000: loss = 0.18 (0.042 sec)
Step 9100: loss = 0.21 (0.027 sec)
Step 9200: loss = 0.17 (0.027 sec)
Step 9300: loss = 0.20 (0.027 sec)
Step 9400: loss = 0.19 (0.026 sec)
Step 9500: loss = 0.13 (0.026 sec)
Step 9600: loss = 0.14 (0.026 sec)
Step 9700: loss = 0.17 (0.028 sec)
Step 9800: loss = 0.18 (0.028 sec)
Step 9900: loss = 0.11 (0.137 sec)
Training Data Eval:
  Num examples: 55000  Num correct: 52590  Precision @ 1: 0.9562
Validation Data Eval:
  Num examples: 5000  Num correct: 4789  Precision @ 1: 0.9578
Test Data Eval:
  Num examples: 10000  Num correct: 9532  Precision @ 1: 0.9532
Exception ignored in: <generator object get_controller at 0x1168186d0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2791, in get_controller
    assert self.stack[-1] is default
AssertionError: 
Exception ignored in: <generator object get_controller at 0x108cf1e60>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2791, in get_controller
    assert self.stack[-1] is default
AssertionError: 

In [ ]: