Thinking Axes for ML-Packages

Model Specification or
- Writing a configuration file
- Choice of Grammer
  - JSON
    - Caffe
    - Google DistBelief
    - CNTK
Programmatic generation
- Writing Code
- Choice of a High Level Language
  - Lua
    - Torch
  - Python
    - Theano
    - TensorFlow
    - Rich Community
    - Library Infrastructure

TensorFlow vs Theano

Theano
- Deep learning librray
- Has python wrapper
- Inspiration for TensorFlow
- Academic Project
- Been longer
- More stable ?
TensorFlow
- Very similar system
- Better support for distributed systems
- Google's Project
- Started slow compared to Theano
- Pickedup heat after open-sourcing

What is TensorFlow ?

Its a deep learning library rite ?
- Yes, Its a open-source deep learning library
- Why was it not called DeepFlow ?
  - Its more general
  - Provides primitives for defining functions on Tensors
  - Automatically computing derivatives of arbitrary functions on Tensors
  - One can use TensorFlow to solve PDEs
    - But whats with the Flow ?
    - TensorFlow programs are structured into 2 phases
      1. A Graph Construction Phase
      2. A Graph Execution Phase
- Graph Construction Phase
  - Assembles a graph, a computation graph
  - Think of Tensors Flowing throught this graph, undergoing transformation at each node
- Graph Execution Phase
  - Uses a session to execute operations that are specified in this graph.

Wait.. Whats a Tensor ?

Formally,
- Tensors are multilinear maps from vector spaces to the real numbers
- Add more formalism.
Ideas
- A scalar is a tensor
- A vector is a tensor
- A matrix is a tensor
A Tensor can be represented as
- A multidimensional array of number
- So need a N-d array library
- Why not just use Numpy ?
  - Yes, It has Ndarray Support
  - But cannot create tensor Functions
  - Cannot automatically compute derivatives
  - No GPU support
  - So TensorFlow is a Feature rich N-d Array Library, nothing more

Thinking in Numpy-Land



In [1]:

    
import numpy as np



In [2]:

    
a = np.zeros((2,2))



In [3]:

    
a









    Out[3]:





array([[ 0.,  0.],
       [ 0.,  0.]])



In [4]:

    
a.shape









    Out[4]:





(2, 2)



In [5]:

    
np.reshape(a, (1,4))









    Out[5]:





array([[ 0.,  0.,  0.,  0.]])



In [6]:

    
b = np.ones((2,2))



In [7]:

    
b









    Out[7]:





array([[ 1.,  1.],
       [ 1.,  1.]])



In [8]:

    
np.sum(b, axis=1)









    Out[8]:





array([ 2.,  2.])

Thinking in TensorFlow-Land



In [9]:

    
import tensorflow as tf



In [10]:

    
tf.InteractiveSession()









    Out[10]:





<tensorflow.python.client.session.InteractiveSession at 0x7f0e18e4f9d0>



In [11]:

    
a = tf.zeros((2,2))



In [12]:

    
a









    Out[12]:





<tf.Tensor 'zeros:0' shape=(2, 2) dtype=float32>



In [13]:

    
b = tf.ones((2,2))



In [14]:

    
b









    Out[14]:





<tf.Tensor 'ones:0' shape=(2, 2) dtype=float32>



In [15]:

    
tf.reduce_sum(b, reduction_indices=1).eval()









    Out[15]:





array([ 2.,  2.], dtype=float32)



In [16]:

    
a.get_shape()









    Out[16]:





TensorShape([Dimension(2), Dimension(2)])



In [17]:

    
tf.reshape(a, (1,4)).eval()









    Out[17]:





array([[ 0.,  0.,  0.,  0.]], dtype=float32)

Numpy to TensorFlow Dictionary

Explicit Evaluation



In [18]:

    
# TensorFlow computations define a computation graph
# This means no numerical value until evaluated explicitly



In [19]:

    
a = np.zeros((2,2))



In [20]:

    
a









    Out[20]:





array([[ 0.,  0.],
       [ 0.,  0.]])



In [21]:

    
ta = tf.zeros((2,2))



In [22]:

    
print(a)









    



[[ 0.  0.]
 [ 0.  0.]]



In [23]:

    
print(ta)









    



Tensor("zeros_1:0", shape=(2, 2), dtype=float32)



In [24]:

    
print(ta.eval())









    



[[ 0.  0.]
 [ 0.  0.]]

Session Object



In [25]:

    
# A session object encapsulates the environment in which 
# Tensor objects are evaluated



In [26]:

    
a = tf.constant(5.0)



In [27]:

    
a









    Out[27]:





<tf.Tensor 'Const:0' shape=() dtype=float32>



In [28]:

    
b = tf.constant(6.0)



In [29]:

    
b









    Out[29]:





<tf.Tensor 'Const_1:0' shape=() dtype=float32>



In [30]:

    
c = a * b



In [31]:

    
with tf.Session() as session:
    print(session.run(c))
    print(c.eval)









    



30.0
<bound method Tensor.eval of <tf.Tensor 'mul:0' shape=() dtype=float32>>

NOTE-1
- tf.InteractiveSession()
- syntactic sugar for keeping a default session
- open in jupyter or ipython
NOTE-2
- session.run(c)
- is an example of a tensorFlow Fetch
- coming up soon

Computation Graph

Idea Repeated to stress..
- Tensorflow program structured as
  - A Graph construction phase
  - A Graph execution phase
    - This uses a session to execute operations in the graph
- All computations add nodes to global default graph

Variables

All tensors we have used previously have been constant tensors
None of the tensors we have used untill now were variables
Lets define our first ever tensor variable



In [32]:

    
w1 = tf.ones((2,2))
w1









    Out[32]:





<tf.Tensor 'ones_1:0' shape=(2, 2) dtype=float32>



In [33]:

    
w2 = tf.Variable(tf.zeros((2,2)), name='weights')
w2









    Out[33]:





<tf.Variable 'weights:0' shape=(2, 2) dtype=float32_ref>



In [34]:

    
with tf.Session() as sess:
    # this is a tensor flow constant
    print(sess.run(w1))
    # this is a tensor flow variable
    # please note the init call
    sess.run(tf.global_variables_initializer())
    print(sess.run(w2))









    



[[ 1.  1.]
 [ 1.  1.]]
[[ 0.  0.]
 [ 0.  0.]]

TensorFlow variables must be initialized before they have values!
Please contrast with constant tensors.



In [35]:

    
W = tf.Variable(tf.zeros((2,2)), name="weights")
# variable objects can be initialized from constants



In [36]:

    
R = tf.Variable(tf.random_normal((2,2)), name="random_weights")
# variable objects can be initialized from random values



In [37]:

    
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(W))
    print(sess.run(R))









    



[[ 0.  0.]
 [ 0.  0.]]
[[-0.82578623  0.22288336]
 [ 1.10728753 -0.52197558]]

Updating Variable State



In [38]:

    
state = tf.Variable(0, name="counter")



In [39]:

    
new_value = tf.add(state, tf.constant(1))
# think of this as doing
# new_value = state + 1



In [40]:

    
update = tf.assign(state, new_value)
# think of this as state = new_value



In [41]:

    
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(state))
    for _ in range(3):
        sess.run(update)
        print(sess.run(state))
        
#        
# state = 0
#
# print(state)
# for _ in range(3)
#     state = state + 1
#     print(state)
#

Fetching Variable State



In [42]:

    
input1 = tf.constant(3.0)



In [43]:

    
input2 = tf.constant(2.0)



In [44]:

    
input3 = tf.constant(5.0)



In [45]:

    
intermed = tf.add(input2, input3)



In [46]:

    
mul = tf.multiply(input1, intermed)



In [47]:

    
with tf.Session() as sess:
    result = sess.run([mul, intermed])
    print(result)
    

# calling sess.run(var) on a tf.Session() object
# retrives its value.

# if you want to retrieve multiple variables
# simultaneously can do
# like sess.run([var1, var2])









    



[21.0, 7.0]

Input External Data into TensorFlow

All Previous examples have manually defined tensors
How to get external data sets into TensorFlow land ?
Import from numpy works ...



In [48]:

    
a = np.zeros((3,3))



In [49]:

    
ta = tf.convert_to_tensor(a)



In [50]:

    
with tf.Session() as sess:
    print(sess.run(ta))









    



[[ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]]

Placeholders

Getting the data with tf.convert_to_tensor() is cool, but as you see it does not scale.
Use Dummy nodes that provide entry points for data to the computational graph
- This takes us to tf.placeholder variables
Now we need a mapping from tf.placeholder variables or their names to data like numpy arrays, lists, etc..
- This takes us to feed_dict
- This is a python dictionary



In [51]:

    
# define placeholder objects for data entry
input1 = tf.placeholder(tf.float32)
input2 = tf.placeholder(tf.float32)
output = tf.multiply(input1, input2)



In [52]:

    
with tf.Session() as sess:
    print(sess.run([output], feed_dict={input1:[7.], input2:[2.] }))
    
# fetch value of input from computation graph
# feed data into the compuation graph









    



[array([ 14.], dtype=float32)]

Feed Dictionaries

Variable Scope

Complicated tensorFlow models can have 100's of variables
- tf.variable_scope() provides simple name-spacing to avoid clashes
- tf.get_variable() creates/accesses variables from withing a variable scope

Variable scope is a simple type of namespacing that adds prefixes to variable names within scope



In [53]:

    
# with tf.variable_scope("foo"):
#    with tf.variable_scope("bar"):
#        v = tf.get_variable("v", [1])
# assert v.name == "foo/bar/v:0"

Variable scope control variable reuse



In [54]:

    
# with tf.variable_scope("foo"):
#    v = tf.get_variable("v", [1])
#    tf.get_variable_scope().reuse_variables()
#    v1 = tf.get_variable("v", [1])
# assert v1 == v

Get Variable

Behaviour of get_variable() depends on
- reuse is set to false
  - create and return new variable
- reuse is set to true
  - search for existing variable with given name
  - raise ValueError if none found



In [55]:

    
#
# with tf.variable_scope("foo"):
#     v = tf.get_variable("v", [1])
# assert v.name == "foo/v:0"
#

#
# with tf.variable_scope("foo"):
#     v = tf.get_variable("v", [1])
# with tf.variable_scope("foo", reuse=True):
#    v1 = tf.get_variable("v", [1])
# assert v1 = v
#

Simple TensorFlow Scripts

TensorFlow Constants



In [56]:

    
# this is our first tensorflow line of code
# 1. its a tensorflow constant !
# welcome to tensorflow land
x = tf.constant(35, name='x')

Build the computation Graph



In [57]:

    
y = x + 5



In [58]:

    
print(y)









    



Tensor("add:0", shape=(), dtype=int32)

Run the Computation Graph



In [59]:

    
with tf.Session() as session:
    print(session.run(y))



In [60]:

    
%matplotlib inline



In [61]:

    
import matplotlib.image as mpimg



In [62]:

    
import matplotlib.pyplot as plt



In [63]:

    
# which image
filename = "ganesha.jpg"



In [64]:

    
# load image
raw_image_data = mpimg.imread(filename)



In [65]:

    
# Lord Ganesha was a scribe for Mr. Veda Vyasa
# who was narrating the Mahabharata.
#
# Later today we want to see if GAN's can learn
# the joint distribution over b/w images of Ganesha's
#
# For now, here is how, our god looks like ...
#
# Notice that there are 13 discrete features.

plt.imshow(raw_image_data)









    Out[65]:





<matplotlib.image.AxesImage at 0x7f0dcd61fcd0>

TensorFlow Variables

Stepping into TensorFlow-land from Numpy-land



In [66]:

    
# create a
# 1, tenforflow constant (last time)
# 2. tensorflow variable (now)
x = tf.Variable(raw_image_data, name='x')



In [67]:

    
# tf.initialize_all_variables() was deprecated recently
model = tf.global_variables_initializer()



In [68]:

    
with tf.Session() as session:
    # perform a basic operation
    transpose_op = tf.transpose(x, perm=[1, 0, 2])
    session.run(model)
    result = session.run(transpose_op)



In [69]:

    
# he may not like it, but here is the transpose
plt.imshow(result)









    Out[69]:





<matplotlib.image.AxesImage at 0x7f0dcd548510>

TensorFlow Placeholders



In [70]:

    
x = tf.placeholder("float", 3)
# size is optional, but helps



In [71]:

    
y = x * 2



In [72]:

    
with tf.Session() as session:
    result = session.run(y, feed_dict={x: [1, 2, 3]})
    print(result)









    



[ 2.  4.  6.]

2D Placeholder



In [73]:

    
x = tf.placeholder("float", [None, 3])
# size can be multidimensional
# None means , you dont know the size now
# Like Data sets used in ML
# You dont want to hardcode the number of samples



In [74]:

    
y = x * 2



In [75]:

    
x_data = [[1, 2, 3],
          [4, 5, 6],]
#
# this is 2 by 3
# can be  3 by 3
# can be  4 by 3 ...



In [76]:

    
with tf.Session() as session:
    result = session.run(y, feed_dict={x: x_data})
    print(result)









    



[[  2.   4.   6.]
 [  8.  10.  12.]]

3D Placeholder



In [77]:

    
image = tf.placeholder("uint8", [None, None, 3])



In [78]:

    
reverse = tf.reverse(image, [True, False]) # [True, False, False]



In [79]:

    
with tf.Session() as session:
    result = session.run(reverse, feed_dict={image: raw_image_data})
    print(result.shape)









    



(753, 1024, 3)



In [80]:

    
plt.imshow(result)
plt.show()

Maximum Entropy Classifier (logistic regression)

Minimizing cross-entrpy loss (logistic loss)



In [ ]:



In [ ]:



In [ ]:

Custom Functions (todo)



In [ ]:



In [ ]:



In [ ]:

TensorFlow Learn



In [ ]:



In [ ]:



In [ ]:

Keras



In [ ]:



In [ ]:



In [ ]: