Tensor flow can be visualised as computations in a graph with nodes that has definite operations. In simple terms, tensor flow carries out computations in the form of graphs. Nodes of the graph are Ops (Operations): Handles zero or multiple tensors to output zero or multiple tensors. Tensors are multidimensional array / list. Data is represented using tensor data structure which includes: Ranks, Shapes, Types. It is the tensors that are passed between the nodes in the graph. Tensor flow computationsa are mainly carried out using tensors such as variable tensors constant tensors and placeholder tensors.
Every Op or operation object execution and tensor evaluations, happens within a session. A session owns all the results generated and its related resources used within that session. Launch a session using the following command:
If the above commands are used then it is important that these resources are released at the end of a session using close() command. But for simplicity, it is adviced to use context manager session; so that resource release happens automatically. For the execution of an Op (node) in tensor flow to happen, execute using session.run() command as:
Example 1: Lets consider a simple tensor computation:
In [18]:
#```python
# Objective of the program is to do a simple multiplication on an input tensor of
# constant values
# To use tensor flow; import it
import tensorflow as tf
# tf.constant creates constant values. The below command creates a tensor;shape (2,2)
# with constant values. Be sure that each element are separated by comma delimiter and also
# that the two operands with which operation is performed are of the same data type!
val = tf.constant([[2,4],[7,7]],name = "constantValue")
# tf.constant creates a constant value with no shape
singleValue = tf.constant(7,name = "SingleValueConstant")
# tf.Variable creates a storage variable, where values can be updated.
# In a graph, each variable corresponds corresponds to a node
storageVal = tf.Variable(val *singleValue, name = "StorageVaraible" )
# The below command takes care of the Initialization part.
# In the absence of this command; you would receive a
# 'Failed precondition error; which is to do with uninitialised varaibles'
initialization = tf.initialize_all_variables()
# USING CONTEXT MANAGER SESSION FOR EXECUTION
with tf.Session() as session:
session.run(initialization)
print "Product is: ",session.run(storageVal)
print "The shape of the product tensor: ",(storageVal.get_shape())
# ```
Tensor dimensionality is denoted using shape, rank and dimension number. Tensor with rank 0 is a scalar. A rank 1 tensor is a vector. A rank 2 tensor is a matrix, ie you can access element using 2 variables like tensor[i,j]; here the dimension is 2D. In the case of rank 3, access it using tensor[i,j,k];here dimension is 3D.
The tensor elements can be subjected to slice, tile operations. The below example deals with slice command.
Example 2:
In [ ]:
import tensorflow as tf
import numpy as np
val1 = tf.placeholder(tf.float32,shape = (8,8))
val2 = tf.placeholder(tf.float32, shape = (8,8))
randomNos2 = np.random.rand(8,8)
product = tf.matmul(val1,val2)
inpSlice = tf.constant([[[1,11,111],[2,22,222]],
[[3,33,333],[4,44,444]],
[[5,55,555],[6,66,666]]])
init = tf.initialize_all_variables()
# Softplus activation function computes log(exp(featurePoints)+1)
softPlus = tf.nn.softplus(product)
with tf.Session() as sess:
sess.run(init)
print (sess.run(product,feed_dict={val1:randomNos2, val2:randomNos2}))
result = sess.run(softPlus,feed_dict={val1:randomNos2, val2:randomNos2})
# The slicing function takes the arguments input, size and begin
print (result)
print "SLICING ON TENSORS: "
print "Applying slicing on softplus output",(sess.run(tf.slice(result, [1, 1], [1, 1])))
print "Access1: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,1])))
print "Access2: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,2])))
print "Access3: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,3])))
# Interpretation of slice arguments:
# The begin parameter begin(i,j,k): will enable access to 'i' ranks; spanning 'j' dimensions;
# the number of values returned is 'k'
# This will access elements at the 2ranks, from both dimensions- starting from the range 0 as size is specified as 0
print "Access7: ",(sess.run(tf.slice(inpSlice,[0,0,0],[2,1,1])))
print "Access8: ",(sess.run(tf.slice(inpSlice,[0,0,0],[3,2,1])))
Example 3: Tensor flow offers a variety of basic arithemtic functions to do computations. A few of them are implemented below for ease.
# The following program shows the basic arithemtic operations using Tenspr Flow and the # arguments required to be passed.
# Takes arguments: x input,y input, name=None (optional; without space)
# The x input and y input should be tensors;Must be one of the types: float32, float 64, uint8, int16, int32, int 64, complex64
# y should be of the same type as x input; variables are case sensitive
import tensorflow as tf
# Input constant float values.
a = tf.constant([[4.0, 8.0,10.0],[20.0,40.0,80.0]])
b = tf.constant([[8.0, 16.0,15.0],[25.0,40.0,50.0]])
# The common arithemtic operators can be implemented as below:
sumVal = tf.add (a,b, name = "AdditiveOp")
#TrueDiv function produces floating point quotients; Here integer operands are converted to floating point values
#If integer values of type int8 or int16: Casted to float32 / If integer values of type int32 or int64: Casted to float64
#If a and b are of different types, it gives error
QuoValtrue = tf.truediv(a,b)
# The cross function provides the cross product of 2 values whose innermost dimension 3
crossVal = tf.cross(a,b)
#Provides remainder with respect to each element in the tensor
modVal = tf.mod(a,b)
# Provides absolute value of input tensor
negTensor = tf.constant([-2,-4,-6])
absVal = tf.abs(negTensor)
# Produces -1 if negative, 0 if value is 0 & 1 if positive value
# The input tensor should be either float32, float64, int32, int64
signVal = tf.sign(negTensor, name = "direction")
# Provides the reciprocal of a value
# The input tensor should be of type float32,float64,int32,complex64,int64
invVal = tf.inv(a, name = "Inverse")
# The below function finds the reciprocal of the square root a number; tensor accepts :float32,float64,int32,complex64,int64
# Mathematical functions: tf.sqrt(a), tf.round(a): Rounds values to ceiling value; the tensor should be either float or double,
# tf.square(a): finds the square
revSqVal = tf.rsqrt(a, name = "ReverseSqrt")
# The below function computes to the power of the tensor value;If output exceeds the limit of datatype allowed, it gives 0 as output for that tensor element
powVal = tf.pow(absVal,absVal)
#The exponential value is obtained for tensors of types: float32,float64,complex64,int64
#Similarly, natural log is found using tf.log(<input tensor>) function
expVal = tf.exp(a, name = "Exponential")
#Other functions include:
# tf.ceil(a,name = "Ceil";produces smaller value > x & tf.floor(a, name = "floor");produces bigger integer < x;
# both accepts values of type :float32 or float64
# tf.maximum function & tf.minimum produces the max between two values
tf.maximum(a,b,name = "Max")
tf.minimum(a,b,name = "Min")
init = tf.initialize_all_variables()
# With the below commands, the Op gets executed
with tf.Session() as sess:
sess.run(init)
print "Additive: "
print (sess.run(sumVal))
# sub,div,mul resluts can be obtained in the same manner.
print "Quotient from Truediv function: "
print (sess.run(QuoValtrue))
print "Cross Product: ",(sess.run(crossVal))
print "Element by element Mod value: ",(sess.run(modVal))
print "Absolute: ",(sess.run(absVal))
print "Element by element Sign value: ",(sess.run(signVal))
print "Element to the Power a: ",(sess.run(powVal))
print "Function to generate exponential values: ",(sess.run(expVal))
Example 4: The objective of this program is to generate a simple tensor board that describes the main events in the program below. The following tensor flow program produces a tensor using random numbers, and reducing functions that calculates: mean, sum, products of tensor elements are carried out:
In [ ]:
import tensorflow as tf
import numpy as np
# The below command creates Op nodes.
# A constant tensor is denoted using a circle.
with tf.name_scope('Values') as scope:
x = tf.Variable(0, name='x')
y = tf.constant(5)
z = x+y
model = tf.initialize_all_variables()
with tf.Session() as session:
for i in range(5):
session.run(model)
print "At Loop: ",i
data = np.random.randint(10, size=10)
for j in range(len(data)-1):
sumVal = tf.add(z,data[0:j+1])
print "Sum is:",
result = session.run(sumVal)
print result
print "Last sumval: ",result
with tf.name_scope('Mean') as scope:
mean = tf.reduce_mean(tf.to_float((sumVal)))
mean = tf.reduce_mean(tf.to_float((sumVal)))
print "Reduced Mean is : ",session.run(tf.reduce_mean(tf.to_float((sumVal))))
_=tf.scalar_summary('Mean', mean)
merged = tf.merge_all_summaries()
writer = tf.train.SummaryWriter("/tmp/mnist_logs", session.graph_def)
model = tf.initialize_all_variables()
session.run(model)
Tensor Board provides a visualization of the model that you create. It deals with 2 types of connections: Control dependency(denoted using dotted lines) and Data dependency(denoted using solid lines). To record the variations produced by a particular function, provide them as inputs to scalar_summary ops with a tag name. Histogram_summary provides the distribution of an output variable from a layer. tf.merge_all_summaries will merge the summaries created within the program, which is then directed to a summary_writer. Specify a logdir for summary writer to write the events.
To get the tensor board launched as below:
tensorboard --logdir = /tmp/mnist_logs http:<server>:6006Example 5: Consider a simple model which calculates a new state using the error obtained from the previous state.The functions add, sub forms a node in the graph with their own definite operations. They produce zero or more tensors as inputs for the next set of nodes:
In [ ]:
#```python
import tensorflow as tf
# Variables: Memory Buffers, with tensors that update and hold values
# Needs explicit initializations
# tf.Variables creates a variable with value 1.
currState = tf.Variable(1)
constValue = tf.constant(5)
cProd = currState *5
#Subtraction operation: cProd - currState
interValue = tf.sub(cProd,currState)
#Addition
newState = tf.add(currState,interValue)
#The below operation is performed once the run command is executed
update = tf.assign(currState,newState)
#Whenever a variable is created, it is an empty node. It is only by initializing them the
# variable gets filled with the content, which is a value
init_op = tf.initialize_all_variables()
#Within Session class, the operations are executed and datas in the form of
#tensors are evaluated
with tf.Session() as sess:
sess.run(init_op)
#The below statement will execute to generate the currState value
print(sess.run(currState))
for _ in range(5):
sess.run(update)
#print(sess.run(currState))
print (sess.run([currState, interValue]))
#```
They are used to provide input whenever tensor flow is made to run computation. To mention a dimension of any length, None can be specified in the shape argument. The below example which deals with matrix multiplication of tensors-generated using the random function, provides an understanding of working with placeholder values.
Example 6:
In [ ]:
# The objective of this program is to familiarise with the use of placeholders and
# a few matrix arithmetic functions
# Using placeholder value inidicates directing the tensor
# value into Operations in graph.
# Placeholders should always fed with feed_dict argument while using Session.run()/tensor.eval()/operation.run() command
# Takes arguments: dtype, shape(optional),name(optional)
# the dtype elements indicate the type of tensor. For eg: float32, complex64, int8, qint8, bool, string etc
import tensorflow as tf
import numpy as np
val1 = tf.placeholder(tf.float32,shape = (8,8))
val2 = tf.placeholder(tf.float32, shape = (8,8))
randomNos2 = np.random.rand(8,8)
# tf.random_normal : Generates random numbers following normal distribution with mean =0.0, stddev = 1.0, dtype = tf.float32
randomNormal = tf.Variable(tf.random_normal([8,8]),name = 'NormalDistribution')
# Below command creates an OP (Operation) of type MatMul;
product = tf.matmul(val1,val2)
matInv = tf.matrix_inverse(product, name="MatrixInverse")
matDeter = tf.matrix_determinant(val1,name ="MatrixDeterminant")
eigenVec = tf.self_adjoint_eig(randomNormal, name = "EigenVectors")
# Unless specified, both val1 and val2 has to be of placeholder type input or they both
# have to be of Variable type which doesnt require initialization using feed_dict
# addition = tf.add(randomNormal,val1) # This command will give an invalid argument error as val1
#randomNormal are variable and placeholder values respectively.
init = tf.initialize_all_variables()
# With the below commands, the Op gets executed
with tf.Session() as sess:
sess.run(init)
print (sess.run(product,feed_dict={val1:randomNos2, val2:randomNos2}))
print "INVERSE: ",(sess.run(matInv,feed_dict={val1:randomNos2, val2:randomNos2}))
# get_shape(): Returns the shape of a tensor; which would be either Fully-known shape(size and number of dimensions are known) or
# Partially-known shape(number of dimensions known but size is unknown) or Unknown Shape(as the name says, number of
# dimensions and size are unknown)
print "SHAPE OF PRODUCT TENSOR: ",(product.get_shape())
print "MATRIX DETERMINANT: ",(sess.run(matDeter,feed_dict={val1:randomNos2}))
print "EIGEN VECTORS: ",sess.run(eigenVec)
#print "Addition",sess.run(addition)
In the biological world, Neurons does the decision making process. The below figure shows a 3 layer neural network: with 2 hidden layers and one output layer. Each element in the input layer(Blue layer), hidden layer(yellow layer) and output layer(green layer) are considered as neurons that communicates between the layers.
A Perceptron neuron is an artificial neuron, that takes many binary inputs and gives binary output. To compute result, weights (real numbers indicating the role of the inputs in the corresponding output) were used. The weighted sum of the product of input and weights, added with bias are compared with a threshold value to decide the neuron output as 0 or 1. The output from one layer is fed as input to the next layer. Such a network is called as a feed forward network. Each element in the hidden and output layer is called a neuron and it just does some simple tasks like: Reads input data, Process it and gives the output. By having a network of such a model we make it to ‘learn’, producing intelligent results that can be converged towards a decision making process. Perceptron is a basic network model which gets a weighted input from the previous layer. The results from a perceptron is found to be flipping as the weights / bias values change.
Sigmoid neurons are considered to be better than perceptron. Though with some similarities with perceptron, sigmoid neurons deals with exponent values of inputs between 0 and 1. The output is a real number between 0 and 1. Sigmoid neuron is a smoothed version of a perceptron.
There are many other learning algorithms and they will make our model learn and adapt the weights and biases of our model so as to provide us with the decision made. The activation functions produces a tensor of the same shape as their inputs.
The most common activation function is called the Sigmoid.
$$\sigma(x) = 1/(1+e^{-x})$$Other activation functions include:Softplus, Sigmoid, tanh, elu and tensor flow have their respective commands for these activation functions.
Softmax Regression used for classification operation. It is a supervised learning algorithm. It has an added feature of being able to classify the input data to multiple groups, rather than to just 2 groups. In the case of MNIST digit recognition task, the number of groups is 10. Softmax regression requires evidence information, which is calculated using bias, weights and the input image. A weighted sum of the tallies up the evidence that it is in a specific class. A positive weight indicates that the evidence is in favor and belongs to that class. If the weight is negative then the pixel is evidence against the image belonging to that class.
Lets a try a simple implementation on tensor flow using the activation functions: Relu
Example 5:
In [1]:
import tensorflow as tf
import numpy as np
input = tf.placeholder("float",[1,3])
weights = tf.Variable(tf.random_normal([3,3]),name = "Weights")
y = tf.matmul(input,weights)
relu_output = tf.nn.relu(y)
softmax = tf.nn.softmax(relu_output)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
print "Matrix PRoduct: ",sess.run(y,feed_dict={input:np.array([[1.0,2.0,3.0]])})
print "Weights: ",sess.run(weights)
print "Applying Activation function Softmax: ",sess.run(softmax,feed_dict={input:np.array([[1.0,2.0,3.0]])})