T81-558: Applications of Deep Neural Networks

Class 2: Introduction to TensorFlow.

Neural Network Basics

Neural networks were one of the first machine learning models. Their popularity has fallen twice and is now on its third rise. Deep learning implies the use of neural networks. The "deep" in deep learning refers to a neural network with many hidden layers. Because neural networks have been around for so long, they have quite a bit of baggage. Many different training algorithms, activation/transfer functions, and structures have been added over the years. This course is only concerned with the latest, most current state of the art techniques for deep neural networks. I am not going to spend any time discussing the history of neural networks. If you would like to learn about some of the more classic structures of neural networks, there are several chapters dedicated to this in your course book. For the latest technology, I wrote an article for the Society of Actuaries on deep learning as the third generation of neural networks.

Neural networks accept input and produce output. The input to a neural network is called the feature vector. The size of this vector is always a fixed length. Changing the size of the feature vector means recreating the entire neural network. Though the feature vector is called a "vector," this is not always the case. A vector implies a 1D array. Historically the input to a neural network was always 1D. However, with modern neural networks you might see inputs, such as:

  • 1D Vector - Classic input to a neural network, similar to rows in a spreadsheet. Common in predictive modeling.
  • 2D Matrix - Grayscale image input to a convolutional neural network (CNN).
  • 3D Matrix - Color image input to a convolutional neural network (CNN).
  • nD Matrix - Higher order input to a CNN.

Prior to CNN's, the image input was sent to a neural network simply by squashing the image matrix into a long array by placing the image's rows side-by-side. CNNs are different, as the nD matrix literally passes through the neural network layers.

Initially this course will focus upon 1D input to neural networks. However, later sessions will focus more heavily upon higher dimension input.

Dimensions The term dimension can be confusing in neural networks. In the sense of a 1D input vector, dimension refers to how many elements are in that 1D array. For example a neural network with 10 input neurons has 10 dimensions. However, now that we have CNN's, the input has dimensions too. The input to the neural network will usually have 1, 2 or 3 dimensions. 4 or more dimensions is unusual. You might have a 2D input to a neural network that has 64x64 pixels. This would result in 4,096 input neurons. This network is either 2D or 4,096D, depending on which set of dimensions you are talking about!

Classification or Regression

Like many models, neural networks can function in classification or regression:

  • Regression - You expect a number as your neural network's prediction.
  • Classification - You expect a class/category as your neural network's prediction.

The following shows a classification and regression neural network:

Notice that the output of the regression neural network is numeric and the output of the classification is a class. Regression, or two-class classification, networks always have a single output. Classification neural networks have an output neuron for each class.

The following diagram shows a typical neural network:

There are usually four types of neurons in a neural network:

  • Input Neurons - Each input neuron is mapped to one element in the feature vector.
  • Hidden Neurons - Hidden neurons allow the neural network to abstract and process the input into the output.
  • Output Neurons - Each output neuron calculates one part of the output.
  • Context Neurons - Holds state between calls to the neural network to predict.
  • Bias Neurons - Work similar to the y-intercept of a linear equation.

These neurons are grouped into layers:

  • Input Layer - The input layer accepts feature vectors from the dataset. Input layers usually have a bias neuron.
  • Output Layer - The output from the neural network. The output layer does not have a bias neuron.
  • Hidden Layers - Layers that occur between the input and output layers. Each hidden layer will usually have a bias neuron.

Neuron Calculation

The output from a single neuron is calculated according to the following formula:

$ f(x,\theta) = \phi(\sum_i(\theta_i \cdot x_i)) $

The input vector (x) represents the feature vector and the vector $\theta$ represents the weights. To account for the bias neuron, a value of 1 is always appended to the end of the input feature vector. This causes the last weight to be interpreted as a bias value that is simply added to the summation. The $\phi$ is the transfer/activation function.

Consider using the above equation to calculate the output from the following neuron:

The above neuron has 2 inputs plus the bias as a third. This neuron might accept the following input feature vector:

[1,2]

To account for the bias neuron, a 1 is appended, as follows:

[1,2,1]

The weights for a 3-input layer (2 real inputs + bias) will always have an additional weight, for the bias. A weight vector might be:

[ 0.1, 0.2, 0.3]

To calculate the summation, perform the following:

0.1*1 + 0.2*2 + 0.3*3 = 1.4

The value of 1.4 is passed to the $\theta$ function, which represents the activation function.

Activation Functions

Activation functions, also known as transfer functions, are used to calculate the output of each layer of a neural network. Historically neural networks have used a hyperbolic tangent, sigmoid/logistic, or linear activation function. However, modern deep neural networks primarily make use of the following activation functions:

  • Rectified Linear Unit (ReLU) - Used for the output of hidden layers.
  • Softmax - Used for the output of classification neural networks. Softmax Example
  • Linear - Used for the output of regression neural networks (or 2-class classification).

The ReLU function is calculated as follows:

$ \phi(x) = \max(0, x) $

The Softmax is calculated as follows:

$ \phi_i(z) = \frac{e^{z_i}}{\sum\limits_{j \in group}e^{z_j}} $

The Softmax activation function is only useful with more than one output neuron. It ensures that all of the output neurons sum to 1.0. This makes it very useful for classification where it shows the probability of each of the classes as being the correct choice.

To experiment with the Softmax, click here.

The linear activation function is essentially no activation function:

$ \phi(x) = x $

For regression problems, this is the activation function of choice.

Why ReLU?

Why is the ReLU activation function so popular? It was one of the key improvements to neural networks that makes deep learning work. Prior to deep learning, the sigmoid activation function was very common:

$ \phi(x) = \frac{1}{1 + e^{-x}} $

The graph of the sigmoid function is shown here:

Neural networks are often trained using gradient descent. To make use of gradient descent, it is necessary to take the derivative of the activation function. This allows the partial derivatives of each of the weights to be calculated with respect to the error function. A derivative is the instantaneous rate of change:

The derivative of the sigmoid function is given here:

$ \phi'(x)=\phi(x)(1-\phi(x)) $

This derivative is often given in other forms. The above form is used for computational efficiency. To see how this derivative was taken, see this.

The graph of the sigmoid derivative is given here:

The derivative quickly saturates to zero as x moves from zero. This is not a problem for the derivative of the ReLU, which is given here:

$ \phi'(x) = \begin{cases} 1 & x > 0 \\ 0 & x \leq 0 \end{cases} $

Why are Bias Neurons Needed?

The activation functions seen in the previous section specifies the output of a single neuron. Together, the weight and bias of a neuron shape the output of the activation to produce the desired output. To see how this process occurs, consider the following equation. It represents a single-input sigmoid activation neural network.

$ f(x,w,b) = \frac{1}{1 + e^{-(wx+b)}} $

The x variable represents the single input to the neural network. The w and b variables specify the weight and bias of the neural network. The above equation is a combination of the weighted sum of the inputs and the sigmoid activation function. For this section, we will consider the sigmoid function because it clearly demonstrates the effect that a bias neuron has.

The weights of the neuron allow you to adjust the slope or shape of the activation function. The following figure shows the effect on the output of the sigmoid activation function if the weight is varied:

The above diagram shows several sigmoid curves using the following parameters:

f(x,0.5,0.0)
f(x,1.0,0.0)
f(x,1.5,0.0)
f(x,2.0,0.0)

To produce the curves, we did not use bias, which is evident in the third parameter of 0 in each case. Using four weight values yields four different sigmoid curves in the above figure. No matter the weight, we always get the same value of 0.5 when x is 0 because all of the curves hit the same point when x is 0. We might need the neural network to produce other values when the input is near 0.5.

Bias does shift the sigmoid curve, which allows values other than 0.5 when x is near 0. The following figure shows the effect of using a weight of 1.0 with several different biases:

The above diagram shows several sigmoid curves with the following parameters:

f(x,1.0,1.0)
f(x,1.0,0.5)
f(x,1.0,1.5)
f(x,1.0,2.0)

We used a weight of 1.0 for these curves in all cases. When we utilized several different biases, sigmoid curves shifted to the left or right. Because all the curves merge together at the top right or bottom left, it is not a complete shift.

When we put bias and weights together, they produced a curve that created the necessary output from a neuron. The above curves are the output from only one neuron. In a complete network, the output from many different neurons will combine to produce complex output patterns.

TensorFlow - Google

https://www.tensorflow.org/

What version of TensorFlow do you have?


In [1]:
import tensorflow as tf
print("Tensor Flow Version: {}".format(tf.__version__))


Tensor Flow Version: 0.8.0

Installing TensorFlow

TensorFlow Setup Instructions

Why TensorFlow

  • Supported by Google
  • Works well on Linux/Mac
  • Excellent GPU support
  • Python is an easy to learn programming language
  • Python is extremely popular in the data science community

Other Deep Learning Tools

TensorFlow is not the only only game in town. These are some of the best supported alternatives. Most of these are written in C++. In order of my own preference (I have used all of these):

  • Theano - Python, from the academics that created deep learning. GPU support.
  • Computational Network Toolkit (CNTK) - Microsoft. Support for Windows/Linux, command line only. Bindings for predictions for C#/Python. GPU support.
  • Caffe Academic project from Berkeley. Python.
  • H2O - Java based. Supports all major platforms. Limited support for computer vision. No GPU support.
  • Deeplearning4J - Java based. Supports all major platforms. GPU support in Java!
  • PaddlePaddle - Baidu's offering, very new.

Other/Advanced Deep Learning

Torch is used by Google DeepMind, the Facebook AI Research Group, IBM, Yandex and the Idiap Research Institute. It has been used for some of the most advanced deep learning projects in the world. However, it requires the LUA) programming language. It is very advanced, but it is not mainstream. I have not worked with Torch (yet!).

Using TensorFlow

TensorFlow is a low-level mathematics API, similar to Numpy. However, unlike Numpy, TensorFlow is built for deep learning. TensorFlow works by allowing you to define compute graphs with Python. In this regard, it is similar to Spark. TensorFlow compiles these compute graphs into highly efficient C++/CUDA code.

The TensorBoard command line utility can be used to view these graphs. The iris neural network's graph used in this class is shown here:

Expanding the DNN gives:

Using Skflow

Skflow is a layer on top of Tensorflow that makes it much easier to create neural networks. Rather than define the graphs, like you see above, you define the individual layers of the network with a much more high level API. Unless you are performing research into entirely new structures of deep neural networks it is unlikely that you need to program TensorFlow directly.

For this class, we will use SKFLOW, rather than direct TensorFlow

SKFLOW is built into TensorFlow, as of v0.8. This makes it very easy to use.

All examples in this class will use SKFLOW, and you are encouraged to use it for the programming assignments.

Helpful Functions for Tensorflow

The following functions will be used in conjunction with TensorFlow to help preprocess the data. It is okay to just use them. For better understanding, try to understand how they work.

These functions allow you to build the feature vector for a neural network. Consider the following:

  • Predictors/Inputs
    • Fill any missing inputs with the median for that column. Use missing_median.
    • Encode textual/categorical values with encode_text_dummy.
    • Encode numeric values with encode_numeric_zscore.
  • Output
    • Discard rows with missing outputs.
    • Encode textual/categorical values with encode_text_index.
    • Do not encode output numeric values.
  • Produce final feature vectors (x) and expected output (y) with to_xy.

In [2]:
import numpy as np
import pandas as pd

# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)    
def encode_text_dummy(df,name):
    dummies = pd.get_dummies(df[name])
    for x in dummies.columns:
        dummy_name = "{}-{}".format(name,x)
        df[dummy_name] = dummies[x]
    df.drop(name, axis=1, inplace=True)
    
# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).    
def encode_text_index(df,name): 
    le = preprocessing.LabelEncoder()
    df[name] = le.fit_transform(df[name])
    return le.classes_
                
# Encode a numeric column as zscores    
def encode_numeric_zscore(df,name,mean=None,sd=None):
    if mean is None:
        mean = df[name].mean()
        
    if sd is None:
        sd = df[name].std()
        
    df[name] = (df[name]-mean)/sd
    
# Convert all missing values in the specified column to the median
def missing_median(df, name):
    med = df[name].median()
    df[name] = df[name].fillna(med)

# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs
def to_xy(df,target):
    result = []
    for x in df.columns:
        if x != target:
            result.append(x)
    return df.as_matrix(result).astype(np.float32),df[target].astype(np.int32)

Simple TensorFlow Classification: Iris

This is a very simple example of how to perform the Iris classification using TensorFlow. The iris.csv file is used, rather than using the built-in files that many of the Google examples require.

Make sure that you always run previous code blocks. If you run the code block below, without the codeblock above, you will get errors


In [3]:
import tensorflow.contrib.learn as skflow
from sklearn import metrics
import pandas as pd
import os
from sklearn import preprocessing

path = "./data/"

# Read iris dataset
filename_read = os.path.join(path,"iris.csv")
df = pd.read_csv(filename_read,na_values=['NA','?'])

# Extract just the columns we shall predict on
encode_numeric_zscore(df,'sepal_l')
encode_numeric_zscore(df,'sepal_w')
encode_numeric_zscore(df,'petal_l')
encode_numeric_zscore(df,'petal_w')
species = encode_text_index(df,'species')

# Create x(predictors) and y (expected outcome)
x,y = to_xy(df,'species')

# Create a deep neural network with 3 hidden layers of 10, 20, 10
classifier = skflow.TensorFlowDNNClassifier(hidden_units=[10, 20, 10], n_classes=3,steps=200)

# Fit/train neural network
classifier.fit(x, y)

# Measure accuracy
score = metrics.accuracy_score(y, classifier.predict(x))
print("Final score: {}".format(score))

# How to make many predictions
pred = classifier.predict(x)
predDF = pd.DataFrame(pred)
pred_nameDF = pd.DataFrame(species[pred])
actual_nameDF = pd.DataFrame(species[df['species']])
df2 = pd.concat([df,predDF,pred_nameDF,actual_nameDF],axis=1)
df2.columns = ['sepal_l','sepal_w','petal_l','petal_w','expected','predicted','expected_str','predicted_str']
df2


Step #100, epoch #20, avg. train loss: 0.26326
Step #200, epoch #40, avg. train loss: 0.05465
Final score: 0.9933333333333333
Out[3]:
sepal_l sepal_w petal_l petal_w expected predicted expected_str predicted_str
0 -0.897674 1.015602 -1.335752 -1.311052 0 0 Iris-setosa Iris-setosa
1 -1.139200 -0.131539 -1.335752 -1.311052 0 0 Iris-setosa Iris-setosa
2 -1.380727 0.327318 -1.392399 -1.311052 0 0 Iris-setosa Iris-setosa
3 -1.501490 0.097889 -1.279104 -1.311052 0 0 Iris-setosa Iris-setosa
4 -1.018437 1.245030 -1.335752 -1.311052 0 0 Iris-setosa Iris-setosa
5 -0.535384 1.933315 -1.165809 -1.048667 0 0 Iris-setosa Iris-setosa
6 -1.501490 0.786174 -1.335752 -1.179859 0 0 Iris-setosa Iris-setosa
7 -1.018437 0.786174 -1.279104 -1.311052 0 0 Iris-setosa Iris-setosa
8 -1.743017 -0.360967 -1.335752 -1.311052 0 0 Iris-setosa Iris-setosa
9 -1.139200 0.097889 -1.279104 -1.442245 0 0 Iris-setosa Iris-setosa
10 -0.535384 1.474458 -1.279104 -1.311052 0 0 Iris-setosa Iris-setosa
11 -1.259964 0.786174 -1.222456 -1.311052 0 0 Iris-setosa Iris-setosa
12 -1.259964 -0.131539 -1.335752 -1.442245 0 0 Iris-setosa Iris-setosa
13 -1.863780 -0.131539 -1.505695 -1.442245 0 0 Iris-setosa Iris-setosa
14 -0.052331 2.162743 -1.449047 -1.311052 0 0 Iris-setosa Iris-setosa
15 -0.173094 3.080455 -1.279104 -1.048667 0 0 Iris-setosa Iris-setosa
16 -0.535384 1.933315 -1.392399 -1.048667 0 0 Iris-setosa Iris-setosa
17 -0.897674 1.015602 -1.335752 -1.179859 0 0 Iris-setosa Iris-setosa
18 -0.173094 1.703886 -1.165809 -1.179859 0 0 Iris-setosa Iris-setosa
19 -0.897674 1.703886 -1.279104 -1.179859 0 0 Iris-setosa Iris-setosa
20 -0.535384 0.786174 -1.165809 -1.311052 0 0 Iris-setosa Iris-setosa
21 -0.897674 1.474458 -1.279104 -1.048667 0 0 Iris-setosa Iris-setosa
22 -1.501490 1.245030 -1.562342 -1.311052 0 0 Iris-setosa Iris-setosa
23 -0.897674 0.556746 -1.165809 -0.917474 0 0 Iris-setosa Iris-setosa
24 -1.259964 0.786174 -1.052513 -1.311052 0 0 Iris-setosa Iris-setosa
25 -1.018437 -0.131539 -1.222456 -1.311052 0 0 Iris-setosa Iris-setosa
26 -1.018437 0.786174 -1.222456 -1.048667 0 0 Iris-setosa Iris-setosa
27 -0.776911 1.015602 -1.279104 -1.311052 0 0 Iris-setosa Iris-setosa
28 -0.776911 0.786174 -1.335752 -1.311052 0 0 Iris-setosa Iris-setosa
29 -1.380727 0.327318 -1.222456 -1.311052 0 0 Iris-setosa Iris-setosa
... ... ... ... ... ... ... ... ...
120 1.276066 0.327318 1.100097 1.443994 2 2 Iris-virginica Iris-virginica
121 -0.293857 -0.590395 0.646916 1.050416 2 2 Iris-virginica Iris-virginica
122 2.242172 -0.590395 1.666574 1.050416 2 2 Iris-virginica Iris-virginica
123 0.551486 -0.819823 0.646916 0.788031 2 2 Iris-virginica Iris-virginica
124 1.034539 0.556746 1.100097 1.181609 2 2 Iris-virginica Iris-virginica
125 1.638355 0.327318 1.270040 0.788031 2 2 Iris-virginica Iris-virginica
126 0.430722 -0.590395 0.590269 0.788031 2 2 Iris-virginica Iris-virginica
127 0.309959 -0.131539 0.646916 0.788031 2 2 Iris-virginica Iris-virginica
128 0.672249 -0.590395 1.043450 1.181609 2 2 Iris-virginica Iris-virginica
129 1.638355 -0.131539 1.156745 0.525645 2 2 Iris-virginica Iris-virginica
130 1.879882 -0.590395 1.326688 0.919223 2 2 Iris-virginica Iris-virginica
131 2.483699 1.703886 1.496631 1.050416 2 2 Iris-virginica Iris-virginica
132 0.672249 -0.590395 1.043450 1.312801 2 2 Iris-virginica Iris-virginica
133 0.551486 -0.590395 0.760211 0.394453 2 2 Iris-virginica Iris-virginica
134 0.309959 -1.049251 1.043450 0.263260 2 2 Iris-virginica Iris-virginica
135 2.242172 -0.131539 1.326688 1.443994 2 2 Iris-virginica Iris-virginica
136 0.551486 0.786174 1.043450 1.575187 2 2 Iris-virginica Iris-virginica
137 0.672249 0.097889 0.986802 0.788031 2 2 Iris-virginica Iris-virginica
138 0.189196 -0.131539 0.590269 0.788031 2 2 Iris-virginica Iris-virginica
139 1.276066 0.097889 0.930154 1.181609 2 2 Iris-virginica Iris-virginica
140 1.034539 0.097889 1.043450 1.575187 2 2 Iris-virginica Iris-virginica
141 1.276066 0.097889 0.760211 1.443994 2 2 Iris-virginica Iris-virginica
142 -0.052331 -0.819823 0.760211 0.919223 2 2 Iris-virginica Iris-virginica
143 1.155302 0.327318 1.213393 1.443994 2 2 Iris-virginica Iris-virginica
144 1.034539 0.556746 1.100097 1.706379 2 2 Iris-virginica Iris-virginica
145 1.034539 -0.131539 0.816859 1.443994 2 2 Iris-virginica Iris-virginica
146 0.551486 -1.278680 0.703564 0.919223 2 2 Iris-virginica Iris-virginica
147 0.793012 -0.131539 0.816859 1.050416 2 2 Iris-virginica Iris-virginica
148 0.430722 0.786174 0.930154 1.443994 2 2 Iris-virginica Iris-virginica
149 0.068433 -0.131539 0.760211 0.788031 2 2 Iris-virginica Iris-virginica

150 rows × 8 columns


In [4]:
import numpy as np
# How to make predictions one at a time, in a loop
for fv in x:
    # Flip the feature vector to the right shape
    fv2 = fv.reshape((1,4))
    #  Compute a prediction for the 3 classes, this will return 0, 1 or 2.
    pred = classifier.predict(fv2)
    # Turn the numeric prediction to a text string (e.g. Iris-virginica)
    pred_name = species[pred][0]
    # Output result
    print("{} : {} ({})".format(fv,pred,pred_name))


[-0.8976739   1.01560199 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-1.13920045 -0.13153881 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-1.38072705  0.32731751 -1.39239931 -1.3110522 ] : [0] (Iris-setosa)
[-1.50149035  0.09788935 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  1.24503016 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-0.535384    1.93331468 -1.16580868 -1.04866683] : [0] (Iris-setosa)
[-1.50149035  0.78617382 -1.33575165 -1.17985952] : [0] (Iris-setosa)
[-1.01843715  0.78617382 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.74301696 -0.36096698 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-1.13920045  0.09788935 -1.27910399 -1.44224477] : [0] (Iris-setosa)
[-0.535384    1.47445834 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.25996375  0.78617382 -1.22245634 -1.3110522 ] : [0] (Iris-setosa)
[-1.25996375 -0.13153881 -1.33575165 -1.44224477] : [0] (Iris-setosa)
[-1.86378026 -0.13153881 -1.50569463 -1.44224477] : [0] (Iris-setosa)
[-0.05233077  2.16274285 -1.44904697 -1.3110522 ] : [0] (Iris-setosa)
[-0.17309406  3.08045554 -1.27910399 -1.04866683] : [0] (Iris-setosa)
[-0.535384    1.93331468 -1.39239931 -1.04866683] : [0] (Iris-setosa)
[-0.8976739   1.01560199 -1.33575165 -1.17985952] : [0] (Iris-setosa)
[-0.17309406  1.70388651 -1.16580868 -1.17985952] : [0] (Iris-setosa)
[-0.8976739   1.70388651 -1.27910399 -1.17985952] : [0] (Iris-setosa)
[-0.535384    0.78617382 -1.16580868 -1.3110522 ] : [0] (Iris-setosa)
[-0.8976739   1.47445834 -1.27910399 -1.04866683] : [0] (Iris-setosa)
[-1.50149035  1.24503016 -1.56234229 -1.3110522 ] : [0] (Iris-setosa)
[-0.8976739   0.55674565 -1.16580868 -0.91747409] : [0] (Iris-setosa)
[-1.25996375  0.78617382 -1.05251336 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715 -0.13153881 -1.22245634 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  0.78617382 -1.22245634 -1.04866683] : [0] (Iris-setosa)
[-0.7769106   1.01560199 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-0.7769106   0.78617382 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-1.38072705  0.32731751 -1.22245634 -1.3110522 ] : [0] (Iris-setosa)
[-1.25996375  0.09788935 -1.22245634 -1.3110522 ] : [0] (Iris-setosa)
[-0.535384    0.78617382 -1.27910399 -1.04866683] : [0] (Iris-setosa)
[-0.7769106   2.39217091 -1.27910399 -1.44224477] : [0] (Iris-setosa)
[-0.41462067  2.6215992  -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-1.13920045  0.09788935 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  0.32731751 -1.44904697 -1.3110522 ] : [0] (Iris-setosa)
[-0.41462067  1.01560199 -1.39239931 -1.3110522 ] : [0] (Iris-setosa)
[-1.13920045  1.24503016 -1.33575165 -1.44224477] : [0] (Iris-setosa)
[-1.74301696 -0.13153881 -1.39239931 -1.3110522 ] : [0] (Iris-setosa)
[-0.8976739   0.78617382 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  1.01560199 -1.39239931 -1.17985952] : [0] (Iris-setosa)
[-1.62225366 -1.73753595 -1.39239931 -1.17985952] : [0] (Iris-setosa)
[-1.74301696  0.32731751 -1.39239931 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  1.01560199 -1.22245634 -0.78628147] : [0] (Iris-setosa)
[-0.8976739   1.70388651 -1.05251336 -1.04866683] : [0] (Iris-setosa)
[-1.25996375 -0.13153881 -1.33575165 -1.17985952] : [0] (Iris-setosa)
[-0.8976739   1.70388651 -1.22245634 -1.3110522 ] : [0] (Iris-setosa)
[-1.50149035  0.32731751 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[-0.6561473   1.47445834 -1.27910399 -1.3110522 ] : [0] (Iris-setosa)
[-1.01843715  0.55674565 -1.33575165 -1.3110522 ] : [0] (Iris-setosa)
[ 1.39682889  0.32731751  0.53362089  0.26325998] : [1] (Iris-versicolor)
[ 0.67224902  0.32731751  0.42032558  0.39445266] : [1] (Iris-versicolor)
[ 1.27606559  0.09788935  0.64691621  0.39445266] : [1] (Iris-versicolor)
[-0.41462067 -1.73753595  0.13708732  0.13206729] : [1] (Iris-versicolor)
[ 0.79301232 -0.59039515  0.47697324  0.39445266] : [1] (Iris-versicolor)
[-0.17309406 -0.59039515  0.42032558  0.13206729] : [1] (Iris-versicolor)
[ 0.55148572  0.55674565  0.53362089  0.52564532] : [1] (Iris-versicolor)
[-1.13920045 -1.50810778 -0.25944623 -0.26151073] : [1] (Iris-versicolor)
[ 0.91377568 -0.36096698  0.47697324  0.13206729] : [1] (Iris-versicolor)
[-0.7769106  -0.81982327  0.08043966  0.26325998] : [1] (Iris-versicolor)
[-1.01843715 -2.42582035 -0.14615095 -0.26151073] : [1] (Iris-versicolor)
[ 0.06843254 -0.13153881  0.25038263  0.39445266] : [1] (Iris-versicolor)
[ 0.18919584 -1.96696413  0.13708732 -0.26151073] : [1] (Iris-versicolor)
[ 0.30995914 -0.36096698  0.53362089  0.26325998] : [1] (Iris-versicolor)
[-0.29385737 -0.36096698 -0.08950329  0.13206729] : [1] (Iris-versicolor)
[ 1.03453898  0.09788935  0.36367792  0.26325998] : [1] (Iris-versicolor)
[-0.29385737 -0.13153881  0.42032558  0.39445266] : [1] (Iris-versicolor)
[-0.05233077 -0.81982327  0.19373497 -0.26151073] : [1] (Iris-versicolor)
[ 0.43072245 -1.96696413  0.42032558  0.39445266] : [1] (Iris-versicolor)
[-0.29385737 -1.27867961  0.08043966 -0.13031806] : [1] (Iris-versicolor)
[ 0.06843254  0.32731751  0.59026855  0.78803068] : [1] (Iris-versicolor)
[ 0.30995914 -0.59039515  0.13708732  0.13206729] : [1] (Iris-versicolor)
[ 0.55148572 -1.27867961  0.64691621  0.39445266] : [1] (Iris-versicolor)
[ 0.30995914 -0.59039515  0.53362089  0.00087462] : [1] (Iris-versicolor)
[ 0.67224902 -0.36096698  0.30703026  0.13206729] : [1] (Iris-versicolor)
[ 0.91377568 -0.13153881  0.36367792  0.26325998] : [1] (Iris-versicolor)
[ 1.15530229 -0.59039515  0.59026855  0.26325998] : [1] (Iris-versicolor)
[ 1.03453898 -0.13153881  0.70356381  0.656838  ] : [1] (Iris-versicolor)
[ 0.18919584 -0.36096698  0.42032558  0.39445266] : [1] (Iris-versicolor)
[-0.17309406 -1.04925144 -0.14615095 -0.26151073] : [1] (Iris-versicolor)
[-0.41462067 -1.50810778  0.02379201 -0.13031806] : [1] (Iris-versicolor)
[-0.41462067 -1.50810778 -0.03285564 -0.26151073] : [1] (Iris-versicolor)
[-0.05233077 -0.81982327  0.08043966  0.00087462] : [1] (Iris-versicolor)
[ 0.18919584 -0.81982327  0.76021147  0.52564532] : [2] (Iris-virginica)
[-0.535384   -0.13153881  0.42032558  0.39445266] : [1] (Iris-versicolor)
[ 0.18919584  0.78617382  0.42032558  0.52564532] : [1] (Iris-versicolor)
[ 1.03453898  0.09788935  0.53362089  0.39445266] : [1] (Iris-versicolor)
[ 0.55148572 -1.73753595  0.36367792  0.13206729] : [1] (Iris-versicolor)
[-0.29385737 -0.13153881  0.19373497  0.13206729] : [1] (Iris-versicolor)
[-0.41462067 -1.27867961  0.13708732  0.13206729] : [1] (Iris-versicolor)
[ -4.14620668e-01  -1.04925144e+00   3.63677919e-01   8.74617836e-04] : [1] (Iris-versicolor)
[ 0.30995914 -0.13153881  0.47697324  0.26325998] : [1] (Iris-versicolor)
[ -5.23307659e-02  -1.04925144e+00   1.37087315e-01   8.74617836e-04] : [1] (Iris-versicolor)
[-1.01843715 -1.73753595 -0.25944623 -0.26151073] : [1] (Iris-versicolor)
[-0.29385737 -0.81982327  0.25038263  0.13206729] : [1] (Iris-versicolor)
[-0.17309406 -0.13153881  0.25038263  0.00087462] : [1] (Iris-versicolor)
[-0.17309406 -0.36096698  0.25038263  0.13206729] : [1] (Iris-versicolor)
[ 0.43072245 -0.36096698  0.30703026  0.13206729] : [1] (Iris-versicolor)
[-0.8976739  -1.27867961 -0.42938921 -0.13031806] : [1] (Iris-versicolor)
[-0.17309406 -0.59039515  0.19373497  0.13206729] : [1] (Iris-versicolor)
[ 0.55148572  0.55674565  1.27004039  1.70637941] : [2] (Iris-virginica)
[-0.05233077 -0.81982327  0.76021147  0.91922337] : [2] (Iris-virginica)
[ 1.51759219 -0.13153881  1.21339273  1.18160868] : [2] (Iris-virginica)
[ 0.55148572 -0.36096698  1.04344976  0.78803068] : [2] (Iris-virginica)
[ 0.79301232 -0.13153881  1.15674508  1.31280136] : [2] (Iris-virginica)
[ 2.1214087  -0.13153881  1.60992622  1.18160868] : [2] (Iris-virginica)
[-1.13920045 -1.27867961  0.42032558  0.656838  ] : [2] (Iris-virginica)
[ 1.7591188  -0.36096698  1.43998337  0.78803068] : [2] (Iris-virginica)
[ 1.03453898 -1.27867961  1.15674508  0.78803068] : [2] (Iris-virginica)
[ 1.63835549  1.24503016  1.32668805  1.70637941] : [2] (Iris-virginica)
[ 0.79301232  0.32731751  0.76021147  1.05041599] : [2] (Iris-virginica)
[ 0.67224902 -0.81982327  0.87350678  0.91922337] : [2] (Iris-virginica)
[ 1.15530229 -0.13153881  0.9868021   1.18160868] : [2] (Iris-virginica)
[-0.17309406 -1.27867961  0.70356381  1.05041599] : [2] (Iris-virginica)
[-0.05233077 -0.59039515  0.76021147  1.57518673] : [2] (Iris-virginica)
[ 0.67224902  0.32731751  0.87350678  1.44399405] : [2] (Iris-virginica)
[ 0.79301232 -0.13153881  0.9868021   0.78803068] : [2] (Iris-virginica)
[ 2.242172    1.70388651  1.66657388  1.31280136] : [2] (Iris-virginica)
[ 2.242172   -1.04925144  1.7798692   1.44399405] : [2] (Iris-virginica)
[ 0.18919584 -1.96696413  0.70356381  0.39445266] : [2] (Iris-virginica)
[ 1.27606559  0.32731751  1.10009742  1.44399405] : [2] (Iris-virginica)
[-0.29385737 -0.59039515  0.64691621  1.05041599] : [2] (Iris-virginica)
[ 2.242172   -0.59039515  1.66657388  1.05041599] : [2] (Iris-virginica)
[ 0.55148572 -0.81982327  0.64691621  0.78803068] : [2] (Iris-virginica)
[ 1.03453898  0.55674565  1.10009742  1.18160868] : [2] (Iris-virginica)
[ 1.63835549  0.32731751  1.27004039  0.78803068] : [2] (Iris-virginica)
[ 0.43072245 -0.59039515  0.59026855  0.78803068] : [2] (Iris-virginica)
[ 0.30995914 -0.13153881  0.64691621  0.78803068] : [2] (Iris-virginica)
[ 0.67224902 -0.59039515  1.04344976  1.18160868] : [2] (Iris-virginica)
[ 1.63835549 -0.13153881  1.15674508  0.52564532] : [2] (Iris-virginica)
[ 1.8798821  -0.59039515  1.32668805  0.91922337] : [2] (Iris-virginica)
[ 2.48369861  1.70388651  1.49663091  1.05041599] : [2] (Iris-virginica)
[ 0.67224902 -0.59039515  1.04344976  1.31280136] : [2] (Iris-virginica)
[ 0.55148572 -0.59039515  0.76021147  0.39445266] : [2] (Iris-virginica)
[ 0.30995914 -1.04925144  1.04344976  0.26325998] : [2] (Iris-virginica)
[ 2.242172   -0.13153881  1.32668805  1.44399405] : [2] (Iris-virginica)
[ 0.55148572  0.78617382  1.04344976  1.57518673] : [2] (Iris-virginica)
[ 0.67224902  0.09788935  0.9868021   0.78803068] : [2] (Iris-virginica)
[ 0.18919584 -0.13153881  0.59026855  0.78803068] : [2] (Iris-virginica)
[ 1.27606559  0.09788935  0.93015444  1.18160868] : [2] (Iris-virginica)
[ 1.03453898  0.09788935  1.04344976  1.57518673] : [2] (Iris-virginica)
[ 1.27606559  0.09788935  0.76021147  1.44399405] : [2] (Iris-virginica)
[-0.05233077 -0.81982327  0.76021147  0.91922337] : [2] (Iris-virginica)
[ 1.15530229  0.32731751  1.21339273  1.44399405] : [2] (Iris-virginica)
[ 1.03453898  0.55674565  1.10009742  1.70637941] : [2] (Iris-virginica)
[ 1.03453898 -0.13153881  0.81685913  1.44399405] : [2] (Iris-virginica)
[ 0.55148572 -1.27867961  0.70356381  0.91922337] : [2] (Iris-virginica)
[ 0.79301232 -0.13153881  0.81685913  1.05041599] : [2] (Iris-virginica)
[ 0.43072245  0.78617382  0.93015444  1.44399405] : [2] (Iris-virginica)
[ 0.06843254 -0.13153881  0.76021147  0.78803068] : [2] (Iris-virginica)

In [5]:
# ad hoc prediction
sample_flower = np.ndarray(buffer=np.array([5.0,3.0,4.0,2.0]),shape=(1,4))
pred = classifier.predict(sample_flower)
print("Predict that {} is: {}".format(sample_flower,species[pred]))


Predict that [[ 5.  3.  4.  2.]] is: ['Iris-virginica']

Simple TensorFlow Regression: MPG

This example shows how to encode the MPG dataset for regression. This is slightly more complex than Iris, because:

  • Input has both numeric and categorical
  • Input has missing values

To encode categorical values that are part of the feature vector, use the functions from above. If the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions.


In [6]:
import tensorflow.contrib.learn as skflow
import pandas as pd
import os
import numpy as np
from sklearn import metrics
from scipy.stats import zscore

path = "./data/"

filename_read = os.path.join(path,"auto-mpg.csv")
df = pd.read_csv(filename_read,na_values=['NA','?'])

# create feature vector
missing_median(df, 'horsepower')
df.drop('name',1,inplace=True)
encode_numeric_zscore(df, 'horsepower')
encode_numeric_zscore(df, 'weight')
encode_numeric_zscore(df, 'cylinders')
encode_numeric_zscore(df, 'displacement')
encode_numeric_zscore(df, 'acceleration')
encode_text_dummy(df, 'origin')

# Display training data
df

# Encode to a 2D matrix for training
x,y = to_xy(df,['mpg'])

# Create a deep neural network with 3 hidden layers of 50, 25, 10
regressor = skflow.TensorFlowDNNRegressor(hidden_units=[50, 25, 10], steps=5000)

# Fit/train neural network
regressor.fit(x, y)

# Measure RMSE error.  RMSE is common for regression.
score = np.sqrt(metrics.mean_squared_error(regressor.predict(x),y))
print("Final score (RMSE): {}".format(score))

# How to make many predictions
pred = regressor.predict(x)
predDF = pd.DataFrame(pred)
df2 = pd.concat([df,predDF,pd.DataFrame(y)],axis=1)

df2.columns = list(df.columns)+['pred','ideal']
df2


Step #100, epoch #7, avg. train loss: 23.02969
Step #200, epoch #15, avg. train loss: 2.67576
Step #300, epoch #23, avg. train loss: 1.33839
Step #400, epoch #30, avg. train loss: 0.86830
Step #500, epoch #38, avg. train loss: 0.67166
Step #600, epoch #46, avg. train loss: 0.54569
Step #700, epoch #53, avg. train loss: 0.47544
Step #800, epoch #61, avg. train loss: 0.39358
Step #900, epoch #69, avg. train loss: 0.36052
Step #1000, epoch #76, avg. train loss: 0.31864
Step #1100, epoch #84, avg. train loss: 0.28259
Step #1200, epoch #92, avg. train loss: 0.27536
Step #1300, epoch #100, avg. train loss: 0.25650
Step #1400, epoch #107, avg. train loss: 0.23358
Step #1500, epoch #115, avg. train loss: 0.22679
Step #1600, epoch #123, avg. train loss: 0.21257
Step #1700, epoch #130, avg. train loss: 0.19146
Step #1800, epoch #138, avg. train loss: 0.19052
Step #1900, epoch #146, avg. train loss: 0.17987
Step #2000, epoch #153, avg. train loss: 0.17291
Step #2100, epoch #161, avg. train loss: 0.15884
Step #2200, epoch #169, avg. train loss: 0.15134
Step #2300, epoch #176, avg. train loss: 0.15167
Step #2400, epoch #184, avg. train loss: 0.14684
Step #2500, epoch #192, avg. train loss: 0.14620
Step #2600, epoch #200, avg. train loss: 0.14161
Step #2700, epoch #207, avg. train loss: 0.13537
Step #2800, epoch #215, avg. train loss: 0.13965
Step #2900, epoch #223, avg. train loss: 0.11203
Step #3000, epoch #230, avg. train loss: 0.10849
Step #3100, epoch #238, avg. train loss: 0.11199
Step #3200, epoch #246, avg. train loss: 0.09931
Step #3300, epoch #253, avg. train loss: 0.09825
Step #3400, epoch #261, avg. train loss: 0.10070
Step #3500, epoch #269, avg. train loss: 0.09909
Step #3600, epoch #276, avg. train loss: 0.09264
Step #3700, epoch #284, avg. train loss: 0.09839
Step #3800, epoch #292, avg. train loss: 0.09322
Step #3900, epoch #300, avg. train loss: 0.09552
Step #4000, epoch #307, avg. train loss: 0.09385
Step #4100, epoch #315, avg. train loss: 0.09013
Step #4200, epoch #323, avg. train loss: 0.08828
Step #4300, epoch #330, avg. train loss: 0.08521
Step #4400, epoch #338, avg. train loss: 0.07708
Step #4500, epoch #346, avg. train loss: 0.07587
Step #4600, epoch #353, avg. train loss: 0.07538
Step #4700, epoch #361, avg. train loss: 0.07510
Step #4800, epoch #369, avg. train loss: 0.07322
Step #4900, epoch #376, avg. train loss: 0.07285
Step #5000, epoch #384, avg. train loss: 0.06910
Final score (RMSE): 0.3777039872803281
Out[6]:
mpg cylinders displacement horsepower weight acceleration year origin-1 origin-2 origin-3 pred ideal
0 18.0 1.496308 1.089233 0.672271 0.630077 -1.293870 70 1.0 0.0 0.0 17.551966 18
1 15.0 1.496308 1.501624 1.587959 0.853259 -1.475181 70 1.0 0.0 0.0 14.673651 15
2 18.0 1.496308 1.194728 1.195522 0.549778 -1.656492 70 1.0 0.0 0.0 17.551807 18
3 16.0 1.496308 1.060461 1.195522 0.546236 -1.293870 70 1.0 0.0 0.0 15.635429 16
4 17.0 1.496308 1.041280 0.933897 0.565130 -1.837804 70 1.0 0.0 0.0 16.590120 17
5 15.0 1.496308 2.259274 2.451322 1.618455 -2.019115 70 1.0 0.0 0.0 14.645038 15
6 14.0 1.496308 2.499036 3.026898 1.633806 -2.381737 70 1.0 0.0 0.0 13.721073 14
7 14.0 1.496308 2.364769 2.896085 1.584210 -2.563048 70 1.0 0.0 0.0 13.713527 14
8 14.0 1.496308 2.508627 3.157710 1.717647 -2.019115 70 1.0 0.0 0.0 13.717277 14
9 15.0 1.496308 1.885244 2.242022 1.038654 -2.563048 70 1.0 0.0 0.0 14.663959 15
10 15.0 1.496308 1.818111 1.718772 0.699747 -2.019115 70 1.0 0.0 0.0 14.706730 15
11 14.0 1.496308 1.405719 1.457147 0.754067 -2.744360 70 1.0 0.0 0.0 13.735887 14
12 15.0 1.496308 1.981149 1.195522 0.933557 -2.200426 70 1.0 0.0 0.0 14.708412 15
13 14.0 1.496308 2.508627 3.157710 0.136478 -2.019115 70 1.0 0.0 0.0 13.869637 14
14 24.0 -0.855244 -0.771324 -0.243417 -0.706655 -0.206002 70 0.0 0.0 1.0 23.616606 24
15 22.0 0.320532 0.043868 -0.243417 -0.162279 -0.024691 70 1.0 0.0 0.0 21.540794 22
16 18.0 0.320532 0.053459 -0.191092 -0.231950 -0.024691 70 1.0 0.0 0.0 17.652323 18
17 21.0 0.320532 0.063049 -0.505042 -0.452770 0.156620 70 1.0 0.0 0.0 20.608076 21
18 27.0 -0.855244 -0.924773 -0.426554 -0.992422 -0.387314 70 0.0 0.0 1.0 26.583515 27
19 26.0 -0.855244 -0.924773 -1.525380 -1.340775 1.788421 70 0.0 1.0 0.0 25.729187 26
20 25.0 -0.855244 -0.800096 -0.452717 -0.352397 0.700554 70 0.0 1.0 0.0 24.601633 25
21 24.0 -0.855244 -0.828867 -0.374229 -0.638165 -0.387314 70 0.0 1.0 0.0 23.639893 24
22 25.0 -0.855244 -0.857639 -0.243417 -0.703112 0.700554 70 0.0 1.0 0.0 24.630373 25
23 26.0 -0.855244 -0.694600 0.227509 -0.869613 -1.112559 70 0.0 1.0 0.0 25.609526 26
24 21.0 0.320532 0.053459 -0.374229 -0.380738 -0.206002 70 1.0 0.0 0.0 20.588083 21
25 10.0 1.496308 1.597529 2.896085 1.942010 -0.568625 70 1.0 0.0 0.0 9.921564 10
26 10.0 1.496308 1.089233 2.503648 1.659785 -0.206002 70 1.0 0.0 0.0 9.915915 10
27 11.0 1.496308 1.194728 2.765273 1.666870 -0.749936 70 1.0 0.0 0.0 10.842975 11
28 9.0 1.496308 1.060461 2.320510 2.080171 1.063176 70 1.0 0.0 0.0 8.964724 9
29 27.0 -0.855244 -0.924773 -0.426554 -0.992422 -0.387314 71 0.0 0.0 1.0 26.582289 27
... ... ... ... ... ... ... ... ... ... ... ... ...
368 27.0 -0.855244 -0.780915 -0.426554 -0.390185 1.099439 82 1.0 0.0 0.0 26.421032 27
369 34.0 -0.855244 -0.780915 -0.426554 -0.679495 0.881865 82 1.0 0.0 0.0 33.258595 34
370 31.0 -0.855244 -0.780915 -0.505042 -0.466940 0.229145 82 1.0 0.0 0.0 30.332884 31
371 29.0 -0.855244 -0.560333 -0.531204 -0.525983 0.156620 82 1.0 0.0 0.0 28.421907 29
372 27.0 -0.855244 -0.406885 -0.374229 -0.278003 0.881865 82 1.0 0.0 0.0 26.417910 27
373 24.0 -0.855244 -0.512381 -0.321904 -0.124492 0.301669 82 1.0 0.0 0.0 23.435291 24
374 23.0 -0.855244 -0.406885 -0.282660 0.076254 1.788421 82 1.0 0.0 0.0 22.469835 23
375 36.0 -0.855244 -0.848048 -0.792829 -1.169551 -0.097216 82 0.0 1.0 0.0 35.345470 36
376 37.0 -0.855244 -0.982316 -0.949804 -1.116412 0.954390 82 0.0 0.0 1.0 36.408638 37
377 31.0 -0.855244 -0.982316 -0.949804 -1.181360 0.736816 82 0.0 0.0 1.0 30.464426 31
378 38.0 -0.855244 -0.848048 -1.080617 -0.998327 -0.314789 82 1.0 0.0 0.0 37.370407 38
379 36.0 -0.855244 -0.915182 -0.897479 -0.998327 0.628029 82 1.0 0.0 0.0 35.329590 36
380 36.0 -0.855244 -0.704191 -0.426554 -0.956997 -0.387314 82 0.0 0.0 1.0 35.347202 36
381 36.0 -0.855244 -0.828867 -0.766667 -0.903858 -0.387314 82 0.0 0.0 1.0 35.333443 36
382 34.0 -0.855244 -0.819277 -0.897479 -0.856624 0.482980 82 0.0 0.0 1.0 33.329166 34
383 38.0 -0.855244 -0.982316 -0.975967 -1.187264 -0.206002 82 0.0 0.0 1.0 37.382545 38
384 32.0 -0.855244 -0.982316 -0.975967 -1.187264 0.047833 82 0.0 0.0 1.0 31.439056 32
385 38.0 -0.855244 -0.982316 -0.975967 -1.151838 0.229145 82 0.0 0.0 1.0 37.381981 38
386 25.0 0.320532 -0.119170 0.149021 -0.030023 0.301669 82 1.0 0.0 0.0 24.336592 25
387 38.0 0.320532 0.657660 -0.505042 0.052637 0.519243 82 1.0 0.0 0.0 37.337452 38
388 26.0 -0.855244 -0.358933 -0.321904 -0.455132 -0.387314 82 1.0 0.0 0.0 25.446016 26
389 22.0 0.320532 0.369945 0.201346 -0.159917 -0.314789 82 1.0 0.0 0.0 21.478664 22
390 32.0 -0.855244 -0.474019 -0.217254 -0.360663 -0.604887 82 0.0 0.0 1.0 31.332535 32
391 36.0 -0.855244 -0.560333 -0.531204 -0.709016 -0.931247 82 1.0 0.0 0.0 35.276676 36
392 27.0 -0.855244 -0.406885 -0.374229 -0.024119 0.628029 82 1.0 0.0 0.0 26.384054 27
393 27.0 -0.855244 -0.512381 -0.478879 -0.213056 0.011571 82 1.0 0.0 0.0 26.408836 27
394 44.0 -0.855244 -0.924773 -1.368405 -0.992422 3.275173 82 0.0 1.0 0.0 43.302361 44
395 32.0 -0.855244 -0.560333 -0.531204 -0.797581 -1.438919 82 1.0 0.0 0.0 31.366554 32
396 28.0 -0.855244 -0.704191 -0.662017 -0.407897 1.099439 82 1.0 0.0 0.0 27.431047 28
397 31.0 -0.855244 -0.713781 -0.583529 -0.295716 1.389537 82 1.0 0.0 0.0 30.308321 31

398 rows × 12 columns

Load/Save Trained Network

The code below saves a TensorFlow network to a directory. This directory can be used to reload your weights, without retraining. Because training can take a long time, this is critical.

The following code trains an Iris dataset, reports the error, saves/reloads it, and reports the same error.


In [7]:
import tensorflow.contrib.learn as skflow
from sklearn import metrics
import pandas as pd
import os
from sklearn import preprocessing

path = "./data/"

# Read iris dataset
filename_read = os.path.join(path,"iris.csv")
df = pd.read_csv(filename_read,na_values=['NA','?'])

# Encode
encode_text_index(df,'species')

# Create the x-side (feature vectors) of the training
x, y = to_xy(df,'species')

# Create a deep neural network with 3 hidden layers of 10, 20, 10
classifier = skflow.TensorFlowDNNClassifier(hidden_units=[10, 20, 10], n_classes=3,steps=200)

# Fit/train neural network
classifier.fit(x, y)

# Measure accuracy
score = metrics.accuracy_score(y, classifier.predict(x))
print("Final score: {}".format(score))

# Save the neural network to a directory
classifier.save("./iris-network")
classifier = None # Kill it

# Reload it
classifier2 = skflow.TensorFlowEstimator.restore("./iris-network")

# Prove that the reloaded is the same as the original
score = metrics.accuracy_score(y, classifier2.predict(x))
print("Saved final score: {}".format(score))


Step #100, epoch #20, avg. train loss: 0.55053
Step #200, epoch #40, avg. train loss: 0.15876
Final score: 0.9466666666666667
Saved final score: 0.9466666666666667

View Graph for Neural Network

TensorFlow includes the command line utility called tensorboard that can be used to visualize the neural networks. It is not needed for this course, but it can be handy to see your neural network, and I will use it in lecture a few times. It does not work with IBM Data Scientist Workbench, so you will need a native install if you would like to use it.

To make use of it, you must specify a logdir on the fit command, for example:

classifier.fit(x, y, logdir='./log/')

Once the fit occurs, the logdir will be filled with files that tensorboard will use. To view the graph, issue the following command:

tensorboard --logdir ./log

In [ ]: