Tony Reina
28 JUNE 2017
Here's an example of how we can load one of the standard sklearn datasets into a neon model. We'll be using the iris dataset, a classification model which tries to predict the type of iris flower species (Setosa, Versicolour, and Virginica) based on 4 continuous parameters: Sepal Length, Sepal Width, Petal Length and Petal Width. It is based on Ronald Fisher's 1936 paper describing Linear Discriminant Analysis. The dataset is now considered one of the gold standards at monitoring the performance of a new classification method.
In this notebook, we'll walk through loading the data from sklearn into neon's ArrayIterator class and then passing that to a simple multi-layer perceptron model. We should get a misclassification rate of 2% to 8%.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
In [1]:
from sklearn import datasets
In [2]:
iris = datasets.load_iris()
X = iris.data
Y = iris.target
nClasses = len(iris.target_names) # Setosa, Versicolour, and Virginica iris species
In [3]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33) # 66% training, 33% testing
In [4]:
from sklearn.preprocessing import StandardScaler
scl = StandardScaler()
X_train = scl.fit_transform(X_train)
X_test = scl.transform(X_test)
This sets up either our GPU or CPU connection to neon. If we don't start with this, then ArrayIterator won't execute.
We're asking neon to use the cpu, but can change that to a gpu if it is available. Batch size refers to how many data points are taken at a time. Here's a primer on Gradient Descent.
Technical note: Your batch size must always be much less than the number of points in your data. So if you have 50 points, then set your batch size to something much less than 50. I'd suggest setting the batch size to no more than 10% of the number of data points. You can always just set your batch size to 1. In that case, you are no longer performing mini-batch gradient descent, but are performing the standard stochastic gradient descent.
In [5]:
from neon.data import ArrayIterator
from neon.backends import gen_backend
be = gen_backend(backend='cpu', batch_size=X_train.shape[0]//10) # Change to 'gpu' if you have gpu support
In [6]:
training_data = ArrayIterator(X=X_train, y=y_train, nclass=nClasses, make_onehot=True)
testing_data = ArrayIterator(X=X_test, y=y_test, nclass=nClasses, make_onehot=True)
In [7]:
print ('I am using this backend: {}'.format(be))
In [8]:
from neon.initializers import GlorotUniform, Gaussian
from neon.layers import GeneralizedCost, Affine, Dropout
from neon.models import Model
from neon.optimizers import GradientDescentMomentum, Adam
from neon.transforms import Softmax, CrossEntropyMulti, Rectlin, Tanh
from neon.callbacks.callbacks import Callbacks, EarlyStopCallback
from neon.transforms import Misclassification
We could use numbers from the Gaussian distribution ($\mu=0, \sigma=0.3$) to initialize the weights and bias terms for our regression model. However, we can also use other initializations like GlorotUniform.
In [9]:
init = GlorotUniform() #Gaussian(loc=0, scale=0.3)
We just use a simple Python list to add our different layers to the model. The nice thing is that we've already put our data into a neon ArrayIterator. That means the model will automatically know how to handle the input layer.
In this model, the input layer feeds into a 5-neuron rectified linear unit affine layer. That feeds into an 8 neuron hyperbolic tangent layer (with 50% dropout). Finally, that outputs to a softmax of the nClasses. We'll predict based on the argmax of the softmax layer.
I've just thrown together a model haphazardly. There is no reason the model has to be like this. In fact, I would suggest playing with adding different layers, different # of neurons, and different activation functions to see if you can get a better model. What's nice about neon is that we can easily alter the model architecture without much change to our code.
In [10]:
layers = [
Affine(nout=5, init=init, bias=init, activation=Rectlin()), # Affine layer with 5 neurons (ReLU activation)
Affine(nout=8, init=init, bias=init, activation=Tanh()), # Affine layer with 8 neurons (Tanh activation)
Dropout(0.5), # Dropout layer
Affine(nout=nClasses, init=init, bias=init, activation=Softmax()) # Affine layer with softmax
]
In [11]:
mlp = Model(layers=layers)
How "close" is the model's prediction is to the true value? For the case of multi-class prediction we typically use Cross Entropy.
In [12]:
cost = GeneralizedCost(costfunc=CrossEntropyMulti())
All of our models will use gradient descent. We will iteratively update the model weights and biases in order to minimize the cost of the model.
There are many optimizing algorithms we can use for gradient descent. Here we'll use Adam.
In [13]:
#optimizer = GradientDescentMomentum(0.1, momentum_coef=0.2)
optimizer = Adam(learning_rate=0.1, beta_1=0.9, beta_2=0.999)
Callbacks allow us to run custom code at certain points during the training. For example, in the code below we want to find out how well the model is performing against the testing data after every 2 callbacks of training. If the cross entropy error goes up, then we stop the training early. Otherwise, we might be overfitting the model to the training set.
I've added a patience parameter to the early stopping. If the model's performance has not improved after a certain number of callbacks, then we will stop training early.
In [14]:
# define stopping function
# it takes as input a tuple (State,val[t])
# which describes the cumulative validation state (generated by this function)
# and the validation error at time t
# and returns as output a tuple (State', Bool),
# which represents the new state and whether to stop
def stop_func(s, v):
patience = 4 # If model performance has not improved in this many callbacks, then early stop.
if s is None:
return ([v], False)
if (all(v < i for i in s)): # Check to see if this value is smaller than any in the history
history = [v] # New value is smaller so let's reset the history
print('Model improved performance: {}'.format(v))
else:
history = s + [v] # New value is not smaller, so let's add to current history
print('Model has not improved in {} callbacks.'.format(len(history)-1))
if len(history) > patience: # If our history is greater than the patience, then early terminate.
stop = True
print('Stopping training early.')
else:
stop = False # Otherwise, keep training.
return (history, stop)
# The model trains on the training set, but every 2 epochs we calculate
# its performance against the testing set. If the performance increases, then
# we want to stop early because we are overfitting our model.
callbacks = Callbacks(mlp, eval_set=testing_data, eval_freq=2) # Run the callback every 2 epochs
callbacks.add_callback(EarlyStopCallback(stop_func)) # Add our early stopping function call
In [15]:
mlp.fit(training_data, optimizer=optimizer, num_epochs=100, cost=cost, callbacks=callbacks)
In [16]:
results = mlp.get_outputs(testing_data)
prediction = results.argmax(1)
error_pct = 100 * mlp.eval(testing_data, metric=Misclassification())[0]
print ('The model misclassified {:.2f}% of the test data.'.format(error_pct))
In [17]:
mlp.save_params('iris_model.prm')
In [18]:
mlp.get_description()['model']
Out[18]:
In [ ]: