## Base Code

Here the neural net code is created and implemented. For the following notebooks, the neural network code will be saved to disk and imported. All weights are also saved to disk and upload from a previous training run. The code to train the weights has been commented out but remains in the notebook to display how the weights are called.



In [82]:

import numpy as np
from IPython.html.widgets import interact
%matplotlib inline
import matplotlib.pyplot as plt
import NNpix as npx



### Visualizing our Data:



In [107]:

def interact_fun(i):
plt.matshow(digits.images[i])
plt.show()
print("True Number: %d" % digits.target[i])




In [108]:

interact(interact_fun, i=(0,1796));




True Number: 7



### Useful Functions:

In order to change the weights, we'll need the sigmoid and sigmoid prime functions.

$\sigma(x) = \frac {1}{1+e^{-x}} \\ \sigma\prime(x) = \sigma(x) \times (1-\sigma(x))$



In [85]:

""" The activation function. """
def sigmoid(x):
return 1/(1+np.exp(-x))




In [86]:

assert sigmoid(np.log(2)) == 2/3




In [87]:

""" The derivative of the activation function """
def sigmoid_prime(x):
return sigmoid(x)*(1-sigmoid(x))



### Neural Networks 1D Input

Neural networks require a 1D input per output. Our data initially is 2D.



In [88]:

print(digits.images[0])




[[  0.   0.   5.  13.   9.   1.   0.   0.]
[  0.   0.  13.  15.  10.  15.   5.   0.]
[  0.   3.  15.   2.   0.  11.   8.   0.]
[  0.   4.  12.   0.   0.   8.   8.   0.]
[  0.   5.   8.   0.   0.   9.   8.   0.]
[  0.   4.  11.   0.   1.  12.   7.   0.]
[  0.   2.  14.   5.  10.  12.   0.   0.]
[  0.   0.   6.  13.  10.   0.   0.   0.]]



Here, the input is now 1D. Because it has 64 elements per array, we'll need 64 input neurons.



In [89]:

print(digits.images[0].flatten())




[  0.   0.   5.  13.   9.   1.   0.   0.   0.   0.  13.  15.  10.  15.   5.
0.   0.   3.  15.   2.   0.  11.   8.   0.   0.   4.  12.   0.   0.   8.
8.   0.   0.   5.   8.   0.   0.   9.   8.   0.   0.   4.  11.   0.   1.
12.   7.   0.   0.   2.  14.   5.  10.  12.   0.   0.   0.   0.   6.  13.
10.   0.   0.   0.]



### Create training and testing inputs

• The training input for maximum accuracy is 1000 random samples.
• The testing input is the last 797 random samples.
• All inputs are converted to decimals


In [90]:

perm = np.random.permutation(1797)




In [91]:

""" Divide by 100 to get inputs into decimals """
training_input = np.array([digits.images[perm[i]].flatten() for i in range(1000)])/100
test_input = np.array([digits.images[perm[i]].flatten() for i in range(1000,1797)])/100




In [92]:

assert len(training_input[0]) == 64
assert len(test_input[0]) == 64



### Create training solution

• Sigmoid functions can only produce values 0-1
• We want output values 0-9
• This means we need an output layer of 10 neurons
• Whichever number neuron is "firing" (closest to 1) is the number answer
##### Example:
• Output Value: 4
• Neural Network Output: [0,0,0,0,1,0,0,0,0,0]


In [93]:

def create_soln(training_numbers):
""" Creates 2D array for training solutions """
a = np.repeat(0,10,None)
a = np.repeat([a], len(training_numbers), 0)
for i in range(len(training_numbers)):
a[i][training_numbers[i]] = 1
return a




In [94]:

""" True number solutions used to calculate accuracy"""
number_solution = np.array([digits.target[perm[i]] for i in range(1000,1797)])




In [95]:

""" Creates the array of solutions to be entered into the neural network"""
training_solution = create_soln([digits.target[perm[i]] for i in range(1000)])




In [96]:

assert len(training_solution[0]) == 10



### Creating the Training Class

This is the class used to train the neural network. The "train" function iterates through the input arrays and solution arrays and returns the weights.



In [97]:

class NN_training(object):

def __init__(self, input_array, soln, hidnum, iters, lr):
self.input_array = input_array
self.soln = soln
#Number of hidden nodes
self.hidnum = hidnum
#Number of iterations through the training set
self.iters = iters
#Initalize WIJ weights (input to hidden)
self.wij = np.random.uniform(-.5,0.5,(hidnum,65))
#Initalize WJK weights (hidden to output)
self.wjk = np.random.uniform(-0.5,0.5,(10,hidnum+1))
#Set a learning rate
self.lr = lr

def train(self):
iters = self.iters
for n in range(iters):
for i in range(len(self.input_array)):
soln = self.soln[i]
hidnum = self.hidnum
input_array = np.append(self.input_array[i],[1])
#Find sum of weights x input array values for each hidden
self.hidden_sums = (sum((input_array * self.wij).T)).T
#Find outputs of hidden neurons; include bias
self.hidden_out = np.append(sigmoid(self.hidden_sums),[1])
#Find sums of weights x hidden outs for each neuron in output layer
self.output_sums = (sum((self.hidden_out * self.wjk).T)).T
#Find output of the outputs
self.output_out = sigmoid(self.output_sums)
self.E = self.output_out - soln
#Find delta values for each output
self.output_deltas = self.E * sigmoid_prime(self.output_sums)
#Find delta values for each hidden
self.hidden_deltas = sigmoid_prime(np.delete(self.hidden_out,[hidnum],None)) * sum((self.output_deltas * (np.delete(self.wjk, [hidnum], 1)).T).T)
#Change weights
self.wij = -self.lr * (self.hidden_deltas*(np.repeat([input_array],hidnum,0)).T).T + self.wij
self.wjk = -self.lr * (self.output_deltas*(np.repeat([self.hidden_out],10,0)).T).T + self.wjk
return (self.wij, self.wjk)



Our input array is "training_input" our solution array is "training_solution". I chose 40 hidden nodes and 90 iterations with a learning rate of 0.7.



In [98]:

my_net = NN_training(training_input, training_solution, 40, 90, 0.7)



To generate the weights, use the commented out line below. Because it is a slow process, I will be loading weights from a previous training.



In [109]:

# x, y = my_net.train()




In [110]:




In [111]:

list(f)




Out[111]:

['arr_1', 'arr_0']




In [112]:

x = f['arr_0']
y = f['arr_1']




In [113]:

assert len(x) == 40
assert len(y) == 10



### Testing Class

Whent the "get_ans" function is called, the weights and input arrays will be used to calculate the solution.



In [114]:

""" Feed forward using final weights from training backpropagation """
def __init__(self, input_array, wij, wjk):
self.input_array = input_array
self.wij = wij
self.wjk = wjk
def get_ans(self):
wij = self.wij
wjk = self.wjk
soln = []
for i in range(len(self.input_array)):
input_array = np.append(self.input_array[i],[1])
self.hidden_sums = (sum((input_array * wij).T)).T
self.hidden_out = np.append(sigmoid(self.hidden_sums),[1])
self.output_sums = (sum((self.hidden_out * wjk).T)).T
self.output_out = sigmoid(self.output_sums)
for i in range(len(self.output_out)):
if self.output_out[i] == max(self.output_out):
a = i
soln.append(a)
return soln




In [115]:




In [116]:

comp_vals = test_net.get_ans()



### Calculate and Visualize Accuray



In [117]:

print(((sum((comp_vals - number_solution == 0).astype(int)) / (1797-1000)) * 100), "%")




97.7415307403 %




In [118]:

def interacting(i):
plt.matshow(digits.images[perm[i+1000]])
plt.show()
print("Neural Network's Value: %d" %comp_vals[i])
print("True Known Solution %d" %number_solution[i])




In [119]:

interact(interacting, i=(0,796));




Neural Network's Value: 3
True Known Solution 3




In [ ]: