Here the neural net code is created and implemented. For the following notebooks, the neural network code will be saved to disk and imported. All weights are also saved to disk and upload from a previous training run. The code to train the weights has been commented out but remains in the notebook to display how the weights are called.
In [82]:
import numpy as np
from sklearn.datasets import load_digits
digits = load_digits()
from IPython.html.widgets import interact
%matplotlib inline
import matplotlib.pyplot as plt
import NNpix as npx
In [107]:
def interact_fun(i):
plt.matshow(digits.images[i])
plt.show()
print("True Number: %d" % digits.target[i])
In [108]:
interact(interact_fun, i=(0,1796));
In order to change the weights, we'll need the sigmoid and sigmoid prime functions.
$\sigma(x) = \frac {1}{1+e^{-x}} \\ \sigma\prime(x) = \sigma(x) \times (1-\sigma(x)) $
In [85]:
""" The activation function. """
def sigmoid(x):
return 1/(1+np.exp(-x))
In [86]:
assert sigmoid(np.log(2)) == 2/3
In [87]:
""" The derivative of the activation function """
def sigmoid_prime(x):
return sigmoid(x)*(1-sigmoid(x))
Neural networks require a 1D input per output. Our data initially is 2D.
In [88]:
print(digits.images[0])
Here, the input is now 1D. Because it has 64 elements per array, we'll need 64 input neurons.
In [89]:
print(digits.images[0].flatten())
In [90]:
perm = np.random.permutation(1797)
In [91]:
""" Divide by 100 to get inputs into decimals """
training_input = np.array([digits.images[perm[i]].flatten() for i in range(1000)])/100
test_input = np.array([digits.images[perm[i]].flatten() for i in range(1000,1797)])/100
In [92]:
assert len(training_input[0]) == 64
assert len(test_input[0]) == 64
In [93]:
def create_soln(training_numbers):
""" Creates 2D array for training solutions """
a = np.repeat(0,10,None)
a = np.repeat([a], len(training_numbers), 0)
for i in range(len(training_numbers)):
a[i][training_numbers[i]] = 1
return a
In [94]:
""" True number solutions used to calculate accuracy"""
number_solution = np.array([digits.target[perm[i]] for i in range(1000,1797)])
In [95]:
""" Creates the array of solutions to be entered into the neural network"""
training_solution = create_soln([digits.target[perm[i]] for i in range(1000)])
In [96]:
assert len(training_solution[0]) == 10
This is the class used to train the neural network. The "train" function iterates through the input arrays and solution arrays and returns the weights.
In [97]:
class NN_training(object):
def __init__(self, input_array, soln, hidnum, iters, lr):
self.input_array = input_array
self.soln = soln
#Number of hidden nodes
self.hidnum = hidnum
#Number of iterations through the training set
self.iters = iters
#Initalize WIJ weights (input to hidden)
self.wij = np.random.uniform(-.5,0.5,(hidnum,65))
#Initalize WJK weights (hidden to output)
self.wjk = np.random.uniform(-0.5,0.5,(10,hidnum+1))
#Set a learning rate
self.lr = lr
def train(self):
iters = self.iters
for n in range(iters):
for i in range(len(self.input_array)):
soln = self.soln[i]
hidnum = self.hidnum
input_array = np.append(self.input_array[i],[1])
#Find sum of weights x input array values for each hidden
self.hidden_sums = (sum((input_array * self.wij).T)).T
#Find outputs of hidden neurons; include bias
self.hidden_out = np.append(sigmoid(self.hidden_sums),[1])
#Find sums of weights x hidden outs for each neuron in output layer
self.output_sums = (sum((self.hidden_out * self.wjk).T)).T
#Find output of the outputs
self.output_out = sigmoid(self.output_sums)
self.E = self.output_out - soln
#Find delta values for each output
self.output_deltas = self.E * sigmoid_prime(self.output_sums)
#Find delta values for each hidden
self.hidden_deltas = sigmoid_prime(np.delete(self.hidden_out,[hidnum],None)) * sum((self.output_deltas * (np.delete(self.wjk, [hidnum], 1)).T).T)
#Change weights
self.wij = -self.lr * (self.hidden_deltas*(np.repeat([input_array],hidnum,0)).T).T + self.wij
self.wjk = -self.lr * (self.output_deltas*(np.repeat([self.hidden_out],10,0)).T).T + self.wjk
return (self.wij, self.wjk)
Our input array is "training_input" our solution array is "training_solution". I chose 40 hidden nodes and 90 iterations with a learning rate of 0.7.
In [98]:
my_net = NN_training(training_input, training_solution, 40, 90, 0.7)
To generate the weights, use the commented out line below. Because it is a slow process, I will be loading weights from a previous training.
In [109]:
# x, y = my_net.train()
In [110]:
f = np.load("NNweights.npz")
In [111]:
list(f)
Out[111]:
In [112]:
x = f['arr_0']
y = f['arr_1']
In [113]:
assert len(x) == 40
assert len(y) == 10
Whent the "get_ans" function is called, the weights and input arrays will be used to calculate the solution.
In [114]:
class NN_ask (object):
""" Feed forward using final weights from training backpropagation """
def __init__(self, input_array, wij, wjk):
self.input_array = input_array
self.wij = wij
self.wjk = wjk
def get_ans(self):
wij = self.wij
wjk = self.wjk
soln = []
for i in range(len(self.input_array)):
input_array = np.append(self.input_array[i],[1])
self.hidden_sums = (sum((input_array * wij).T)).T
self.hidden_out = np.append(sigmoid(self.hidden_sums),[1])
self.output_sums = (sum((self.hidden_out * wjk).T)).T
self.output_out = sigmoid(self.output_sums)
for i in range(len(self.output_out)):
if self.output_out[i] == max(self.output_out):
a = i
soln.append(a)
return soln
In [115]:
test_net = NN_ask(test_input, x, y)
In [116]:
comp_vals = test_net.get_ans()
In [117]:
print(((sum((comp_vals - number_solution == 0).astype(int)) / (1797-1000)) * 100), "%")
In [118]:
def interacting(i):
plt.matshow(digits.images[perm[i+1000]])
plt.show()
print("Neural Network's Value: %d" %comp_vals[i])
print("True Known Solution %d" %number_solution[i])
In [119]:
interact(interacting, i=(0,796));
In [ ]: