Base Code

Here the neural net code is created and implemented. For the following notebooks, the neural network code will be saved to disk and imported. All weights are also saved to disk and upload from a previous training run. The code to train the weights has been commented out but remains in the notebook to display how the weights are called.


In [82]:
import numpy as np
from sklearn.datasets import load_digits 
digits = load_digits()
from IPython.html.widgets import interact
%matplotlib inline
import matplotlib.pyplot as plt
import NNpix as npx

Visualizing our Data:


In [107]:
def interact_fun(i):
    plt.matshow(digits.images[i])
    plt.show()
    print("True Number: %d" % digits.target[i])

In [108]:
interact(interact_fun, i=(0,1796));


True Number: 7

Useful Functions:

In order to change the weights, we'll need the sigmoid and sigmoid prime functions.

$\sigma(x) = \frac {1}{1+e^{-x}} \\ \sigma\prime(x) = \sigma(x) \times (1-\sigma(x)) $


In [85]:
""" The activation function. """
def sigmoid(x):
    return 1/(1+np.exp(-x))

In [86]:
assert sigmoid(np.log(2)) == 2/3

In [87]:
""" The derivative of the activation function """
def sigmoid_prime(x):
    return sigmoid(x)*(1-sigmoid(x))

Neural Networks 1D Input

Neural networks require a 1D input per output. Our data initially is 2D.


In [88]:
print(digits.images[0])


[[  0.   0.   5.  13.   9.   1.   0.   0.]
 [  0.   0.  13.  15.  10.  15.   5.   0.]
 [  0.   3.  15.   2.   0.  11.   8.   0.]
 [  0.   4.  12.   0.   0.   8.   8.   0.]
 [  0.   5.   8.   0.   0.   9.   8.   0.]
 [  0.   4.  11.   0.   1.  12.   7.   0.]
 [  0.   2.  14.   5.  10.  12.   0.   0.]
 [  0.   0.   6.  13.  10.   0.   0.   0.]]

Here, the input is now 1D. Because it has 64 elements per array, we'll need 64 input neurons.


In [89]:
print(digits.images[0].flatten())


[  0.   0.   5.  13.   9.   1.   0.   0.   0.   0.  13.  15.  10.  15.   5.
   0.   0.   3.  15.   2.   0.  11.   8.   0.   0.   4.  12.   0.   0.   8.
   8.   0.   0.   5.   8.   0.   0.   9.   8.   0.   0.   4.  11.   0.   1.
  12.   7.   0.   0.   2.  14.   5.  10.  12.   0.   0.   0.   0.   6.  13.
  10.   0.   0.   0.]

Create training and testing inputs

  • The training input for maximum accuracy is 1000 random samples.
  • The testing input is the last 797 random samples.
  • All inputs are converted to decimals

In [90]:
perm = np.random.permutation(1797)

In [91]:
""" Divide by 100 to get inputs into decimals """
training_input = np.array([digits.images[perm[i]].flatten() for i in range(1000)])/100
test_input = np.array([digits.images[perm[i]].flatten() for i in range(1000,1797)])/100

In [92]:
assert len(training_input[0]) == 64
assert len(test_input[0]) == 64

Create training solution

  • Sigmoid functions can only produce values 0-1
  • We want output values 0-9
  • This means we need an output layer of 10 neurons
  • Whichever number neuron is "firing" (closest to 1) is the number answer
Example:
  • Output Value: 4
  • Neural Network Output: [0,0,0,0,1,0,0,0,0,0]

In [93]:
def create_soln(training_numbers):
    """ Creates 2D array for training solutions """
    a = np.repeat(0,10,None)
    a = np.repeat([a], len(training_numbers), 0)
    for i in range(len(training_numbers)):
        a[i][training_numbers[i]] = 1
    return a

In [94]:
""" True number solutions used to calculate accuracy"""
number_solution = np.array([digits.target[perm[i]] for i in range(1000,1797)])

In [95]:
""" Creates the array of solutions to be entered into the neural network"""
training_solution = create_soln([digits.target[perm[i]] for i in range(1000)])

In [96]:
assert len(training_solution[0]) == 10

Creating the Training Class

This is the class used to train the neural network. The "train" function iterates through the input arrays and solution arrays and returns the weights.


In [97]:
class NN_training(object):
    
    def __init__(self, input_array, soln, hidnum, iters, lr):
        self.input_array = input_array
        self.soln = soln
        #Number of hidden nodes
        self.hidnum = hidnum
        #Number of iterations through the training set
        self.iters = iters
        #Initalize WIJ weights (input to hidden)
        self.wij = np.random.uniform(-.5,0.5,(hidnum,65))
        #Initalize WJK weights (hidden to output)
        self.wjk = np.random.uniform(-0.5,0.5,(10,hidnum+1))
        #Set a learning rate
        self.lr = lr


    def train(self):
        iters = self.iters   
        for n in range(iters): 
            for i in range(len(self.input_array)):
                soln = self.soln[i]
                hidnum = self.hidnum
                input_array = np.append(self.input_array[i],[1])
                #Find sum of weights x input array values for each hidden
                self.hidden_sums = (sum((input_array * self.wij).T)).T
                #Find outputs of hidden neurons; include bias
                self.hidden_out = np.append(sigmoid(self.hidden_sums),[1])
                #Find sums of weights x hidden outs for each neuron in output layer
                self.output_sums = (sum((self.hidden_out * self.wjk).T)).T
                #Find output of the outputs
                self.output_out = sigmoid(self.output_sums)
                self.E = self.output_out - soln
                 #Find delta values for each output
                self.output_deltas = self.E * sigmoid_prime(self.output_sums)
                 #Find delta values for each hidden
                self.hidden_deltas = sigmoid_prime(np.delete(self.hidden_out,[hidnum],None)) * sum((self.output_deltas * (np.delete(self.wjk, [hidnum], 1)).T).T)
                #Change weights
                self.wij = -self.lr * (self.hidden_deltas*(np.repeat([input_array],hidnum,0)).T).T + self.wij
                self.wjk = -self.lr * (self.output_deltas*(np.repeat([self.hidden_out],10,0)).T).T + self.wjk
        return (self.wij, self.wjk)

Our input array is "training_input" our solution array is "training_solution". I chose 40 hidden nodes and 90 iterations with a learning rate of 0.7.


In [98]:
my_net = NN_training(training_input, training_solution, 40, 90, 0.7)

To generate the weights, use the commented out line below. Because it is a slow process, I will be loading weights from a previous training.


In [109]:
# x, y = my_net.train()

In [110]:
f = np.load("NNweights.npz")

In [111]:
list(f)


Out[111]:
['arr_1', 'arr_0']

In [112]:
x = f['arr_0']
y = f['arr_1']

In [113]:
assert len(x) == 40
assert len(y) == 10

Testing Class

Whent the "get_ans" function is called, the weights and input arrays will be used to calculate the solution.


In [114]:
class NN_ask (object):
    """ Feed forward using final weights from training backpropagation """
    def __init__(self, input_array, wij, wjk):
        self.input_array = input_array
        self.wij = wij
        self.wjk = wjk
    def get_ans(self):
        wij = self.wij
        wjk = self.wjk
        soln = []
        for i in range(len(self.input_array)):
            input_array = np.append(self.input_array[i],[1])
            self.hidden_sums = (sum((input_array * wij).T)).T
            self.hidden_out = np.append(sigmoid(self.hidden_sums),[1]) 
            self.output_sums = (sum((self.hidden_out * wjk).T)).T
            self.output_out = sigmoid(self.output_sums)
            for i in range(len(self.output_out)):
                if self.output_out[i] == max(self.output_out):
                    a = i
                    soln.append(a)
        return soln

In [115]:
test_net = NN_ask(test_input, x, y)

In [116]:
comp_vals = test_net.get_ans()

Calculate and Visualize Accuray


In [117]:
print(((sum((comp_vals - number_solution == 0).astype(int)) / (1797-1000)) * 100), "%")


97.7415307403 %

In [118]:
def interacting(i):
    plt.matshow(digits.images[perm[i+1000]])
    plt.show()
    print("Neural Network's Value: %d" %comp_vals[i])
    print("True Known Solution %d" %number_solution[i])

In [119]:
interact(interacting, i=(0,796));


Neural Network's Value: 3
True Known Solution 3

In [ ]: