# Back propagation algorithm

## Perceptron learning algorithm (Recap)

Output is simply: $$y(x) = \mathbf{w^Tx}$$

The classifying hyperplane is simply $y(x) = 0$. $\mathbf{w}$ is therefore the normal vector to the classifying plane. Let's use this geometric interpretation to understand the perceptron learning algorithm (and later on backpropagation algorithm).

## Perceptron learning algorithm

Weights are updated using the following algorithm:

$$\mathbf{w(n+1)} = \mathbf{w(n)} + \eta (t^k - o^k) \mathbf{x}$$

where, $t^k$ is the ground truth and $o^k$ is the obtained class for the kth output node.

The geometric interpretation of this is simply moving the weight vector towards the missclassified node.

Let's try to train a perceptron on the iris dataset using the above equation, while visualizing the slope change.



In [1]:

# Import libraries
%matplotlib inline
from sklearn import datasets
import matplotlib.pyplot as plt
import numpy as np
from copy import deepcopy



## Let's load some data to train a perceptron



In [2]:

# Import some data to play
X = iris.data[:100,:2]  # we only take 10 samples of the first two features.
Y = iris.target[:100]
ground_truth = Y

# Let's cheat a bit (center the distribution)
X[:,0] = X[:,0] - np.mean(X[:,0])
X[:,1] = X[:,1] - np.mean(X[:,1])



Let's center the distribution. This way, we can keep the threshold at 0 and not use biases (essentially a cheat). We can visualize the learning algorithm better this way.



In [3]:

# Line parameters
m = 1 # Slope
c = 0  # Intercept




In [4]:

# Creating line
x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
y1 = m*x1 + c




In [5]:

# Plot the training points
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(x1,y1)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')




Out[5]:

<matplotlib.text.Text at 0x7f53ed1ae908>



## Perceptron Learning algorithm (implementation)

Let's start of with an intial set of weights and visualize the weight vector.



In [6]:

w = np.array([-0.1,0.6])



Line equation is $$\mathbf{w^Tx} = 0$$

where $w = \begin{bmatrix} 0.1 \\ 0.2\\ \end{bmatrix}$ and $x = \begin{bmatrix} I_1^1 & I_2^1 & I_3^1 & ... & I_n^1\\ I_2^2 & I_2^2 & I_3^2 & ... & I_n^2\\ \end{bmatrix}$



In [7]:

# Calculate slopes
m = -w[0]/w[1]
c = 0
# Creating slope line
x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
y1 = m*x1 + c




In [8]:

plt.figure(2, figsize=(5, 5))
ax = plt.axes()
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(x1,y1,linestyle='--')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')




Out[8]:

<matplotlib.text.Text at 0x7f53eb0f4c88>




In [9]:

# Obtain classes
perceptron_output = (X.dot(w) < 0) + 0

# Compare
classification_status = (perceptron_output == ground_truth)
X_incorrect = X[np.logical_not(classification_status),:]
delta = ground_truth - perceptron_output
delta = -1 * delta[np.logical_not(classification_status)]

print("Classification_status:")
print(classification_status)
print("\nIncorrectly classified:")
print(np.sum(np.logical_not(classification_status)))




Classification_status:
[ True  True  True  True  True  True  True  True False  True  True  True
True  True  True  True  True  True  True  True  True  True  True  True
True False  True  True  True  True  True  True  True  True  True  True
True  True  True  True  True False  True  True  True  True  True  True
True  True  True  True  True  True  True  True False  True  True  True
True  True  True  True  True  True  True  True  True  True False  True
True  True  True  True  True  True  True  True  True  True  True  True
True False  True  True  True  True  True  True  True  True  True  True
True  True  True  True]

Incorrectly classified:
6




In [10]:

plt.figure(2, figsize=(5, 5))
ax = plt.axes()
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(X_incorrect[:, 0], X_incorrect[:, 1],linestyle='None',color='k',marker='x', ms=10.0, mew=1.0)
plt.plot(x1,y1,linestyle='--')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')




Out[10]:

<matplotlib.text.Text at 0x7f53eb066c50>



## Visualization of weight update



In [11]:

# Define a function for plotting
def plot_vectors(X,Y,X_incorrect,w,new_weight):
# Calculate slopes
m = -w[0]/w[1]
c = 0
# Creating slope line
x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
y1 = m*x1 + c
# Plot
plt.figure(2, figsize=(6, 6))
plt.clf()
ax = plt.axes()
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(X_incorrect[:, 0], X_incorrect[:, 1],linestyle='None',color='k',marker='x', ms=10.0, mew=1.0)
plt.plot(x1,y1,linestyle='--')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
# Set limits
max_limit = np.max(np.max(np.abs(X)))
plt.xlim(-max_limit, max_limit)
plt.ylim(-max_limit, max_limit)




In [12]:

# Apply perceptron learning algorithm
learning_rate = 1
new_weight = w + learning_rate * delta[0] * X_incorrect[0,:]
# Plot old and new weight vectors, and the incorrect X
plot_vectors(X,Y,X_incorrect,w,new_weight)







In [13]:

# Obtain classes
w = deepcopy(new_weight)
perceptron_output = (X.dot(w) < 0) + 0

# Compare
classification_status = (perceptron_output == ground_truth)
X_incorrect = X[np.logical_not(classification_status),:]
delta = ground_truth - perceptron_output
delta = -1 * delta[np.logical_not(classification_status)]

print("Classification_status:")
print(classification_status)
print("\nIncorrectly classified:")
print(np.sum(np.logical_not(classification_status)))




Classification_status:
[ True  True  True  True  True  True  True  True  True  True  True  True
True  True False  True  True  True  True  True  True  True  True  True
True  True  True  True  True  True  True  True  True  True  True  True
True  True  True  True  True  True  True  True  True  True  True  True
True  True  True  True  True  True  True  True  True False  True False
False  True  True  True  True  True  True  True  True  True  True  True
True  True  True  True  True  True  True  True  True  True  True  True
False  True  True  True  True  True  True  True  True False  True  True
True  True False  True]

Incorrectly classified:
7




In [14]:

# Apply perceptron learning algorithm
learning_rate = 1
new_weight = w + learning_rate * delta[0] * X_incorrect[0,:]
# Plot old and new weight vectors, and the incorrect X
plot_vectors(X,Y,X_incorrect,w,new_weight)






## Backpropagation



In [ ]: