Lesson 2 SGD/Optimization Tutorial Code Along

See the notebook -- FAI1 - Practical Deep Learning I

Follow Lecture 2 @ around [1:10:00]



In [16]:

    
%matplotlib inline
import numpy as np
from numpy.random import random
# from matplotlib import pyplot as plt, animation
from matplotlib import pyplot as plt, rcParams, animation, rc
rc('animation', html='html5')
rcParams['figure.figsize'] = 3, 3 # sets plot window size
%precision 4
np.set_printoptions(precision=4, linewidth=60)



In [17]:

    
def lin(a, b, x): return a*x + b



In [18]:

    
a = 3.
b = 8.



In [19]:

    
n = 30
x = random(n)
y = lin(a, b, x)



In [20]:

    
x









    Out[20]:





array([ 0.9198,  0.7294,  0.6458,  0.7301,  0.0846,  0.1477,
        0.3399,  0.121 ,  0.9358,  0.8968,  0.1368,  0.5865,
        0.9389,  0.4044,  0.8888,  0.5425,  0.8703,  0.4314,
        0.2092,  0.6894,  0.3578,  0.738 ,  0.0765,  0.6168,
        0.8595,  0.5079,  0.9581,  0.6021,  0.2423,  0.04  ])



In [21]:

    
y









    Out[21]:





array([ 10.7594,  10.1883,   9.9375,  10.1902,   8.2537,
         8.443 ,   9.0196,   8.363 ,  10.8074,  10.6905,
         8.4105,   9.7594,  10.8167,   9.2132,  10.6663,
         9.6274,  10.611 ,   9.2941,   8.6277,  10.0681,
         9.0734,  10.214 ,   8.2294,   9.8505,  10.5784,
         9.5237,  10.8743,   9.8063,   8.7268,   8.1201])



In [22]:

    
plt.scatter(x,y)









    Out[22]:





<matplotlib.collections.PathCollection at 0x11aa8a990>



In [23]:

    
def sse(y, y_pred): return ((y - y_pred)**2).sum()
def loss(y, a, b, x): return sse(y, lin(a,b,x))
def avg_loss(y, a,b,x): return np.sqrt(loss(y,a,b,x)/n)

Let's start out with a line: y = ax + b, where a = 3, b = 8; but where we don't know what a & b are, and starting with guesses for both, use a linear model to find them



In [24]:

    
a_guess = -1
b_guess = 1
avg_loss(y, a_guess, b_guess, x)









    Out[24]:





9.2452



In [25]:

    
Lr = 0.01    # below thanks to Wolfram Alpha
# d[(y - y_pred)**2,b] = d[(y - (a*x+b))**2, b] = 2*(b + a*x - y)
# d[(y - y_pred)**2,a] = d[(y - (a*x+b))**2, a] = 2*x*(b+a*x-y) = x * dy/db

Coming up with an update function that'll make our guesses of a & b closer to what a & b actually are, each time it is called.



In [26]:

    
def update():
    global a_guess, b_guess
    y_pred = lin(a_guess, b_guess, x)
    dydb = 2*(y_pred - y)
    dyda = x*dydb
    a_guess -= Lr * dyda.mean() # new guess is minus deriv * (a little bit)
    b_guess -= Lr * dydb.mean()

Confirm that our line eventually, actually, fits our data, via animation.



In [27]:

    
fig = plt.figure(figsize=(5,4), dpi=100)
plt.scatter(x, y)
line, = plt.plot(x, lin(a_guess, b_guess, x))
plt.close()

def animate(i):
    line.set_ydata(lin(a_guess, b_guess, x))
    for i in xrange(10): update()
    return line,

ani = animation.FuncAnimation(fig, animate, np.arange(0, 40), interval=100)
ani









    Out[27]:

8 May 2017 - Wayne H Nixalo



In [ ]: