In [ ]:

%autosave 0
from IPython.core.display import HTML, display
display(HTML('<style>.container {width:100% !important;} </style>'))



# The Sigmoid Function



In [ ]:

import matplotlib.pyplot as plt
import seaborn           as sns
import numpy             as np
from math import *



The sigmoid function $S:\mathbb{R} \rightarrow [0,1]$ is defined as: $$S(t) = \frac{1}{1 + \exp(-t)}.$$



In [ ]:

def sigmoid(t):
"Returns the sigmoid function S(t)"
return 1 / (1 + np.exp(-t))



Let us plot this function.



In [ ]:

n = 1000
X = np.linspace(-8.0, 8.0, num=n)
Y = sigmoid(X)
plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.margins(0.02)
plt.plot(X, Y)
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel("x")
plt.ylabel("y")
plt.yticks(np.arange(0.0, 1.01, step=0.1))
plt.title("The Sigmoid Function.")



It can be shown that the derivative of the sigmoid function satisfies the equation $$\frac{\mathrm{d}S}{\mathrm{d}x}(t) = S(t) \cdot \bigl(1 - S(t)\bigr)$$



In [ ]:

def sigmoidPrime(t):
"""
compute the derivative of the sigmoid function
"""
return sigmoid(t) * (1 - sigmoid(t))



The Gauss function with standard deviation $\sigma$ is defined as $$t \mapsto \frac{1}{\sqrt{2\cdot\pi}\cdot\sigma} \cdot \exp\left(-\frac{t^2}{2\cdot \sigma^2}\right)$$



In [ ]:

def gauss(t):
"Return the Gauss function with standard deviation sqrt(1/3) * pi"
s = sqrt(1/3) * pi
return np.exp(-t*t/(2*s*s)) / sqrt(2 * pi) / s



Let is plot both the derivative of the sigmoid function and the Gauss function with standard deviation $\large\frac{\pi}{\sqrt{3}}$.



In [ ]:

n = 1000
X = np.linspace(-8.0, 8.0, num=n)
Y = sigmoidPrime(X)
G = gauss(X)
Z = np.zeros((n,1))
sns.set(style='darkgrid')
plt.figure(figsize=(15, 10))
plt.margins(0.02)
plt.plot(X, Y, color='b', label='S´(t)')
plt.plot(X, G, color='r', label='G(t)')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel("x")
plt.ylabel("y")
plt.title("The derivative of the sigmoid function.")



It is apparent that the derivative of the sigmoid function has a shape that is similar to the shape of a Gaussian.



In [ ]: