

In [ ]:

    
%autosave 0
from IPython.core.display import HTML, display
display(HTML('<style>.container {width:100% !important;} </style>'))

The Sigmoid Function



In [ ]:

    
import matplotlib.pyplot as plt 
import seaborn           as sns
import numpy             as np
from math import *

The sigmoid function $S:\mathbb{R} \rightarrow [0,1]$ is defined as: $$ S(t) = \frac{1}{1 + \exp(-t)}. $$



In [ ]:

    
def sigmoid(t):
    "Returns the sigmoid function S(t)"
    return 1 / (1 + np.exp(-t))

Let us plot this function.



In [ ]:

    
n = 1000
X = np.linspace(-8.0, 8.0, num=n)
Y = sigmoid(X)
plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.margins(0.02)
plt.plot(X, Y)
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel("x")
plt.ylabel("y")
plt.yticks(np.arange(0.0, 1.01, step=0.1))
plt.title("The Sigmoid Function.")

It can be shown that the derivative of the sigmoid function satisfies the equation $$ \frac{\mathrm{d}S}{\mathrm{d}x}(t) = S(t) \cdot \bigl(1 - S(t)\bigr) $$



In [ ]:

    
def sigmoidPrime(t):
    """
    compute the derivative of the sigmoid function
    """
    return sigmoid(t) * (1 - sigmoid(t))

The Gauss function with standard deviation $\sigma$ is defined as $$ t \mapsto \frac{1}{\sqrt{2\cdot\pi}\cdot\sigma} \cdot \exp\left(-\frac{t^2}{2\cdot \sigma^2}\right)$$



In [ ]:

    
def gauss(t):
    "Return the Gauss function with standard deviation sqrt(1/3) * pi"
    s = sqrt(1/3) * pi
    return np.exp(-t*t/(2*s*s)) / sqrt(2 * pi) / s

Let is plot both the derivative of the sigmoid function and the Gauss function with standard deviation $\large\frac{\pi}{\sqrt{3}}$.



In [ ]:

    
n = 1000
X = np.linspace(-8.0, 8.0, num=n)
Y = sigmoidPrime(X)
G = gauss(X)
Z = np.zeros((n,1))
sns.set(style='darkgrid')
plt.figure(figsize=(15, 10))
plt.margins(0.02)
plt.plot(X, Y, color='b', label='S´(t)')
plt.plot(X, G, color='r', label='G(t)')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel("x")
plt.ylabel("y")
plt.title("The derivative of the sigmoid function.")

It is apparent that the derivative of the sigmoid function has a shape that is similar to the shape of a Gaussian.



In [ ]: