Derive the gradients of the sigmoid function

2017-04-14 jkang
Python 3.5

Ref:


Sigmoid Function

\begin{align*} \sigma(x) = \dfrac{1}{1 + e^{-x}} \end{align*}

In [2]:
import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    '''
    x is assumed to have (examples) x (features) dataset
    '''
    return 1 / (1 + np.exp(-x))

x = np.linspace(-10, 10, 100)
y = sigmoid(x)

plt.subplot(211)
plt.title('Sigmoid function')
plt.plot(x, y)
plt.grid('on')

plt.subplot(212)
plt.title('1st Derivatie of Sigmoid')
plt.plot(x[:-1], y[1:]-y[:-1])
plt.grid('on')
plt.show()
plt.subplots_adjust(hspace=0.8)


Derivative of Sigmoid Function

\begin{align*} \dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\ \\ &= \dfrac{d}{dx} \left( 1 + e^{-x} \right)^{-1} \\ \\ &= \dfrac{\partial(1 + e^{-x})^{-1}}{\partial(1 + e^{-x})} \cdot \dfrac{\partial(1 + e^{-x})}{\partial e^{-x}} \cdot \dfrac{\partial e^{-x}}{\partial x} \quad(chain\ rule)\\ \\ &= -(1 + e^{-x})^{-2} \cdot -e^{-x} \\ \\ &= \dfrac{e^{-x}}{(1 + e^{-x})^{2}} \\ \\ &= \dfrac{1}{1 + e^{-x}} \cdot \dfrac{e^{-x}}{1 + e^{-x}} \\ \\ &= \dfrac{1}{1 + e^{-x}} \left(1 - \dfrac{1}{1 + e^{-x}} \right) \\ \\ &= \mathbf{\sigma(x)(1 - \sigma(x)) = \sigma^\prime(x)} \end{align*}