Exploratory notebook related to the theory and introductory concepts behind probability. Includes toy examples implementation and visualization.


Probability is the science concerned with the understanding and manipulation of uncertainty.

In [ ]:
import numpy as np
import seaborn as sns
import pandas as pd
from matplotlib import pyplot as plt, animation

%matplotlib notebook
#%matplotlib inline


In [ ]:
# interactive imports
import plotly
import cufflinks as cf

In [ ]:
class RandomVar:
    def __init__(self, probs):
        self.values = np.arange(len(probs))
        self.probs = probs
    def pick(self, n=1):
        return np.random.choice(self.values, p=self.probs)

In [ ]:
coin = RandomVar([0.5, 0.5])

In [ ]:
biased_coin = RandomVar([0.1, 0.9])

In [ ]:
die = RandomVar([1/6]*6)

Information Theory

We interested in understanding the amount of information related to events. For example given a random variable $x$, the amount of information of a specific value can also be seen as "degree of surprise" of seeing $x$ being equal to such value.

$$ h(x) = - \log_2 p(x) $$

For a random variable $x$, the corresponding measure calles entropy is defines as:

$$ H[x] = - \sum_x{ p(x) \log_2 p(x) } $$

In [ ]:
# information content for a target probability
def info_content(p_x):
    return -np.log2(p_x)

# entropy of a random variable probability distribution
def entropy(p_x):
    return -sum(p_x*np.log2(p_x))

In [ ]:

Maximum entropy for a discrete random variable is obtained with a uniform distribution. For a continuous random variable we have an equivalent increase in entropy for an increase in the variance.

In [ ]:
# log function
x = np.linspace(0.00001, 2, 100)
plt.plot(x, np.log(x), label='Log')

In [ ]:
#log of product equals sum of logs

n = 10
#a = np.random.random_sample(n)
#b = np.random.random_sample(n)
plt.plot(a, label='a')
plt.plot(b, label='b')
plt.plot(np.log(a), label='log(a)')
plt.plot(np.log(b), label='log(b)')
#plt.plot(np.log(a)+np.log(b), label='log(a)+log(b)')
plt.plot(np.log(a*b), label='log(a+b)')

In [ ]: