Theory of Probability and Statistics

• Understanding stochastic phenomena (greek word stochos)

Probability Space

Probability space which is also called "sample space", the space of all possible outcomes and is often denoted as $\Omega$.

Example

For example, the possible oucomes for throwing a 6-sided die is $\Omega = \left\{ 1,2,3,4,5,6\right\}$. Therefore, $\Omega$ is the probability space for this experiment.

Definition:

• Elementary Outcome is an outcome that cannot be broken down into a set of other outcomes.

• Probability Space is the set of all possible elementary outcomes from an experiment.

In the example above, each of the six possible outcomes are equally likely. This means that each elementary outcome has probability $1/6$.

• (Q) What is the chance of rolling an even number?
  • (A) The set of even numbers from a die is $\{2,4,6\}$. Therefore, the probability is $3/6 = 1/2$.

Notation

Let $A = \{2,4,6\}$.

• A is evidently a subset of $\Omega$.
• Any subset of $\Omega$ by definition is an event.
• We say probability of event $A$ is $\mathbf{P}[A] = 1/2$
• This equality is because $\mathbf{P}[A] = \mathbf{P}[\{2,4,6\}] = \mathbf{P}[\{2\}] + \mathbf{P}[\{4\}] + \mathbf{P}[\{6\}] = 1/6 + 1/6 + 1/6 = 1/2$

Note: we can separate $\mathbf{P}[\{2,4,6\}]$ into $\mathbf{P}[\{2\}] + \mathbf{P}[\{4\}] + \mathbf{P}[\{6\}]$ because these events are incompatible (disjoint) events. Meaning that they cannot occur at the same time.

• Incomaptible events are also called disjoint events. Two evets $A, B$ are disjoint if $A \cap B = \varnothing \ \ \text{(empty set)}$

Formal Axioms of Probability

For a probability space $\Omega$ and probability measure $\mathbf{P}$ (probability of any event $A \subseteq \Omega$ is $\mathbf{P}[A]$ ):

• Axiom 1: $\mathbf{P}[A] \ge 0$ for every event $A \subseteq \Omega$
• Axiom 2: If $A_1, A_2, ... A_n, ..$ is a countable family of disjoint events, then $\mathbf{P}[A_1 \cup A_2 \cup ... \cup A_n .. ] = \mathbf{P}[A_1] + \mathbf{P}[A_2] + ... + \mathbf{P}[A_n] + ...$
• Axiom 3: $\mathbf{P}[\Omega] = 1$

Discrete Probability Space

If $\Omega$ is countable (finite or infinite), then we say we have a discrete probability space, and we can replace axioms 1&2 with

• Axiom 1: $\forall \omega \in \Omega, \ \ \mathbf{P}[\{\omega\}] \ge 0$
• Axiom 2: $\forall A \subseteq \Omega, \mathbf{P}[A] = \displaystyle \sum_{\omega \in \Omega} \mathbf{P}[\{\omega\}]$

Example: toss of 3 coins

The possible outcomes are $HHH\ \ HHT\ HTT\ TTT\ TTH\ THT$

Based on the rule of "equally likely outcomes", we conclude that each of the above outcomes has probability $1/6$.

Theorem

• If $\Omega$ has equally likely outcomes, then $\forall \omega \in \Omega, \ \ \mathbf{P}[\{\omega\}] = \displaystyle \frac{1}{\left\vert\Omega \right\vert}$

Corollary

• If $\Omega$ has equally likely outcomes, then $\forall A \subseteq \Omega, \ \ \mathbf{P}[A] = \displaystyle \frac{\vert A \vert}{\vert \Omega \vert}$

Properties of Probability Measure

1. $\mathbf{P}[\varnothing] = 0$

   • Proof: $\Omega$ and $\varnothing$ are disjoint sets because $\Omega \cap \varnothing = \varnothing$. Therefore, $\mathbf{P}[\Omega \cup \varnothing] = \mathbf{P}[\Omega] + \mathbf{P}[\varnothing]$
     which is $\mathbf{P}[\Omega] = \mathbf{P}[\Omega] + \mathbf{P}[\varnothing]$
     and results in $\mathbf{P}[\varnothing] = 0$

2. $\mathbf{P}[A^c] = 1 - \mathbf{P}[A]$, where $A^c$ is the complement of event $A$

3. $\mathbf{P}[A\cup B] = \mathbf{P}[A] + \mathbf{P}[B] - \mathbf{P}[A \cap B]$
   • similarly for any three events: $\mathbf{P}[A \cup B \cup C ] = \mathbf{P}[A] + \mathbf{P}[B] + \mathbf{P}[C] - \mathbf{P}[A \cap B] - \mathbf{P}[A \cap C] - \mathbf{P}[B \cap C] + \mathbf{P}[A \cap B \cap C]$

Example:

Given weather data for two consecutive days in a city, we have the following rain events

• $R_1$ event that day 1 had rain
• $R_2$ event that day 2 had rain and these probabilities hold: $\mathbf{P}[R_1] = 0.6$, $\mathbf{P}[R_2] = 0.5$, and $\mathbf{P}[R_1 \cap R_2^c] = 0.2$.

(Question) Find the following probabilities: $\mathbf{P}[R_1 \cap R_2]=?\ \ $ $\mathbf{P}[R_1^c \cap R_2] = ?\ \ $ and $\mathbf{P}[R_1 \cup R_2]=?$

• (Answer for $\mathbf{P}[R_1 \cap R_2]$)
  $R_1 \cap R_2$ and $R_1 \cap R_2^c$ are two disjoint events. Therefore, $\mathbf{P}[R_1\cap R_2 \ \ \cup\ \ R_1 \cap R_2^c] = \mathbf{P}[R_1\cap R_2] + \mathbf{P}[R_1\cap R_2^c]$.
  And we know that $R_1\cap R_2 \ \ \cup R_1 \cap R_2^c = R_1$. Therefore, $\mathbf{P}[R_1] = \mathbf{P}[R_1 \cap R_2] + \mathbf{P}[R_1 \cap R_2^c]$.
  After rearranging the terms, it results in $\mathbf{P}[R_1 \cap R_2] = \mathbf{P}[R_1] - \mathbf{P}[R_1\cap R_2^c] = 0.6 - 0.2 = 0.4$
• (Answer for $\mathbf{P}[R_1^c \cap R_2]$)
  similar to the reason in the previous case, we can conclude that $\mathbf{P}[R_1^c \cap R_2] = \mathbf{P}[R_2] - \mathbf{P}[R_1 \cap R_2] = 0.5 - 0.4 = 0.1$
• *(Answer for $\mathbf{P}[R_1 \cup R_2]$) using property #3: $\mathbf{P}[R_1 \cup R_2] = \mathbf{P}[R_1] + \mathbf{P}[R_2] - \mathbf{P}[R_1 \cap R_2]$
  which results in $\mathbf{P}[R_1 \cup R_2] = 0.6 + 0.5 - 0.4 = 0.7$

Examle:

We are forming a jury of 6 people taken randomy among 15, where 8 of them are men. and 7 are women.
(Question) What is the chance that there are exactly two women in the jury?

• (Answer) For this we need to recall the notion of combination (see below).
  The total number of ways of choosing a jury: $\displaystyle \left(\begin{array}{c} 15 \\ 6\end{array}\right) = \frac{15\times 14 \times 13 \times 12 \times 11 \times 10}{6 \times 5 \times 4 \times 3 \times 2 \times 1} = 5005$

  Number of ways to have exactly 2 women in the jury (which also implicitly implies 4 men in the jury): $\#(\text{ways choosing 2 women}) \times \#(\text{ways choosing 4 men})$
  $\displaystyle \#(\text{ways choosing 2 women out of 7}) = \left(\begin{array}{c} 7 \\ 2\end{array}\right) = \frac{7\times 6}{2 \ times 1} = 21$
  $\displaystyle \#(\text{ways choosing 4 men out of 8}) = \left(\begin{array}{c} 8 \\ 4\end{array}\right) = \frac{8\times 7\times 6\times 5}{4\times 3 \times 2 \ times 1} = 70$

  total number of choosing a jury of 6 with 2 women and 4 men: $70 \times 21 = 1470$
  Finally: Probability of choosing a jury of 6 with 2 women and 4 men: $\mathbf{P}[A] = \displaystyle \frac{1470}{5005} = 0.2937$

Combination and Permutation

Notation: $\left(\begin{array}{c} n \\ k\end{array}\right) = \text{n choose k} $ combination:
is the number of ways to choose a subset of size $k$ among a set of size $n$.

$$\left(\begin{array}{c} n \\ k\end{array}\right) = \frac{n!}{k! (n-k)!} = \frac{n(n-1)(n-2) .. (n-k+1)}{k(k-1)(k-2) .. 1}$$

• Always remember that numerator and denumerator have the same number of multiplication coefficients, which is $k$
• In combination, the order of picking items does not matter.
• It can be proven that combination is always an integer number.

Notation: $P(n,k)$ or $A^k_n$ = permutation of k elements takn among n elements = an "arrangment":
since the are $k!$ ways to order $k$ selected elements, we get

$$P(n,k)=\left(\begin{array}{c} n \\ k\end{array}\right) k! = \frac{n!}{(n-k)!}$$

Example

In game of poker, a full house is a poker hand which contains three cards of the same rank and two other cards of another rank.