All of Statistics

Chapter 2: Random Variables

A random variable is a mapping, $X: \Omega \mapsto \mathbb{R}$, that assigns a real number $X(\omega)$ to each outcome $\omega$.

The cumulative distribution function, or CDF, is the function $F_X(x) = P(X \leq x)$.

Properites of a CDF:

1. F is non-decreasing: $x_1 < x_2$ implies $F(x_1) \leq F(x_2)$
2. F is normalized: $lim_{x \rightarrow -\infty} F(x) = 0$ and $lim_{x \rightarrow \infty} F(x) = 1$
3. F is right-continuous: $F(x) = F(x^+)$ for all x, where $F(x^+) = lim_{y \rightarrow x, y>x}F(y)$.

X is discrete if it takes countably many values $\{x_1, x_2, ...\}$. We define the probablity function or probability mass function for X by $f_X(x) = P(X=x)$.

A random variable X is continuous if there exists a function $f_X$ such that

• $f_X(x) \geq 0$ for all x
• $\int_{-\infty}^{\infty}f_X(x)dx = 1$
• and for every $a \leq b$, $P(a < X < b) = \int_a^b f_X(x)dx$

The function $f_X$ is called the probablity density function (PDF).

$F_X(x) = \int_{-\infty}^x f_X(t)dt$

$f_X(x) = F_X^{'}(x)$ at all points x at which $F_X$ is differentiable.

Let F be the CDF for a random variable X. Then:

• $P(X=x) = F(x) - F(x^-)$
• $P(x < X \leq y) = F(y) - F(x)$
• $P(X > x) = 1 - F(x)$
• If X is continuous, then $F(b) - F(a) = P(a < X < b) = P(a \leq X < b) = P(a < X \leq b) = P(a \leq X \leq b)$

Let X be a random variable with CDF F. The inverse CDF or quantile function is defined by $F^{-1}(q) = inf\{x: F(x) > q\}$ for $q \in [0,1]$.

$F^{-1}(1/4)$ is the first quartile. $F^{-1}(1/2)$ is the median.

Some Important Discrete Random Variables

X ~ F means "X has distribution F".

• Point Mass Distribution.
  X has a point mass distribution at a, f(x)=1 for x=a and 0 otherwise.
• Discrete Uniform Distribution
• Bernoulli Distribution.
  P(X=1)=p, P(X=0)=1-p, for $p \in [0,1]$.
  X ~ Bernoulli(p).
  $f(x) = p^x(1-p)^{1-x}$ for $x \in \{0,1\}$
• Binomial Distribution
  Flip a coin n times with probability of heads $0 \leq p \leq 1$, let X be the number of heads.
  $f(x) = P(X=x)$ is the mass function.
  $f(x) = \binom{n}{x}p^x(1-p)^{n-x}$, for x = 0,...,n, 0 otherwise.
  X ~ Binomial(n,p)
• Geometric Distribution
  $P(X=k) = p(1-p)^{k-1}, k \geq 1$
  Think of X as the number of flips needed until the first head when flipping a coin.
• Poisson Distribution
  X has a Poisson distribution with parameter $\lambda$, written X ~ Poisson($\lambda$) if
  $f(x) = e^{-\lambda}\cfrac{\lambda^x}{x!}, x \geq 0$
  The Poisson is often used as a model for counts of rare events like radioactive decay and traffic accidents.

Some Important Continuous Random Variables

• Uniform Distribution
  X ~ Uniform(a,b)
  $f(x) = \frac{1}{b-a}$ for $x \in [a,b]$, 0 otherwise.
  The distribution function is
  $F(x) = \begin{cases} 0 &\mbox{if } x<a \\ \frac{x-a}{b-a} &\mbox{if } x \in [a,b] \\ 1 &\mbox{if } x > b \end{cases}$
• Normal (Gaussian)
  X has a Normal (or Gaussian) distribution with parameters $\mu$ and $\sigma$, denoted $X \sim N(\mu, \sigma^2)$, if
  $f(x) = \cfrac{1}{\sigma\sqrt{2\pi}}exp{-\cfrac{1}{2\sigma^2}(x - \mu)^2}$
  where $x \in \mathbb{R}, \mu \in \mathbb{R}, \sigma > 0$
  $\mu$ is the center or mean, $\sigma$ is the spread or standard deviation.

X has a standard normal distribution if $\mu=0$ and $\sigma=1$. The standard normal is traditionally denoted by Z, the PDF by $\phi(Z)$ and the CDF by $\Phi(Z)$.

1. If $X \sim N(\mu, \sigma^2)$, then $Z = (X - \mu)/\sigma \sim N(0,1)$
2. If $Z \sim N(0,1)$, then $X = \mu + \sigma Z \sim N(\mu, \sigma^2)$
3. If $X_i \sim N(\mu_i, \sigma_i^2)$, i=1,...n, are independent, then $\sum_{i=1}^{n}X_i \sim N(\sum_{i=1}^{n}\mu_i, \sum_{i=1}^{n}\sigma_i^2)$

If $X \sim N(\mu, \sigma^2)$, then
$P(a < X < b) = P(\cfrac{a-\mu}{\sigma} < Z < \cfrac{b-\mu}{\sigma}) = \Phi(\cfrac{b-\mu}{\sigma}) - \Phi(\cfrac{a-\mu}{\sigma})$

(There is no closed form expression for $\Phi$.)

• Exponential Distribution
  $X \sim Exp(\beta)$, if
  $f(x) = \cfrac{1}{\beta}e^{x/\beta}$
  where x>0, $beta>0$
• Gamma Distribution
  $\Gamma(\alpha) = \int_0^{\infty}y^{\alpha-1}e^{-y}dy$
  , for $\alpha>0$.
  $X \sim Gamma(\alpha, \beta)$, if
  $f(x) = \cfrac{1}{\beta^\alpha \Gamma(\alpha)}x^{\alpha-1}e^{-x/\beta}, x>0$,
  where $\alpha>0, \beta>0$
• Beta Distribution
  $X \sim Beta(\alpha, \beta)$, $\alpha > 0$, $\beta > 0$, if
  $f(x) = \cfrac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$, $0<x<1$
• t Distribution
  X has a $t$ distribution with $\nu$ degrees of freedom, written
  $X \sim t_\nu$
  $f(x) = \cfrac{\Gamma(\frac{\nu+1}{2})}{\Gamma(\frac{\nu}{2})}\cfrac{1}{(1+\frac{x^2}{\nu})^{(\nu+1)/2}}$
• Cauchy Distribution
  The Cauchy Distribution is a special case of the $t$ distribution corresponding to $\nu=1$. The density is
  $f(x) = \cfrac{1}{\pi(1 + x^2)}$
• $\chi^2$ Distribution
  X has a $\chi^2$ distribution with p degrees of freedom, written
  $X \sim \chi_p^2$, if
  $f(x) = \cfrac{1}{\Gamma(p/2)2^{p/2}}x^{(p/2)-1}e^{-x/2}$, x>0
  If $Z_1,...,Z_p$ are independent standard Normal random variables, then $\sum_{i=1}^{p}Z_i^2 \sim \chi_p^2$

Bivariate Distributions

Join mass function of a pair of discrete random variables X and Y, f(x,y) = P(X=x and Y=y) = P(X=x, Y=y)

In the continuous case, a function f(x,y) is a PDF of the random variables (X,Y) if

• $f(x,y) \geq 0$ for all (x,y)
• $\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f(x,y)dxdy = 1$
• For any set $A \subset \mathbb{R}^2$, $P((X,Y) \in A) = \int\int_A f(x,y)dxdy$

In the discrete or continuous case we define the join CDF as
$F_{X,Y}(x,y) = P(X \leq x, Y \leq y)$

Marginal Distributions

If (X,Y) have a joint distribution with mass function $f_{X,Y}$, then the marginal mass function for X is defined by
$f_X(x) = P(X=x) = \sum_y P(X=x,Y=y) = \sum_y f(x,y)$

Similarly the marginal mass function for Y is defined by
$f_Y(y) = P(Y=y) = \sum_x P(X=x,Y=y) = \sum_x f(x,y)$

And for continuous random variables, the marginal densities are
$f_X(x) = \int f(x,y)dy$
$f_Y(y) = \int f(x,y)dx$

Independent Random Variables

Two random variables X and Y are independent if, for every A and B,
$P(X \in A,Y \in B) = P(X \in A)P(Y \in B)$

and we write $X \coprod Y$

Let X and Y have join PDF $f_{X,Y}$. then $X \coprod Y$ iff $f_{X,Y}(x,y) = f_X(x)f_Y(y)$ for all values of x and y.

Suppose the range of X and Y is a (possibly infinite) rectangle. If f(x,y)=g(x)h(y) for some functions g and h (not necessarily probablity density functions) then X and Y are independent.

Conditional Distributions

The conditional probability mass function is
$f_{X|Y}(x|y) = P(X=x|Y=y) = \cfrac{P(X=x,Y=y)}{P(Y=y)} = \cfrac{f_{X,Y}(x,y)}{f_Y(y)}$

For continuous random variables, the conditional probability density function is
$f_{X|Y}(x|y) = \cfrac{f_{X,Y}(x,y)}{f_Y(y)}$
assuming $f_Y(y) > 0$. Then
$P(X \in A|Y=y) = \int_A f_{X|Y}(x|y)dx$

Multivariate Distributions and IID Samples

If $X_1,...,X_n$ are independent and each has the same marginal distribution cith CDF F, we say that $X_1,...X_n$ are independent and identically distributed (IID)
$X_1,...,X_n \sim F$