# Lecture 18: MGFs to get moments of Expo and Normal, sums of Poissons, joint distributions

## MGF for $\operatorname{Expo}(1)$

Let $X \sim \operatorname{Expo}(1)$.

We begin by finding the MGF of $X$:

\begin{align} M(t) &= \mathbb{E}(e^{tX}) &\quad \text{ definition of MGF} \\ &= \int_{0}^{\infty} e^{-x} \, e^{tx} dx \\ &= \int_{0}^{\infty} e^{-x(1-t)} dx \\ &= \boxed{\frac{1}{1-t}} &\quad \text{ for } t < 1 \end{align}

In finding the moments, by definition we have:

• $M'(0) = \mathbb{E}(X)$
• $M''(0) = \mathbb{E}(X^2)$
• $M'''(0) = \mathbb{E}(X^3)$
• ... and so on ...

Even though finding derivatives of $\frac{1}{1-t}$ is not all that bad, it is nevertheless annoying busywork. But since we know that the $n^{th}$ moment is the coefficient of the $n^{th}$ term of the Taylor Expansion of $X$, we can leverage that fact instead.

\begin{align} \frac{1}{1-t} &= \sum_{n=0}^{\infty} t^n &\quad \text{ for } |t| < 1 \\ &= \sum_{n=0}^{\infty} \frac{n! \, t^n}{n!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\ \\ \Rightarrow \mathbb{E}(X^n) &= \boxed{n!} \end{align}

And now we can simply generate arbitary moments for r.v. $X$!

• $\mathbb{E}(X) = 1! = 1$
• $\mathbb{E}(X^2) = 2! = 2$
• $\Rightarrow \operatorname{Var}(X) = \mathbb{E}(X^2) - \mathbb{E}X^2 = 2 - 1 = 1$

## MGF for $\operatorname{Expo}(\lambda)$

Let $Y \sim \operatorname{Expo}(\lambda)$.

We begin with

\begin{align} \text{let } X &= \lambda Y \\ \text{and so } X &= \lambda Y \sim \operatorname{Expo}(1) \\ \\ \text{then } Y &= \frac{X}{\lambda} \\ Y^n &= \frac{X^n}{\lambda^n} \\ \\ \Rightarrow \mathbb{E}(Y^n) &= \frac{\mathbb{E}(X^n)}{\lambda^n} \\ &= \boxed{\frac{n!}{\lambda^n}} \end{align}

And as before, we now can simply generate arbitary moments for r.v. $Y$!

• $\mathbb{E}(Y) = \frac{1!}{\lambda^1} = \frac{1}{\lambda}$
• $\mathbb{E}(Y^2) = \frac{2!}{\lambda^2} = \frac{2}{\lambda^2}$
• $\Rightarrow \operatorname{Var}(Y) = \mathbb{E}(Y^2) - \mathbb{E}Y^2 = \frac{2}{\lambda^2} - \left(\frac{1}{\lambda}\right)^2 = \frac{1}{\lambda^2}$

## MGF for standard Normal $\mathcal{N}(0, 1)$

Let $Z \sim \mathcal{N}(0,1)$; find all its moments.

We have seen before that, by symmetry, $\mathbb{E}(Z^{2n+1}) = 0$ for all odd values of $n$.

So we will focus in on the even values of $n$.

Now the MGF $M(t) = e^{t^2/2}$. Without taking any derivatives, we can immediately Taylor expand that, since it is continuous everywhere.

\begin{align} M(t) &= e^{t^2/2} \\ &= \sum_{n=0}^{\infty} \frac{\left(t^2/2\right)^n}{n!} \\ &= \sum_{n=0}^{\infty} \frac{t^{2n}}{2^n \, n!} \\ &= \sum_{n=0}^{\infty} \frac{(2n)! \, t^{2n}}{2^n \, n! \, (2n)!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\ \\ \Rightarrow \mathbb{E}(Z^{2n}) &= \boxed{\frac{(2n)!}{2^n \, n!}} \end{align}

Let's double-check this with what know about $\mathbb{Var}(Z)$

• by symmetry, we know that $\mathbb{E}(Z) = 0$
• at $n = 1$, $\mathbb{E}(Z^2) = \frac{2!}{2 \times 1!} = 1$
• $\Rightarrow \operatorname{Var}(Z) = \mathbb{E}(Z^2) - \mathbb{E}Z^2 = 1 - 0 = 1$
• at $n = 2$, $\mathbb{E}(Z^4) = \frac{4!}{4 \times 2!} = 3$
• at $n = 3$, $\mathbb{E}(Z^6) = \frac{6!}{8 \times 3!} = 15$

And so you might have noticed a pattern here. Let us rewrite those even moments once more:

• at $n = 1$, $\mathbb{E}(Z^2) = 1$
• at $n = 2$, $\mathbb{E}(Z^4) = 1 \times 3 = 3$
• at $n = 3$, $\mathbb{E}(Z^6) = 1 \times 3 \times 5 = 15$

## MGF for $\operatorname{Pois}(\lambda)$

Let $X \sim \operatorname{Pois}(\lambda)$; now let's consider MGFs and how to use them to find sums of random variables (convolutions).

\begin{align} M(t) &= \mathbb{E}(e^{tX}) \\ &= \sum_{k=0}^{\infty} e^{tk} \, \frac{\lambda^k e^{-\lambda}}{k!} \\ &= e^{-\lambda} \sum_{k=0}^{\infty} \frac{\lambda^k e^{tk}}{k!} \\ &= e^{-\lambda} e^{\lambda e^t} &\quad \text{but the right is just another Taylor expansion} \\ &= \boxed{e^{\lambda (e^t - 1)}} \end{align}

Now let's let $Y \sim \operatorname{Pois}(\mu)$, and it is independent of $X$. Find the distribution of $(X + Y)$.

You may recall that with MGFs, all we need to do is multiply the MGFs.

\begin{align} e^{\lambda (e^t - 1)} \, e^{\mu (e^t - 1)} &= e^{(\lambda + \mu)(e^t - 1)} \\ \\ \Rightarrow X + Y &\sim \mathcal{Pois}(\lambda + \mu) \end{align}

When adding a Poisson distribution $X$ with another independent Poisson distribution $Y$, the resulting convolution $X + Y$ will also be Poisson. This interesting relation only happens with Poisson distributions.

Now think about what happens with $X$ and $Y$ are not independent.

Let $Y = X$, so that $X + Y = 2X$, which is clearly not Poisson, as

• $X + Y = 2X$ is an even function which cannot be Poisson, since Poisson can take on all values both even and odd
• Mean $\mathbb{E}(2x) = 2\lambda$, but $\operatorname{Var}(2X) = 4\lambda$, and since the mean and variance are not equal, this cannot be Poisson

### Some definitions

In the most basic case of two r.v. in a joint distribution, consider both r.v.'s together:

Joint CDF

In the general case, the joint CDF fof two r.v.'s is $F(x,y) = P(X \le x, Y \le y)$

Joint PDF $f(x, y)$ such that, in the continuous case $P((X,Y) \in B) = \iint_B f(x,y)\,dx\,dy$

#### Joint PMF

$f(x, y)$ such that, in the discrete case

\begin{align} P(X=x, Y=y) \end{align}

We also can consider a single r.v. of a joint distribution:

#### Independence and Joint Distributions

$X, Y$ are independent iff $F(x,y) = F_X(x) \, F_Y(y)$.

\begin{align} P(X=x, Y=y) &= P(X=x) \, P(Y=y) &\quad \text{discrete case} \\ \\\\ f(x, y) &= f_X(x) \, f_Y(y) &\quad \text{continuous case} \end{align}

... with the caveat that this must be so for all $\text{x, y} \in \mathbb{R}$

#### Marginals (and how to get them)

$P(X \le x)$ is the marginal distribution of $X$, where we consider one r.v. at a time.

In the case of a two-r.v. joint distribution, we can get the marginals by using the joint distribution itself:

\begin{align} P(X=x) &= \sum_y P(X=x, Y=y) &\quad \text{marginal PMF, discrete case, for } x \\ \\\\ f_Y(y) &= \int_{-\infty}^{\infty} f_{(X,Y)}(x,y) \, dx &\quad \text{marginal PDF, continuous case, for } y \end{align}

## Example: Discrete Joint Distribution

Let $X, Y$ be both Bernoulli. $X$ and $Y$ may be independent; or they might be dependent. They may or may not have the same $p$. But they are both related in the form of a joint distribution.

We can lay out this joint distribution in a $2 \times 2$ contigency table like below:

$Y=0$ $Y=1$
$X=0$ 2/6 1/6
$X=1$ 2/6 1/6

In order to be a joint distribution, all of the values in our contigency table must be positive; and they must all sum up to 1. The example above shows such a PMF.

Let's add the marginals for $X$ and $Y$ to our $2 \times 2$ contigency table:

$Y=0$ $Y=1$ ...
$X=0$ 2/6 1/6 3/6
$X=1$ 2/6 1/6 3/6
... 4/6 2/6

Observe how in our example, we have:

\begin{align} P(X=0,Y=0) &= P(X=0) \, P(Y=0) \\ &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\ \\ P(X=0,Y=1) &= P(X=0) \, P(Y=1) \\ &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\ P(X=1,Y=0) &= P(X=1) \, P(Y=0) \\ &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\ \\ P(X=1,Y=1) &= P(X=1) \, P(Y=1) \\ &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\ \end{align}

and so you can see that $X$ and $Y$ are independent.

Now here's an example of a two r.v. joint distribution where $X$ and $Y$ are dependent; check it out for yourself.

$Y=0$ $Y=1$
$X=0$ 1/3 0
$X=1$ 1/3 1/3

## Example: Continuous Joint Distribution

Now say we had Uniform distributions on a square such that $x,y \in [0,1]$.

The joint PDF would be constant on/within the square; and 0 outside.

\begin{align} \text{joint PDF} &= \begin{cases} c &\quad \text{if } 0 \le x \le 1 \text{, } 0 \le y \le 1 \\ \\ 0 &\quad \text{otherwise} \end{cases} \end{align}

In 1-dimension space, if you integrate $1$ over some interval you get the length of that interval.

In 2-dimension space, if you integrate $1$ over some region, you get the area of that region.

Normalizing $c$, we know that $c = \frac{1}{area} = 1$.

Marginally, $X$ and $Y$ are independent $\mathcal{Unif}(1)$.

## Example: Dependent, Continuous Joint Distribution

Now say we had Uniform distributions on a disc such that $x^2 + y^2 \le 1$.

#### Joint PDF

In this case, the joint PDF is $\operatorname{Unif}$ over the area of a disc centered at the origin with radius 1.

\begin{align} \text{joint PDF} &= \begin{cases} \frac{1}{\pi} &\quad \text{if } x^2 + y^2 \le 1 \\ \\ 0 &\quad \text{otherwise} \end{cases} \end{align}