# Examples and Exercises from Think Stats, 2nd Edition

http://thinkstats2.com

``````

In :

from __future__ import print_function, division

%matplotlib inline

import numpy as np

import nsfg
import first
import thinkstats2
import thinkplot

``````

Again, I'll load the NSFG pregnancy file and select live births:

``````

In :

live = preg[preg.outcome == 1]

``````

Here's the histogram of birth weights:

``````

In :

hist = thinkstats2.Hist(live.birthwgt_lb, label='birthwgt_lb')
thinkplot.Hist(hist)
thinkplot.Config(xlabel='Birth weight (pounds)', ylabel='Count')

``````
``````

``````

To normalize the disrtibution, we could divide through by the total count:

``````

In :

n = hist.Total()
pmf = hist.Copy()
for x, freq in hist.Items():
pmf[x] = freq / n

``````

The result is a Probability Mass Function (PMF).

``````

In :

thinkplot.Hist(pmf)
thinkplot.Config(xlabel='Birth weight (pounds)', ylabel='PMF')

``````
``````

``````

More directly, we can create a Pmf object.

``````

In :

pmf = thinkstats2.Pmf([1, 2, 2, 3, 5])
pmf

``````
``````

Out:

Pmf({1: 0.2, 2: 0.4, 3: 0.2, 5: 0.2})

``````

`Pmf` provides `Prob`, which looks up a value and returns its probability:

``````

In :

pmf.Prob(2)

``````
``````

Out:

0.4

``````

The bracket operator does the same thing.

``````

In :

pmf

``````
``````

Out:

0.4

``````

The `Incr` method adds to the probability associated with a given values.

``````

In :

pmf.Incr(2, 0.2)
pmf

``````
``````

Out:

0.6000000000000001

``````

The `Mult` method multiplies the probability associated with a value.

``````

In :

pmf.Mult(2, 0.5)
pmf

``````
``````

Out:

0.30000000000000004

``````

`Total` returns the total probability (which is no longer 1, because we changed one of the probabilities).

``````

In :

pmf.Total()

``````
``````

Out:

0.8999999999999999

``````

`Normalize` divides through by the total probability, making it 1 again.

``````

In :

pmf.Normalize()
pmf.Total()

``````
``````

Out:

1.0

``````

Here's the PMF of pregnancy length for live births.

``````

In :

pmf = thinkstats2.Pmf(live.prglngth, label='prglngth')

``````

Here's what it looks like plotted with `Hist`, which makes a bar graph.

``````

In :

thinkplot.Hist(pmf)
thinkplot.Config(xlabel='Pregnancy length (weeks)', ylabel='Pmf')

``````
``````

``````

Here's what it looks like plotted with `Pmf`, which makes a step function.

``````

In :

thinkplot.Pmf(pmf)
thinkplot.Config(xlabel='Pregnancy length (weeks)', ylabel='Pmf')

``````
``````

``````

We can use `MakeFrames` to return DataFrames for all live births, first babies, and others.

``````

In :

live, firsts, others = first.MakeFrames()

``````

Here are the distributions of pregnancy length.

``````

In :

first_pmf = thinkstats2.Pmf(firsts.prglngth, label='firsts')
other_pmf = thinkstats2.Pmf(others.prglngth, label='others')

``````

And here's the code that replicates one of the figures in the chapter.

``````

In :

width=0.45
axis = [27, 46, 0, 0.6]
thinkplot.PrePlot(2, cols=2)
thinkplot.Hist(first_pmf, align='right', width=width)
thinkplot.Hist(other_pmf, align='left', width=width)
thinkplot.Config(xlabel='Pregnancy length(weeks)', ylabel='PMF', axis=axis)

thinkplot.PrePlot(2)
thinkplot.SubPlot(2)
thinkplot.Pmfs([first_pmf, other_pmf])
thinkplot.Config(xlabel='Pregnancy length(weeks)', axis=axis)

``````
``````

``````

Here's the code that generates a plot of the difference in probability (in percentage points) between first babies and others, for each week of pregnancy (showing only pregnancies considered "full term").

``````

In :

weeks = range(35, 46)
diffs = []
for week in weeks:
p1 = first_pmf.Prob(week)
p2 = other_pmf.Prob(week)
diff = 100 * (p1 - p2)
diffs.append(diff)

thinkplot.Bar(weeks, diffs)
thinkplot.Config(xlabel='Pregnancy length(weeks)', ylabel='Difference (percentage points)')

``````
``````

``````

### Biasing and unbiasing PMFs

Here's the example in the book showing operations we can perform with `Pmf` objects.

Suppose we have the following distribution of class sizes.

``````

In :

d = { 7: 8, 12: 8, 17: 14, 22: 4,
27: 6, 32: 12, 37: 8, 42: 3, 47: 2 }

pmf = thinkstats2.Pmf(d, label='actual')

``````

This function computes the biased PMF we would get if we surveyed students and asked about the size of the classes they are in.

``````

In :

def BiasPmf(pmf, label):
new_pmf = pmf.Copy(label=label)

for x, p in pmf.Items():
new_pmf.Mult(x, x)

new_pmf.Normalize()
return new_pmf

``````

The following graph shows the difference between the actual and observed distributions.

``````

In :

biased_pmf = BiasPmf(pmf, label='observed')
thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, biased_pmf])
thinkplot.Config(xlabel='Class size', ylabel='PMF')

``````
``````

``````

The observed mean is substantially higher than the actual.

``````

In :

print('Actual mean', pmf.Mean())
print('Observed mean', biased_pmf.Mean())

``````
``````

Actual mean 23.692307692307693
Observed mean 29.123376623376625

``````

If we were only able to collect the biased sample, we could "unbias" it by applying the inverse operation.

``````

In :

def UnbiasPmf(pmf, label=None):
new_pmf = pmf.Copy(label=label)

for x, p in pmf.Items():
new_pmf[x] *= 1/x

new_pmf.Normalize()
return new_pmf

``````

We can unbias the biased PMF:

``````

In :

unbiased = UnbiasPmf(biased_pmf, label='unbiased')
print('Unbiased mean', unbiased.Mean())

``````
``````

Unbiased mean 23.69230769230769

``````

And plot the two distributions to confirm they are the same.

``````

In :

thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, unbiased])
thinkplot.Config(xlabel='Class size', ylabel='PMF')

``````
``````

``````

### Pandas indexing

Here's an example of a small DataFrame.

``````

In :

import numpy as np
import pandas
array = np.random.randn(4, 2)
df = pandas.DataFrame(array)
df

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

0
1

0
0.792518
0.200374

1
-0.697067
0.067361

2
-0.731358
-1.011806

3
1.440653
-0.980818

``````

We can specify column names when we create the DataFrame:

``````

In :

columns = ['A', 'B']
df = pandas.DataFrame(array, columns=columns)
df

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

A
B

0
0.792518
0.200374

1
-0.697067
0.067361

2
-0.731358
-1.011806

3
1.440653
-0.980818

``````

We can also specify an index that contains labels for the rows.

``````

In :

index = ['a', 'b', 'c', 'd']
df = pandas.DataFrame(array, columns=columns, index=index)
df

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

A
B

a
0.792518
0.200374

b
-0.697067
0.067361

c
-0.731358
-1.011806

d
1.440653
-0.980818

``````

Normal indexing selects columns.

``````

In :

df['A']

``````
``````

Out:

a    0.792518
b   -0.697067
c   -0.731358
d    1.440653
Name: A, dtype: float64

``````

We can use the `loc` attribute to select rows.

``````

In :

df.loc['a']

``````
``````

Out:

A    0.792518
B    0.200374
Name: a, dtype: float64

``````

If you don't want to use the row labels and prefer to access the rows using integer indices, you can use the `iloc` attribute:

``````

In :

df.iloc

``````
``````

Out:

A    0.792518
B    0.200374
Name: a, dtype: float64

``````

`loc` can also take a list of labels.

``````

In :

indices = ['a', 'c']
df.loc[indices]

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

A
B

a
0.792518
0.200374

c
-0.731358
-1.011806

``````

If you provide a slice of labels, `DataFrame` uses it to select rows.

``````

In :

df['a':'c']

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

A
B

a
0.792518
0.200374

b
-0.697067
0.067361

c
-0.731358
-1.011806

``````

If you provide a slice of integers, `DataFrame` selects rows by integer index.

``````

In :

df[0:2]

``````
``````

Out:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

A
B

a
0.792518
0.200374

b
-0.697067
0.067361

``````

But notice that one method includes the last elements of the slice and one does not.

In general, I recommend giving labels to the rows and names to the columns, and using them consistently.

## Exercises

Exercise: Something like the class size paradox appears if you survey children and ask how many children are in their family. Families with many children are more likely to appear in your sample, and families with no children have no chance to be in the sample.

Use the NSFG respondent variable `numkdhh` to construct the actual distribution for the number of children under 18 in the respondents' households.

Now compute the biased distribution we would see if we surveyed the children and asked them how many children under 18 (including themselves) are in their household.

Plot the actual and biased distributions, and compute their means.

``````

In :

``````
``````

In :

# Solution

pmf = thinkstats2.Pmf(resp.numkdhh, label='numkdhh')

``````
``````

In :

# Solution

thinkplot.Pmf(pmf)
thinkplot.Config(xlabel='Number of children', ylabel='PMF')

``````
``````

``````
``````

In :

# Solution

biased = BiasPmf(pmf, label='biased')

``````
``````

In :

# Solution

thinkplot.PrePlot(2)
thinkplot.Pmfs([pmf, biased])
thinkplot.Config(xlabel='Number of children', ylabel='PMF')

``````
``````

``````
``````

In :

# Solution

pmf.Mean()

``````
``````

Out:

1.024205155043831

``````
``````

In :

# Solution

biased.Mean()

``````
``````

Out:

2.403679100664282

``````

Exercise: I started this book with the question, "Are first babies more likely to be late?" To address it, I computed the difference in means between groups of babies, but I ignored the possibility that there might be a difference between first babies and others for the same woman.

To address this version of the question, select respondents who have at least two live births and compute pairwise differences. Does this formulation of the question yield a different result?

Hint: use `nsfg.MakePregMap`:

``````

In :

live, firsts, others = first.MakeFrames()

``````
``````

In :

preg_map = nsfg.MakePregMap(live)

``````
``````

In :

# Solution

hist = thinkstats2.Hist()

for caseid, indices in preg_map.items():
if len(indices) >= 2:
pair = preg.loc[indices[0:2]].prglngth
diff = np.diff(pair)
hist[diff] += 1

``````
``````

In :

# Solution

thinkplot.Hist(hist)

``````
``````

``````
``````

In :

# Solution

pmf = thinkstats2.Pmf(hist)
pmf.Mean()

``````
``````

Out:

-0.056367432150313125

``````

Exercise: In most foot races, everyone starts at the same time. If you are a fast runner, you usually pass a lot of people at the beginning of the race, but after a few miles everyone around you is going at the same speed. When I ran a long-distance (209 miles) relay race for the first time, I noticed an odd phenomenon: when I overtook another runner, I was usually much faster, and when another runner overtook me, he was usually much faster.

At first I thought that the distribution of speeds might be bimodal; that is, there were many slow runners and many fast runners, but few at my speed.

Then I realized that I was the victim of a bias similar to the effect of class size. The race was unusual in two ways: it used a staggered start, so teams started at different times; also, many teams included runners at different levels of ability.

As a result, runners were spread out along the course with little relationship between speed and location. When I joined the race, the runners near me were (pretty much) a random sample of the runners in the race.

So where does the bias come from? During my time on the course, the chance of overtaking a runner, or being overtaken, is proportional to the difference in our speeds. I am more likely to catch a slow runner, and more likely to be caught by a fast runner. But runners at the same speed are unlikely to see each other.

Write a function called `ObservedPmf` that takes a `Pmf` representing the actual distribution of runners’ speeds, and the speed of a running observer, and returns a new `Pmf` representing the distribution of runners’ speeds as seen by the observer.

To test your function, you can use `relay.py`, which reads the results from the James Joyce Ramble 10K in Dedham MA and converts the pace of each runner to mph.

Compute the distribution of speeds you would observe if you ran a relay race at 7 mph with this group of runners.

``````

In :

import relay

speeds = relay.GetSpeeds(results)
speeds = relay.BinData(speeds, 3, 12, 100)

``````
``````

In :

pmf = thinkstats2.Pmf(speeds, 'actual speeds')
thinkplot.Pmf(pmf)
thinkplot.Config(xlabel='Speed (mph)', ylabel='PMF')

``````
``````

``````
``````

In :

# Solution

def ObservedPmf(pmf, speed, label=None):
"""Returns a new Pmf representing speeds observed at a given speed.

The chance of observing a runner is proportional to the difference
in speed.

Args:
pmf: distribution of actual speeds
speed: speed of the observing runner
label: string label for the new dist

Returns:
Pmf object
"""
new = pmf.Copy(label=label)
for val in new.Values():
diff = abs(val - speed)
new[val] *= diff
new.Normalize()
return new

``````
``````

In :

# Solution

biased = ObservedPmf(pmf, 7, label='observed speeds')
thinkplot.Pmf(biased)
thinkplot.Config(xlabel='Speed (mph)', ylabel='PMF')

``````
``````

``````
``````

In [ ]:

``````