This notebook will walk you through some exercises to get practice using Pandas for data manipulation.
As you use this, feel free to make ample use of the Pandas Documentation, the Pandas StackOverflow Channel, and your favorite search engine. For example, if you search phrases like "Pandas sum all columns", you're very likely to find an answer to the question you have in mind.
Also, if it comes down to it, note that solutions are available in the Git repository.
In [1]:
# Start with our normal batch of imports and settings
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
# Following is optional: set plotting styles
import seaborn; seaborn.set()
In [2]:
def load_year(year):
data = pd.read_csv('../data/names/yob{0}.txt'.format(year),
names=['name', 'gender', 'births'])
data['year'] = year
return data
names = pd.concat([load_year(year) for year in range(1880, 2014)])
names.head()
Out[2]:
In [3]:
# Plot the prevalence of my name over time
my_name = 'Jacob'
subset = names[names.name == my_name]
births = subset.pivot_table('births', index='year',
columns='gender', aggfunc='sum')
births.plot();
In [4]:
# Whoah... there are some female "Jacob"s? Let's look at this:
births['F'].fillna(0).plot();
In [5]:
# Now we'll normalize the births against the total for each year
def add_birth_frac(group):
group['birth_frac'] = group.births / group.births.sum()
return group
names = names.groupby(['year', 'gender']).apply(add_birth_frac)
names.head()
Out[5]:
In [6]:
names[names.name == my_name].pivot_table('birth_frac', index='year',
columns='gender', aggfunc='sum').plot();
My name (Jacob) seems to have dipped in popularity in the 1960s, only to make a profound rise around the time I was born (1981). I'll choose to presume that I was the trend-setter.
This is a bit tricky: you might be tempted to use a groupby and apply over the multiple indices ['year', 'gender', 'name'], but if you try this you'll find that it's very computationally intensive.
I'd suggest doing the following:
Is a name more likely to transition from female to male, or from male to female?
In [7]:
births = names.pivot_table('births', index=['year', 'name'], columns='gender')
births = births.fillna(0)
births.head()
Out[7]:
In [8]:
early_period = births[:1920].fillna(0).sum(axis=0, level=1)
late_period = births[1980:].fillna(0).sum(axis=0, level=1)
early_period.head()
Out[8]:
In [9]:
early_pct_M = 100 * early_period['M'] / early_period.sum(1)
late_pct_M = 100 * late_period['M'] / late_period.sum(1)
fractions = pd.DataFrame({'early_pct_M': early_pct_M,
'late_pct_M': late_pct_M})
fractions.head()
Out[9]:
In [10]:
fractions[(fractions.early_pct_M < 40) & (fractions.late_pct_M > 60)]
Out[10]:
In [11]:
fractions[(fractions.late_pct_M < 40) & (fractions.early_pct_M > 60)]
Out[11]:
Apparently it's much more common for a name to transition from male to female than the other way around!