These exercises will give you the chance to practice downloading a csv file using requests and dealing with the data in that file using Pandas.
Should you struggle with the download of the data, the data file is included in the folder for your convenience.
The questions are in increasing difficulty, where the first question should take you less than a minute and the last one you might not be able to figure out. Good luck!
Note: I appologize for the solutions and questions not being next to each other, but there is a numbering issue in the Markdown that generates this text. Sorry, but it is a known bug that has yet to be fixed!
BytesIO and Pandas to read in the csv data to two seperate data frames.head, tail, describe, shape, and any other methods you find interesting to learn more about your data sets.join and merge to do an inner join of the two data frames, keeping only years that appear in both data sets. NOTE: These two will do the same thing, it's just for practice. If you struggle with join don't forget about the index!map function to make all the column headers lower case and to strip white space from the left and right. NOTE: To see why this is necessary, print the column names. There are extra spaces!shift method to create a lagged value for capital to labor ratio k. Given this, generate a column of first differences in k.groupby to generate average values of all variables during and outside of war (ie when w is 1 or 0 respectively).merge to add a column for average technology level during war time and during peace time to your data frame, then create a column for technology deviation from the mean.conda install seaborn or pip install seaborn. NOTE: Apparently the pip version is more up to date.pairplot of your data, just to take a look. With so few observations, we can't say much... NOTE: If you get an error like AttributeError: max must be larger than min in range parameter., it is probably caused by the Nan in the lag. Come up with some way to deal with this!
In [89]:
import requests
import numpy as np
import pandas as pd
from io import BytesIO
import seaborn as sns
%matplotlib inline
In [32]:
c = requests.Session()
URL = 'http://www.stern.nyu.edu/~wgreene/Text/Edition7/TableF2-1.csv'
page = c.get(URL)
c = requests.Session()
URL = 'http://www.stern.nyu.edu/~wgreene/Text/Edition7/TableF6-4.csv'
page2 = c.get(URL)
In [33]:
data1 = pd.read_csv(BytesIO(page.content))
data2 = pd.read_csv(BytesIO(page2.content))
In [34]:
data1.head()
Out[34]:
In [35]:
data2.head()
Out[35]:
In [36]:
data1.tail()
Out[36]:
In [37]:
data2.tail()
Out[37]:
In [38]:
data1.describe()
Out[38]:
In [39]:
data2.describe()
Out[39]:
In [40]:
merged = pd.merge(data1, data2, how="inner", on='YEAR')
print(data1.shape, data2.shape, merged.shape)
merged.head()
Out[40]:
In [46]:
# NOTE: Join requires the index to be defined, so set the index for both frames
joined = data1.set_index('YEAR').join(data2.set_index('YEAR'), how="inner")
print(data1.shape, data2.shape, joined.shape)
joined.head()
Out[46]:
In [58]:
merged.columns = map(str.lower, merged.columns)
merged.columns = map(str.lstrip, merged.columns)
merged.columns = map(str.rstrip, merged.columns)
print(merged.columns)
In [51]:
merged['lagged_k'] = merged['k'].shift()
merged.head()
Out[51]:
In [54]:
merged['diff_k'] = merged['k'] - merged['lagged_k']
merged.head()
Out[54]:
In [60]:
grouped = merged.groupby('w')
In [64]:
grouped.aggregate(np.mean)
Out[64]:
In [78]:
means = grouped['a'].aggregate(np.mean).reset_index()
means.columns = ['w', 'a_w']
In [80]:
merged1 = pd.merge(merged, means, how="left", on="w")
merged1
Out[80]:
In [84]:
merged1['a_dev'] = merged1['a'] - merged1['a_w']
merged1
Out[84]:
In [90]:
sns.pairplot(merged1.set_index('year').dropna())
Out[90]: