You are currently looking at version 1.1 of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the Jupyter Notebook FAQ course resource.

``````

In [2]:

import pandas as pd
import numpy as np
from scipy.stats import ttest_ind

``````

# Assignment 4 - Hypothesis Testing

This assignment requires more individual learning than previous assignments - you are encouraged to check out the pandas documentation to find functions or methods you might not have used yet, or ask questions on Stack Overflow and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff.

Definitions:

• A quarter is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.
• A recession is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.
• A recession bottom is the quarter within a recession which had the lowest GDP.
• A university town is a city which has a high percentage of university students compared to the total population of the city.

Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (`price_ratio=quarter_before_recession/recession_bottom`)

The following data files are available for this assignment:

• From the Zillow research data site there is housing data for the United States. In particular the datafile for all homes at a city level, `City_Zhvi_AllHomes.csv`, has median home sale prices at a fine grained level.
• From the Wikipedia page on college towns is a list of university towns in the United States which has been copy and pasted into the file `university_towns.txt`.
• From Bureau of Economic Analysis, US Department of Commerce, the GDP over time of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file `gdplev.xls`. For this assignment, only look at GDP data from the first quarter of 2000 onward.

Each function in this assignment below is worth 10%, with the exception of `run_ttest()`, which is worth 50%.

``````

In [3]:

# Use this dictionary to map state names to two letter acronyms
states = {'OH': 'Ohio', 'KY': 'Kentucky', 'AS': 'American Samoa', 'NV': 'Nevada', 'WY': 'Wyoming', 'NA': 'National', 'AL': 'Alabama', 'MD': 'Maryland', 'AK': 'Alaska', 'UT': 'Utah', 'OR': 'Oregon', 'MT': 'Montana', 'IL': 'Illinois', 'TN': 'Tennessee', 'DC': 'District of Columbia', 'VT': 'Vermont', 'ID': 'Idaho', 'AR': 'Arkansas', 'ME': 'Maine', 'WA': 'Washington', 'HI': 'Hawaii', 'WI': 'Wisconsin', 'MI': 'Michigan', 'IN': 'Indiana', 'NJ': 'New Jersey', 'AZ': 'Arizona', 'GU': 'Guam', 'MS': 'Mississippi', 'PR': 'Puerto Rico', 'NC': 'North Carolina', 'TX': 'Texas', 'SD': 'South Dakota', 'MP': 'Northern Mariana Islands', 'IA': 'Iowa', 'MO': 'Missouri', 'CT': 'Connecticut', 'WV': 'West Virginia', 'SC': 'South Carolina', 'LA': 'Louisiana', 'KS': 'Kansas', 'NY': 'New York', 'NE': 'Nebraska', 'OK': 'Oklahoma', 'FL': 'Florida', 'CA': 'California', 'CO': 'Colorado', 'PA': 'Pennsylvania', 'DE': 'Delaware', 'NM': 'New Mexico', 'RI': 'Rhode Island', 'MN': 'Minnesota', 'VI': 'Virgin Islands', 'NH': 'New Hampshire', 'MA': 'Massachusetts', 'GA': 'Georgia', 'ND': 'North Dakota', 'VA': 'Virginia'}

``````
``````

In [4]:

def get_list_of_university_towns():
'''Returns a DataFrame of towns and the states they are in from the
university_towns.txt list. The format of the DataFrame should be:
DataFrame( [ ["Michigan", "Ann Arbor"], ["Michigan", "Yipsilanti"] ],
columns=["State", "RegionName"]  )

The following cleaning needs to be done:

1. For "State", removing characters from "[" to the end.
2. For "RegionName", when applicable, removing every character from " (" to the end.
3. Depending on how you read the data, you may need to remove newline character '\n'. '''

region_state_list = []
with open("university_towns.txt") as fh:
region_name = ""
for line in fh:
region_state = []
if "" in line:
index_val = line.index("[")
region_name = line[:index_val]
else:
if line.count("(") > 0:
region_state = [region_name, line[:line.index("(")].strip()]
else:
region_state = [region_name, line.strip()]
region_state_list.append(region_state)

labels = ["State", "RegionName"]
df = pd.DataFrame.from_records(region_state_list, columns=labels)
return df
get_list_of_university_towns()

``````
``````

Out[4]:

State
RegionName

0
Alabama
Auburn

1
Alabama
Florence

2
Alabama
Jacksonville

3
Alabama
Livingston

4
Alabama
Montevallo

5
Alabama
Troy

6
Alabama
Tuscaloosa

7
Alabama
Tuskegee

8
Fairbanks

9
Arizona
Flagstaff

10
Arizona
Tempe

11
Arizona
Tucson

12
Arkansas

13
Arkansas
Conway

14
Arkansas
Fayetteville

15
Arkansas
Jonesboro

16
Arkansas
Magnolia

17
Arkansas
Monticello

18
Arkansas
Russellville

19
Arkansas
Searcy

20
California
Angwin

21
California
Arcata

22
California
Berkeley

23
California
Chico

24
California
Claremont

25
California
Cotati

26
California
Davis

27
California
Irvine

28
California
Isla Vista

29
California
University Park, Los Angeles

...
...
...

487
Virginia
Wise

488
Virginia
Chesapeake

489
Washington
Bellingham

490
Washington
Cheney

491
Washington
Ellensburg

492
Washington
Pullman

493
Washington
University District, Seattle

494
West Virginia
Athens

495
West Virginia
Buckhannon

496
West Virginia
Fairmont

497
West Virginia
Glenville

498
West Virginia
Huntington

499
West Virginia
Montgomery

500
West Virginia
Morgantown

501
West Virginia
Shepherdstown

502
West Virginia
West Liberty

503
Wisconsin
Appleton

504
Wisconsin
Eau Claire

505
Wisconsin
Green Bay

506
Wisconsin
La Crosse

507
Wisconsin

508
Wisconsin
Menomonie

509
Wisconsin
Milwaukee

510
Wisconsin
Oshkosh

511
Wisconsin
Platteville

512
Wisconsin
River Falls

513
Wisconsin
Stevens Point

514
Wisconsin
Waukesha

515
Wisconsin
Whitewater

516
Wyoming
Laramie

517 rows × 2 columns

``````
``````

In [5]:

def get_recession_start():
'''Returns the year and quarter of the recession start time as a
string value in a format such as 2005q3'''
df = pd.read_excel("gdplev.xls", header=None, skiprows=220, names=["Quarter", "GDP"], parse_cols="E,G")
for i in range(0, len(df)-2):
if df.loc[i]["GDP"] > df.loc[i+1]["GDP"] and df.loc[i+1]["GDP"] > df.loc[i+2]["GDP"]:
return df.loc[i+1]["Quarter"]
get_recession_start()

``````
``````

Out[5]:

'2008q3'

``````
``````

In [6]:

def get_recession_end():
'''Returns the year and quarter of the recession end time as a
string value in a format such as 2005q3'''
df = pd.read_excel("gdplev.xls", header=None, skiprows=220, names=["Quarter", "GDP"], parse_cols="E,G")
recession_start = None
for i in range(0, len(df)-2):
if df.loc[i]["GDP"] > df.loc[i+1]["GDP"] and df.loc[i+1]["GDP"] > df.loc[i+2]["GDP"]:
recession_start = i+1

for i in range(recession_start+2, len(df)-2):
if df.loc[i]["GDP"] < df.loc[i+1]["GDP"] and df.loc[i+1]["GDP"] < df.loc[i+2]["GDP"]:
return df.loc[i+1]["Quarter"]
get_recession_end()

``````
``````

Out[6]:

'2009q4'

``````
``````

In [7]:

def get_recession_bottom():
'''Returns the year and quarter of the recession bottom time as a
string value in a format such as 2005q3'''
df = pd.read_excel("gdplev.xls", header=None, skiprows=220, names=["Quarter", "GDP"], parse_cols="E,G")
recession_start = None
for i in range(0, len(df)-2):
if df.loc[i]["GDP"] > df.loc[i+1]["GDP"] and df.loc[i+1]["GDP"] > df.loc[i+2]["GDP"]:
recession_start = i+1
break

recession_end = None
for i in range(recession_start+2, len(df)-2):
if df.loc[i]["GDP"] < df.loc[i+1]["GDP"] and df.loc[i+1]["GDP"] < df.loc[i+2]["GDP"]:
recession_end = i+1
break

df = df.loc[recession_start:recession_end+1, ]
return df.loc[df["GDP"].idxmin()]["Quarter"]
get_recession_bottom()

``````
``````

Out[7]:

'2009q2'

``````
``````

In [8]:

def convert_housing_data_to_quarters():
global states
'''Converts the housing data to quarters and returns it as mean
values in a dataframe. This dataframe should be a dataframe with
columns for 2000q1 through 2016q3, and should have a multi-index
in the shape of ["State","RegionName"].

Note: Quarters are defined in the assignment description, they are
not arbitrary three month periods.

The resulting dataframe should have 67 columns, and 10,730 rows.
'''
new_df = pd.DataFrame()
mean_df = pd.DataFrame()

column_name_list = df.columns.values.tolist()
for col_index in range(6,len(df.columns)+1,3):
new_df[col_index] = df[column_name_list[col_index]]
new_df[col_index+1] = df[column_name_list[col_index+1]]
if col_index+2 < len(column_name_list):
new_df[col_index+2] = df[column_name_list[col_index+2]]
mean_df["new_col"+str(col_index)] = new_df.ix[:,col_index:col_index+2].mean(axis=1)
else:
mean_df["new_col"+str(col_index)] = new_df.ix[:,col_index:col_index+1].mean(axis=1)

column_names = [col.split("-")[0] for col in df.columns if len(col.split("-")) > 1 ]
final_names = []
for name_index in range(0, len(column_names),3):
if name_index % 4 == 0:
final_names.append(column_names[name_index]+"q2")
elif name_index % 4 == 1:
final_names.append(column_names[name_index]+"q1")
elif name_index % 4 == 2:
final_names.append(column_names[name_index]+"q4")
else:
final_names.append(column_names[name_index]+"q3")

mean_df.columns = final_names
mean_df = mean_df.ix[:,15:]
for new_col in column_name_list[1:3]:
mean_df[new_col] = df[new_col]

mean_df["State"] = mean_df["State"].apply(lambda x:states[x])
mean_df = mean_df.set_index(["State","RegionName"])
return mean_df
convert_housing_data_to_quarters()

``````
``````

Out[8]:

2000q1
2000q2
2000q3
2000q4
2001q1
2001q2
2001q3
2001q4
2002q1
2002q2
...
2014q2
2014q3
2014q4
2015q1
2015q2
2015q3
2015q4
2016q1
2016q2
2016q3

State
RegionName

New York
New York
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
5.154667e+05
5.228000e+05
5.280667e+05
5.322667e+05
5.408000e+05
5.572000e+05
5.728333e+05
5.828667e+05
5.916333e+05
587200.0

California
Los Angeles
2.070667e+05
2.144667e+05
2.209667e+05
2.261667e+05
2.330000e+05
2.391000e+05
2.450667e+05
2.530333e+05
2.619667e+05
2.727000e+05
...
4.980333e+05
5.090667e+05
5.188667e+05
5.288000e+05
5.381667e+05
5.472667e+05
5.577333e+05
5.660333e+05
5.774667e+05
584050.0

Illinois
Chicago
1.384000e+05
1.436333e+05
1.478667e+05
1.521333e+05
1.569333e+05
1.618000e+05
1.664000e+05
1.704333e+05
1.755000e+05
1.775667e+05
...
1.926333e+05
1.957667e+05
2.012667e+05
2.010667e+05
2.060333e+05
2.083000e+05
2.079000e+05
2.060667e+05
2.082000e+05
212000.0

Pennsylvania
5.300000e+04
5.363333e+04
5.413333e+04
5.470000e+04
5.533333e+04
5.553333e+04
5.626667e+04
5.753333e+04
5.913333e+04
6.073333e+04
...
1.137333e+05
1.153000e+05
1.156667e+05
1.162000e+05
1.179667e+05
1.212333e+05
1.222000e+05
1.234333e+05
1.269333e+05
128700.0

Arizona
Phoenix
1.118333e+05
1.143667e+05
1.160000e+05
1.174000e+05
1.196000e+05
1.215667e+05
1.227000e+05
1.243000e+05
1.265333e+05
1.283667e+05
...
1.642667e+05
1.653667e+05
1.685000e+05
1.715333e+05
1.741667e+05
1.790667e+05
1.838333e+05
1.879000e+05
1.914333e+05
195200.0

Las Vegas
1.326000e+05
1.343667e+05
1.354000e+05
1.370000e+05
1.395333e+05
1.417333e+05
1.433667e+05
1.461333e+05
1.493333e+05
1.509333e+05
...
1.700667e+05
1.734000e+05
1.754667e+05
1.775000e+05
1.816000e+05
1.867667e+05
1.906333e+05
1.946000e+05
1.972000e+05
199950.0

California
San Diego
2.229000e+05
2.343667e+05
2.454333e+05
2.560333e+05
2.672000e+05
2.762667e+05
2.845000e+05
2.919333e+05
3.012333e+05
3.128667e+05
...
4.802000e+05
4.890333e+05
4.964333e+05
5.033667e+05
5.120667e+05
5.197667e+05
5.254667e+05
5.293333e+05
5.362333e+05
539750.0

Texas
Dallas
8.446667e+04
8.386667e+04
8.486667e+04
8.783333e+04
8.973333e+04
8.930000e+04
8.906667e+04
9.090000e+04
9.256667e+04
9.380000e+04
...
1.066333e+05
1.089000e+05
1.115333e+05
1.137000e+05
1.211333e+05
1.285667e+05
1.346000e+05
1.405000e+05
1.446000e+05
149300.0

California
San Jose
3.742667e+05
4.065667e+05
4.318667e+05
4.555000e+05
4.706667e+05
4.702000e+05
4.568000e+05
4.455667e+05
4.414333e+05
4.577667e+05
...
6.794000e+05
6.970333e+05
7.149333e+05
7.314333e+05
7.567333e+05
7.764000e+05
7.891333e+05
8.036000e+05
8.189333e+05
822200.0

Florida
Jacksonville
8.860000e+04
8.970000e+04
9.170000e+04
9.310000e+04
9.440000e+04
9.560000e+04
9.706667e+04
9.906667e+04
1.012333e+05
1.034333e+05
...
1.207667e+05
1.217333e+05
1.231667e+05
1.241667e+05
1.269000e+05
1.301333e+05
1.320000e+05
1.339667e+05
1.372000e+05
139900.0

California
San Francisco
4.305000e+05
4.644667e+05
4.835333e+05
4.930000e+05
4.940667e+05
4.961333e+05
5.041000e+05
5.134000e+05
5.204333e+05
5.381667e+05
...
9.269333e+05
9.545333e+05
9.687667e+05
1.000733e+06
1.060800e+06
1.095100e+06
1.105467e+06
1.121767e+06
1.119267e+06
1106400.0

Texas
Austin
1.429667e+05
1.452667e+05
1.494667e+05
1.557333e+05
1.612333e+05
1.607333e+05
1.595333e+05
1.600333e+05
1.589667e+05
1.575000e+05
...
2.488667e+05
2.528000e+05
2.581333e+05
2.665000e+05
2.750333e+05
2.816333e+05
2.872333e+05
2.935000e+05
3.014333e+05
304450.0

Michigan
Detroit
6.616667e+04
6.830000e+04
6.676667e+04
6.703333e+04
6.750000e+04
6.836667e+04
6.926667e+04
6.996667e+04
7.100000e+04
7.233333e+04
...
3.730000e+04
3.710000e+04
3.713333e+04
3.620000e+04
3.583333e+04
3.706667e+04
3.836667e+04
3.796667e+04
3.746667e+04
37900.0

Ohio
Columbus
9.436667e+04
9.583333e+04
9.713333e+04
9.826667e+04
9.940000e+04
1.002667e+05
1.010667e+05
1.022000e+05
1.034000e+05
1.048000e+05
...
1.031333e+05
1.045000e+05
1.064333e+05
1.078667e+05
1.094333e+05
1.115667e+05
1.150000e+05
1.167000e+05
1.182000e+05
120100.0

Tennessee
Memphis
7.250000e+04
7.320000e+04
7.386667e+04
7.400000e+04
7.416667e+04
7.493333e+04
7.550000e+04
7.606667e+04
7.633333e+04
7.676667e+04
...
6.810000e+04
6.910000e+04
7.116667e+04
7.053333e+04
6.870000e+04
6.866667e+04
6.953333e+04
7.090000e+04
7.416667e+04
75900.0

North Carolina
Charlotte
1.269333e+05
1.283667e+05
1.302000e+05
1.315667e+05
1.329333e+05
1.332000e+05
1.328000e+05
1.331000e+05
1.343667e+05
1.353667e+05
...
1.494667e+05
1.506333e+05
1.527333e+05
1.551667e+05
1.579000e+05
1.601667e+05
1.628667e+05
1.664667e+05
1.694333e+05
172400.0

Texas
El Paso
7.626667e+04
7.686667e+04
7.673333e+04
7.730000e+04
7.823333e+04
7.830000e+04
7.743333e+04
7.680000e+04
7.660000e+04
7.640000e+04
...
1.118000e+05
1.117333e+05
1.117667e+05
1.115000e+05
1.113000e+05
1.110667e+05
1.102667e+05
1.106667e+05
1.114667e+05
112200.0

Massachusetts
Boston
2.069333e+05
2.191667e+05
2.331000e+05
2.425000e+05
2.496000e+05
2.570667e+05
2.669333e+05
2.749667e+05
2.825000e+05
2.893000e+05
...
4.266667e+05
4.314333e+05
4.407333e+05
4.485000e+05
4.553667e+05
4.639667e+05
4.716333e+05
4.826000e+05
4.903667e+05
501700.0

Washington
Seattle
2.486000e+05
2.556000e+05
2.625333e+05
2.674000e+05
2.710000e+05
2.724333e+05
2.741667e+05
2.781667e+05
2.805000e+05
2.846000e+05
...
4.418000e+05
4.515000e+05
4.591667e+05
4.679333e+05
4.933667e+05
5.142667e+05
5.334667e+05
5.517333e+05
5.755333e+05
589700.0

Maryland
Baltimore
5.966667e+04
5.950000e+04
5.883333e+04
5.950000e+04
5.956667e+04
6.013333e+04
6.210000e+04
6.340000e+04
6.366667e+04
6.490000e+04
...
1.092333e+05
1.095333e+05
1.073667e+05
1.080667e+05
1.114333e+05
1.139667e+05
1.139000e+05
1.146667e+05
1.147333e+05
115150.0

Denver
1.622333e+05
1.678333e+05
1.743333e+05
1.803333e+05
1.865000e+05
1.925333e+05
1.964000e+05
1.991000e+05
2.012333e+05
2.024333e+05
...
2.708667e+05
2.775000e+05
2.872333e+05
2.976333e+05
3.103667e+05
3.205000e+05
3.301000e+05
3.355667e+05
3.427667e+05
351550.0

District of Columbia
Washington
1.377667e+05
1.442000e+05
1.487000e+05
1.477000e+05
1.497667e+05
1.551333e+05
1.646333e+05
1.725333e+05
1.805000e+05
1.933000e+05
...
4.469333e+05
4.530000e+05
4.603000e+05
4.661667e+05
4.810667e+05
4.934000e+05
5.009000e+05
5.041000e+05
5.058000e+05
516250.0

Tennessee
Nashville
1.138333e+05
1.152667e+05
1.158667e+05
1.169333e+05
1.180333e+05
1.191667e+05
1.201000e+05
1.208000e+05
1.215667e+05
1.226333e+05
...
1.607000e+05
1.623000e+05
1.669000e+05
1.714667e+05
1.762667e+05
1.818000e+05
1.892000e+05
1.950667e+05
2.003667e+05
206100.0

Wisconsin
Milwaukee
7.803333e+04
7.906667e+04
8.103333e+04
8.233333e+04
8.403333e+04
8.556667e+04
8.706667e+04
8.840000e+04
8.953333e+04
9.136667e+04
...
9.216667e+04
9.216667e+04
9.196667e+04
9.333333e+04
9.410000e+04
9.413333e+04
9.456667e+04
9.466667e+04
9.636667e+04
98850.0

Arizona
Tucson
1.018333e+05
1.029667e+05
1.044667e+05
1.056667e+05
1.072000e+05
1.087667e+05
1.105667e+05
1.128000e+05
1.150000e+05
1.172000e+05
...
1.424667e+05
1.434333e+05
1.442333e+05
1.441667e+05
1.451333e+05
1.466000e+05
1.481667e+05
1.495333e+05
1.511667e+05
152700.0

Oregon
Portland
1.528000e+05
1.547667e+05
1.565667e+05
1.574667e+05
1.599000e+05
1.618000e+05
1.642667e+05
1.677667e+05
1.707667e+05
1.741333e+05
...
2.822333e+05
2.872667e+05
2.955333e+05
3.019333e+05
3.119000e+05
3.257333e+05
3.430667e+05
3.560000e+05
3.698000e+05
387050.0

Oklahoma
Oklahoma City
7.643333e+04
7.750000e+04
7.856667e+04
7.916667e+04
7.983333e+04
8.040000e+04
8.113333e+04
8.173333e+04
8.260000e+04
8.343333e+04
...
1.180333e+05
1.189667e+05
1.201000e+05
1.208000e+05
1.223667e+05
1.247000e+05
1.271000e+05
1.279000e+05
1.293000e+05
130300.0

Omaha
1.128000e+05
1.141000e+05
1.167333e+05
1.189000e+05
1.208667e+05
1.197667e+05
1.178667e+05
1.174000e+05
1.180667e+05
1.176333e+05
...
1.301000e+05
1.303000e+05
1.325000e+05
1.330667e+05
1.344667e+05
1.367333e+05
1.400667e+05
1.416333e+05
1.426667e+05
143450.0

New Mexico
Albuquerque
1.258667e+05
1.267000e+05
1.264333e+05
1.267333e+05
1.271000e+05
1.277333e+05
1.285667e+05
1.299000e+05
1.310667e+05
1.321000e+05
...
1.632667e+05
1.640000e+05
1.648000e+05
1.651667e+05
1.659000e+05
1.665333e+05
1.673333e+05
1.691000e+05
1.706333e+05
171900.0

California
Fresno
9.410000e+04
9.526667e+04
9.646667e+04
9.823333e+04
1.005667e+05
1.035667e+05
1.072333e+05
1.103000e+05
1.140333e+05
1.185333e+05
...
1.696333e+05
1.736000e+05
1.781333e+05
1.804667e+05
1.820333e+05
1.857000e+05
1.874667e+05
1.890333e+05
1.927333e+05
196450.0

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

Texas
Granite Shoals
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
1.169667e+05
1.175333e+05
1.175333e+05
1.171667e+05
1.191000e+05
1.216000e+05
1.280000e+05
1.337667e+05
1.400667e+05
146450.0

Maryland
Piney Point
1.556667e+05
1.551667e+05
1.584667e+05
1.637000e+05
1.634000e+05
1.648333e+05
1.647000e+05
1.679000e+05
1.782667e+05
1.812000e+05
...
2.964000e+05
3.090000e+05
3.092333e+05
3.095667e+05
3.017000e+05
3.052333e+05
3.099667e+05
3.195000e+05
3.241667e+05
324600.0

Wisconsin
Maribel
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
1.306000e+05
1.289667e+05
1.296333e+05
1.312667e+05
1.301333e+05
1.297333e+05
1.293000e+05
1.278333e+05
1.292667e+05
134200.0

Idaho
Middleton
1.060667e+05
1.043333e+05
1.019000e+05
1.041667e+05
1.061667e+05
1.083667e+05
1.110333e+05
1.112333e+05
1.141000e+05
1.141667e+05
...
1.443667e+05
1.457000e+05
1.462333e+05
1.461667e+05
1.477333e+05
1.482000e+05
1.511333e+05
1.539000e+05
1.571667e+05
160750.0

Bennett
1.329000e+05
1.358333e+05
1.398000e+05
1.446667e+05
1.483000e+05
1.521000e+05
1.542333e+05
1.562000e+05
1.587333e+05
1.606333e+05
...
1.514667e+05
1.620667e+05
1.714000e+05
1.780333e+05
1.844333e+05
1.916667e+05
1.958000e+05
1.997667e+05
2.074667e+05
212600.0

New Hampshire
1.618333e+05
1.691000e+05
1.739667e+05
1.805000e+05
1.909000e+05
1.950667e+05
1.992667e+05
2.074000e+05
2.123000e+05
2.122333e+05
...
2.495000e+05
2.521000e+05
2.557333e+05
2.587333e+05
2.613667e+05
2.616000e+05
2.688000e+05
2.725333e+05
2.778000e+05
282450.0

Missouri
Garden City
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
1.055000e+05
1.043000e+05
1.047667e+05
1.060333e+05
9.606667e+04
9.930000e+04
1.034333e+05
1.062667e+05
1.116667e+05
113600.0

Arkansas
Mountainburg
5.716667e+04
6.433333e+04
6.783333e+04
6.900000e+04
6.866667e+04
6.386667e+04
6.376667e+04
6.546667e+04
6.533333e+04
6.600000e+04
...
8.160000e+04
8.506667e+04
8.846667e+04
8.903333e+04
8.556667e+04
8.370000e+04
9.043333e+04
9.833333e+04
1.019000e+05
103400.0

Wisconsin
Oostburg
1.072667e+05
1.081000e+05
1.124333e+05
1.155000e+05
1.191000e+05
1.204333e+05
1.203667e+05
1.196333e+05
1.198667e+05
1.185667e+05
...
1.295667e+05
1.279333e+05
1.274333e+05
1.270667e+05
1.274000e+05
1.303333e+05
1.320333e+05
1.327667e+05
1.341000e+05
136350.0

California
Twin Peaks
9.736667e+04
1.001667e+05
1.013333e+05
1.017000e+05
1.040000e+05
1.076667e+05
1.098333e+05
1.111333e+05
1.132000e+05
1.166000e+05
...
1.501000e+05
1.475333e+05
1.460667e+05
1.435000e+05
1.523000e+05
1.552667e+05
1.591667e+05
1.641667e+05
1.679667e+05
173500.0

New York
Upper Brookville
1.230967e+06
1.230967e+06
1.237700e+06
1.261567e+06
1.295167e+06
1.340033e+06
1.403667e+06
1.481933e+06
1.536167e+06
1.562033e+06
...
1.780633e+06
1.749233e+06
1.729467e+06
1.749867e+06
1.789600e+06
1.777267e+06
1.834367e+06
1.904500e+06
1.944067e+06
1968800.0

Hawaii
Volcano
9.870000e+04
1.053667e+05
1.146667e+05
1.247667e+05
1.181333e+05
1.194000e+05
1.232667e+05
1.211667e+05
1.233000e+05
1.169000e+05
...
2.064667e+05
2.276333e+05
2.332000e+05
2.346333e+05
2.323667e+05
2.249667e+05
2.324333e+05
2.420667e+05
2.489667e+05
247850.0

South Carolina
Wedgefield
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
7.436667e+04
7.026667e+04
7.206667e+04
7.570000e+04
7.206667e+04
7.033333e+04
6.903333e+04
6.886667e+04
7.426667e+04
80700.0

Michigan
Williamston
1.591667e+05
1.613000e+05
1.643000e+05
1.662000e+05
1.664333e+05
1.686333e+05
1.716667e+05
1.750333e+05
1.786667e+05
1.793333e+05
...
1.657000e+05
1.689333e+05
1.708667e+05
1.744333e+05
1.758667e+05
1.794667e+05
1.823000e+05
1.814667e+05
1.824000e+05
183000.0

Arkansas
Decatur
6.360000e+04
6.440000e+04
6.566667e+04
6.673333e+04
6.720000e+04
6.770000e+04
6.650000e+04
6.540000e+04
6.460000e+04
6.490000e+04
...
8.966667e+04
9.256667e+04
9.470000e+04
9.350000e+04
9.490000e+04
9.543333e+04
9.700000e+04
9.650000e+04
9.663333e+04
96850.0

Tennessee
Briceville
4.000000e+04
4.173333e+04
4.366667e+04
4.490000e+04
4.480000e+04
4.530000e+04
4.463333e+04
4.370000e+04
4.446667e+04
4.340000e+04
...
5.623333e+04
5.423333e+04
5.260000e+04
4.963333e+04
4.590000e+04
4.793333e+04
4.360000e+04
4.080000e+04
4.180000e+04
40850.0

Indiana
Edgewood
9.170000e+04
9.186667e+04
9.293333e+04
9.490000e+04
9.893333e+04
1.000667e+05
1.008333e+05
1.010000e+05
1.021667e+05
1.017667e+05
...
9.213333e+04
9.406667e+04
9.466667e+04
9.586667e+04
9.433333e+04
9.663333e+04
9.996667e+04
9.943333e+04
9.996667e+04
100950.0

Tennessee
Palmyra
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
1.227667e+05
1.269333e+05
1.262333e+05
1.223000e+05
1.204667e+05
1.198000e+05
1.258000e+05
1.276667e+05
1.328667e+05
137750.0

Maryland
Saint Inigoes
1.480667e+05
1.476000e+05
1.572333e+05
1.633667e+05
1.642333e+05
1.682000e+05
1.665000e+05
1.653333e+05
1.673000e+05
1.688000e+05
...
2.822333e+05
2.884333e+05
2.869667e+05
2.847000e+05
2.807667e+05
2.778333e+05
2.768333e+05
2.793333e+05
2.826333e+05
281400.0

Indiana
Marysville
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
...
1.166000e+05
1.151000e+05
1.165000e+05
1.118667e+05
1.118000e+05
1.156667e+05
1.201667e+05
1.282333e+05
1.232333e+05
124200.0

California
Forest Falls
1.135333e+05
1.144000e+05
1.141667e+05
1.111333e+05
1.134333e+05
1.130000e+05
1.130333e+05
1.151667e+05
1.187000e+05
1.250667e+05
...
1.653667e+05
1.675000e+05
1.771000e+05
1.765333e+05
1.818000e+05
1.911667e+05
1.987333e+05
1.886333e+05
1.898667e+05
186650.0

Missouri
Bois D Arc
1.078000e+05
1.069667e+05
1.071000e+05
1.081000e+05
1.107000e+05
1.136667e+05
1.126333e+05
1.127333e+05
1.130667e+05
1.154000e+05
...
1.375667e+05
1.375667e+05
1.404000e+05
1.450333e+05
1.475667e+05
1.463000e+05
1.494333e+05
1.468667e+05
1.437667e+05
144000.0

Virginia
Henrico
1.285667e+05
1.307667e+05
1.322667e+05
1.332667e+05
1.352333e+05
1.367333e+05
1.386000e+05
1.413333e+05
1.435333e+05
1.461333e+05
...
2.016333e+05
2.040000e+05
2.059000e+05
2.065667e+05
2.104333e+05
2.121000e+05
2.139667e+05
2.160333e+05
2.162000e+05
220150.0

New Jersey
Diamond Beach
1.739667e+05
1.831000e+05
1.889667e+05
1.931333e+05
1.944000e+05
2.102667e+05
2.302667e+05
2.486667e+05
2.599333e+05
2.656333e+05
...
3.818000e+05
3.878667e+05
3.876667e+05
3.931667e+05
3.980000e+05
3.992333e+05
4.004333e+05
4.045333e+05
4.039000e+05
399000.0

Tennessee
Gruetli Laager
3.540000e+04
3.546667e+04
3.666667e+04
3.730000e+04
3.773333e+04
3.790000e+04
3.936667e+04
4.040000e+04
4.156667e+04
4.163333e+04
...
5.556667e+04
5.636667e+04
5.713333e+04
5.890000e+04
6.536667e+04
6.950000e+04
7.170000e+04
7.533333e+04
7.646667e+04
77500.0

Wisconsin
Town of Wrightstown
1.017667e+05
1.054000e+05
1.113667e+05
1.148667e+05
1.259667e+05
1.299000e+05
1.299000e+05
1.294333e+05
1.319000e+05
1.342000e+05
...
1.448667e+05
1.468667e+05
1.492333e+05
1.486667e+05
1.493333e+05
1.498667e+05
1.499333e+05
1.498333e+05
1.512667e+05
155000.0

New York
Urbana
7.920000e+04
8.166667e+04
9.170000e+04
9.836667e+04
9.486667e+04
9.853333e+04
1.029667e+05
9.803333e+04
9.396667e+04
9.460000e+04
...
1.321333e+05
1.370333e+05
1.400667e+05
1.417000e+05
1.378667e+05
1.364667e+05
1.361667e+05
1.389667e+05
1.442000e+05
143000.0

Wisconsin
New Denmark
1.145667e+05
1.192667e+05
1.260667e+05
1.319667e+05
1.438000e+05
1.469667e+05
1.483667e+05
1.491667e+05
1.531333e+05
1.567333e+05
...
1.745667e+05
1.811667e+05
1.861667e+05
1.876000e+05
1.886667e+05
1.884333e+05
1.889333e+05
1.910667e+05
1.928333e+05
197600.0

California
Angels
1.510000e+05
1.559000e+05
1.581000e+05
1.674667e+05
1.768333e+05
1.837667e+05
1.902333e+05
1.845667e+05
1.840333e+05
1.861333e+05
...
2.444667e+05
2.540667e+05
2.599333e+05
2.601000e+05
2.506333e+05
2.635000e+05
2.795000e+05
2.765333e+05
2.716000e+05
269950.0

Wisconsin
Holland
1.510333e+05
1.505000e+05
1.532333e+05
1.558333e+05
1.618667e+05
1.657333e+05
1.680333e+05
1.674000e+05
1.657667e+05
1.619667e+05
...
2.012667e+05
2.015667e+05
2.012667e+05
2.060000e+05
2.076000e+05
2.128667e+05
2.178333e+05
2.219667e+05
2.280333e+05
234950.0

10730 rows × 67 columns

``````
``````

In [13]:

def run_ttest():
'''First creates new data showing the decline or growth of housing prices
between the recession start and the recession bottom. Then runs a ttest
comparing the university town values to the non-university towns values,
return whether the alternative hypothesis (that the two groups are the same)
is true or not as well as the p-value of the confidence.

Return the tuple (different, p, better) where different=True if the t-test is
True at a p<0.01 (we reject the null hypothesis), or different=False if
otherwise (we cannot reject the null hypothesis). The variable p should
be equal to the exact p value returned from scipy.stats.ttest_ind(). The
value for better should be either "university town" or "non-university town"
depending on which has a lower mean price ratio (which is equivilent to a
reduced market loss).'''

df = convert_housing_data_to_quarters()
recession_bottom = get_recession_bottom()
recession_start = get_recession_start()
col = df.columns.values.tolist()
start_index = col.index(recession_start)
finish_index = col.index(recession_bottom)

df = df.ix[:,start_index:finish_index+1]
col = col[start_index:finish_index+1]
col.extend(["State", "RegionName"])
university_df = get_list_of_university_towns()
uni_df = pd.DataFrame(columns=col)

for row in university_df.iterrows():
try:
list_row = df.loc[row[1].State].loc[row[1].RegionName].tolist()
list_row.extend([row[1].State, row[1].RegionName])
uni_df.loc[int(row[0])] = list_row
except Exception as e:
pass
uni_df = uni_df.set_index(["State", "RegionName"])
no_uni_df = df[df.index.map(lambda x:x not in uni_df.index)]
ttest_tuple = ttest_ind(uni_df.mean(), no_uni_df.mean(), nan_policy='omit')
different, better = None, None
if float(ttest_tuple[1]) < 0.01:
different = True
else:
different = False

uni_df = uni_df.mean()
no_uni_df = no_uni_df.mean()
min_uni = uni_df.min()
min_no_uni = no_uni_df.min()
lower_index = [min_uni, min_no_uni].index(min([min_uni, min_no_uni]))
if lower_index == 0:
better = "university town"
else:
better = "non-university town"

return (different, ttest_tuple[1], better)
run_ttest()

``````
``````

Out[13]:

(True, 0.00011220843033507131, 'university town')

``````
``````

In [ ]:

``````