Plotting and statistical packages

Matplotlib / Seaborn
pandas
statsmodels



In [1]:

    
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns

import pandas as pd
import numpy as np

from sklearn.datasets import load_boston

import statsmodels.api as sm
from statsmodels.formula.api import ols

sns.set()









    



/Users/jbw/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))

Boston housing prices



In [2]:

    
# scikit-learn dataset
df_dict = load_boston()
print df_dict["DESCR"]









    



Boston House Prices dataset

Notes
------
Data Set Characteristics:  

    :Number of Instances: 506 

    :Number of Attributes: 13 numeric/categorical predictive
    
    :Median Value (attribute 14) is usually the target

    :Attribute Information (in order):
        - CRIM     per capita crime rate by town
        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
        - INDUS    proportion of non-retail business acres per town
        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
        - NOX      nitric oxides concentration (parts per 10 million)
        - RM       average number of rooms per dwelling
        - AGE      proportion of owner-occupied units built prior to 1940
        - DIS      weighted distances to five Boston employment centres
        - RAD      index of accessibility to radial highways
        - TAX      full-value property-tax rate per $10,000
        - PTRATIO  pupil-teacher ratio by town
        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
        - LSTAT    % lower status of the population
        - MEDV     Median value of owner-occupied homes in $1000's

    :Missing Attribute Values: None

    :Creator: Harrison, D. and Rubinfeld, D.L.

This is a copy of UCI ML housing dataset.
http://archive.ics.uci.edu/ml/datasets/Housing


This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
prices and the demand for clean air', J. Environ. Economics & Management,
vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
...', Wiley, 1980.   N.B. Various transformations are used in the table on
pages 244-261 of the latter.

The Boston house-price data has been used in many machine learning papers that address regression
problems.   
     
**References**

   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)



In [4]:

    
features = pd.DataFrame(data=df_dict.data, columns = df_dict.feature_names)
target = pd.DataFrame(data=df_dict.target, columns = ['MEDV'])



In [5]:

    
features.head()



In [6]:

    
target.head()



In [10]:

    
df = pd.concat([features, target], axis=1)
df.head()



In [11]:

    
for column in df.columns:
    print column, df[column].nunique()









    



CRIM 504
ZN 26
INDUS 76
CHAS 2
NOX 81
RM 446
AGE 356
DIS 412
RAD 9
TAX 66
PTRATIO 46
B 357
LSTAT 455
MEDV 229

Pandas plotting



In [12]:

    
from pandas.tools.plotting import scatter_matrix



In [13]:

    
fig, ax = plt.subplots(figsize=(12, 12))
scatter_matrix(df, alpha=0.2, diagonal='kde', ax=ax);









    



/Users/jbw/anaconda/lib/python2.7/site-packages/pandas/tools/plotting.py:3303: UserWarning: To output multiple subplots, the figure containing the passed axes is being cleared
  "is being cleared", UserWarning)



In [15]:

    
df['RAD'].hist()









    Out[15]:





<matplotlib.axes._subplots.AxesSubplot at 0x11cb890d0>



In [16]:

    
df.RAD.hist()









    Out[16]:





<matplotlib.axes._subplots.AxesSubplot at 0x11cedf090>



In [20]:

    
df['RAD_bool'] = df['RAD'].apply(lambda x: x > 15).astype('bool')



In [21]:

    
type(df['RAD_bool'].iloc[0])









    Out[21]:





numpy.bool_



In [22]:

    
df['RAD_bool'].hist()









    Out[22]:





<matplotlib.axes._subplots.AxesSubplot at 0x11332e650>



In [25]:

    
sns.pairplot(df, hue="CHAS", vars=['RM', 'INDUS', 'NOX', 'MEDV'], diag_kind='kde')









    Out[25]:





<seaborn.axisgrid.PairGrid at 0x1212d0c50>



In [27]:

    
fig, ax = plt.subplots(figsize=(10,7))
sns.kdeplot(df.NOX)









    Out[27]:





<matplotlib.axes._subplots.AxesSubplot at 0x123f39610>



In [28]:

    
fig, ax = plt.subplots(figsize=(10,7))
sns.kdeplot(df.RAD_bool, df.NOX, ax=ax)









    Out[28]:





<matplotlib.axes._subplots.AxesSubplot at 0x12402b0d0>

Pandas plotting



In [29]:

    
from pandas.tools.plotting import andrews_curves
# https://en.wikipedia.org/wiki/Andrews_plot



In [33]:

    
andrews_curves(df.sample(100), 'RAD')









    Out[33]:





<matplotlib.axes._subplots.AxesSubplot at 0x112be0d10>



In [ ]:



In [34]:

    
fig, ax = plt.subplots(figsize=(12,8))
sns.kdeplot(target.MEDV, ax=ax, )









    Out[34]:





<matplotlib.axes._subplots.AxesSubplot at 0x11348ab50>



In [35]:

    
fig, ax = plt.subplots(figsize=(12,8))
sns.distplot(target.MEDV, ax=ax, rug=True, hist=False)









    Out[35]:





<matplotlib.axes._subplots.AxesSubplot at 0x11359c390>



In [ ]:

    
features.columns



In [36]:

    
fig, ax = plt.subplots(figsize=(10,7))
sns.kdeplot(df.MEDV, df.LSTAT, ax=ax)









    Out[36]:





<matplotlib.axes._subplots.AxesSubplot at 0x116312b90>



In [ ]:

	CRIM	ZN	INDUS	NOX	RM	AGE	DIS	RAD	TAX	PTRATIO	B	LSTAT
0	0.00632	18	2.31	0.538	6.575	65.2	4.0900	1	296	15.3	396.90	4.98
1	0.02731	0	7.07	0.469	6.421	78.9	4.9671	2	242	17.8	396.90	9.14
2	0.02729	0	7.07	0.469	7.185	61.1	4.9671	2	242	17.8	392.83	4.03
3	0.03237	0	2.18	0.458	6.998	45.8	6.0622	3	222	18.7	394.63	2.94
4	0.06905	0	2.18	0.458	7.147	54.2	6.0622	3	222	18.7	396.90	5.33

	MEDV
0	24.0
1	21.6
2	34.7
3	33.4
4	36.2

	CRIM	ZN	INDUS	NOX	RM	AGE	DIS	RAD	TAX	PTRATIO	B	LSTAT	MEDV
0	0.00632	18	2.31	0.538	6.575	65.2	4.0900	1	296	15.3	396.90	4.98	24.0
1	0.02731	0	7.07	0.469	6.421	78.9	4.9671	2	242	17.8	396.90	9.14	21.6
2	0.02729	0	7.07	0.469	7.185	61.1	4.9671	2	242	17.8	392.83	4.03	34.7
3	0.03237	0	2.18	0.458	6.998	45.8	6.0622	3	222	18.7	394.63	2.94	33.4
4	0.06905	0	2.18	0.458	7.147	54.2	6.0622	3	222	18.7	396.90	5.33	36.2