(In order to load the stylesheet of this notebook, execute the last code cell in this notebook)

Clustering the data from the Mayor's 24-hour hotline in Boston

Introduction

In this assignment, we will do a clustering analysis on the 311 calls that we used in the previous homework. We will focus on choosing a value for k, clustering based on multiple features and detecting outliers.



In [32]:

    
%matplotlib inline
import pandas as pd
import seaborn as sns
from matplotlib import pyplot as plt



In [31]:

    
boston = pd.read_csv('../Homework-2/cases_2015.csv')

Step 1. Simple clustering

The first thing that we will try is to cluster the geolocation data of the calls. The geolocation of a record consists of a pair (latitude, longitude), which represents the position of the record on the map. Since the dataset that we have spans only a small geographic location, i.e. Boston, we will use the Euclidean distance when measuring how close two points are. Techinically, we should pick a projection in order to transform the data into 2d, and measure the Euclidean distance in the projected space.

Since we are dealing with numeric data, we might consider centering and normalizing them. In order to do that, check the StandardScaler from scikit-learn.

For exampe, the result of a clustering with 10 clusters is the following:

Filter the dataset to get the latitude and longitude, transform it as required and run the k-means clustering algorithm. Argue about the number of clusters you used. (pts 50)



In [31]:

Step 2. Using multiple features

The next step in our analysis is to use more features than just the geolocation of a record (whose clustering only gives us information about the neighborhoods, which should not be a huge revelation!) Filter the dataset to get more features, transform them as required and compute another clustering. You should argue about the quality of your clustering and use domain knowledge (i.e. the fact that you are locals) to explore the clusters you produced and present them to the reader. (pts 30)



In [ ]:

Step 3. Detecting outliers

Now that we have a clustering of the records in the dataset, we can detect the outlying ones. These are the ones who are the farthest from the centroids of their clusters. Track them down and describe any interesting observations that you can make. (pts 20)



In [ ]:



In [21]:

    
# Code for setting the style of the notebook
from IPython.core.display import HTML
def css_styling():
    styles = open("../../theme/custom.css", "r").read()
    return HTML(styles)
css_styling()









    Out[21]: