full_backend

debugging the full backend code


In [3]:
%matplotlib inline
%load_ext autoreload
%autoreload 2


The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

import the new haven report card module


In [34]:
import nhrc2
from nhrc2.backend import get_neighborhoods as get_ngbrhd
from nhrc2.backend import read_issues as ri
import pandas as pd
import numpy as np

now determine the root directory for the repo:


In [5]:
nhrc2dir = '/'.join(str(nhrc2.__file__).split('/')[:-1])+'/'

In [ ]:


In [37]:
scf_df_cat = ri.read_categories()

In [38]:
scf_df_cat


Out[38]:
organization title url type
0 City of New Haven Bins for Trash & Recycling https://seeclickfix.com/api/v2/request_types/5743 5743
1 City of New Haven Graffiti https://seeclickfix.com/api/v2/request_types/122 122
2 City of New Haven Hangers https://seeclickfix.com/api/v2/request_types/6215 6215
3 City of New Haven Health Complaints https://seeclickfix.com/api/v2/request_types/5185 5185
4 City of New Haven Illegal Dumping https://seeclickfix.com/api/v2/request_types/1250 1250
5 City of New Haven Other https://seeclickfix.com/api/v2/request_types/374 374
6 City of New Haven Other - city responsibility https://seeclickfix.com/api/v2/request_types/3018 3018
7 City of New Haven Parking Meter https://seeclickfix.com/api/v2/request_types/372 372
8 City of New Haven Parking Violation/Abandoned Auto https://seeclickfix.com/api/v2/request_types/121 121
9 City of New Haven Parks Request https://seeclickfix.com/api/v2/request_types/126 126
10 City of New Haven Policing Issue https://seeclickfix.com/api/v2/request_types/2626 2626
11 City of New Haven Potholes https://seeclickfix.com/api/v2/request_types/116 116
12 City of New Haven Public Space, Streets and Drains https://seeclickfix.com/api/v2/request_types/1249 1249
13 City of New Haven Private Property Issue https://seeclickfix.com/api/v2/request_types/1251 1251
14 City of New Haven Sidewalks and Curb damage https://seeclickfix.com/api/v2/request_types/117 117
15 City of New Haven Signs / Bus Shelters / Pavement Markings https://seeclickfix.com/api/v2/request_types/373 373
16 City of New Haven Street Lamp https://seeclickfix.com/api/v2/request_types/124 124
17 City of New Haven Street Sweeping https://seeclickfix.com/api/v2/request_types/5251 5251
18 City of New Haven Traffic/Road Safety https://seeclickfix.com/api/v2/request_types/2625 2625
19 City of New Haven Traffic Signal / Pedestrian Signal https://seeclickfix.com/api/v2/request_types/51 51
20 City of New Haven Trash & Recycling https://seeclickfix.com/api/v2/request_types/1966 1966
21 City of New Haven Tree Trimming https://seeclickfix.com/api/v2/request_types/1853 1853
22 Neighborhood Volunteers Request for volunteers https://seeclickfix.com/api/v2/request_types/5998 5998
23 CT Transit General Bus Request/Incident https://seeclickfix.com/api/v2/request_types/4947 4947
24 Community Post to Neighbors https://seeclickfix.com/api/v2/request_types/o... other

In [39]:
scf_df_cat[scf_df_cat['organization'] == 'City of New Haven']


Out[39]:
organization title url type
0 City of New Haven Bins for Trash & Recycling https://seeclickfix.com/api/v2/request_types/5743 5743
1 City of New Haven Graffiti https://seeclickfix.com/api/v2/request_types/122 122
2 City of New Haven Hangers https://seeclickfix.com/api/v2/request_types/6215 6215
3 City of New Haven Health Complaints https://seeclickfix.com/api/v2/request_types/5185 5185
4 City of New Haven Illegal Dumping https://seeclickfix.com/api/v2/request_types/1250 1250
5 City of New Haven Other https://seeclickfix.com/api/v2/request_types/374 374
6 City of New Haven Other - city responsibility https://seeclickfix.com/api/v2/request_types/3018 3018
7 City of New Haven Parking Meter https://seeclickfix.com/api/v2/request_types/372 372
8 City of New Haven Parking Violation/Abandoned Auto https://seeclickfix.com/api/v2/request_types/121 121
9 City of New Haven Parks Request https://seeclickfix.com/api/v2/request_types/126 126
10 City of New Haven Policing Issue https://seeclickfix.com/api/v2/request_types/2626 2626
11 City of New Haven Potholes https://seeclickfix.com/api/v2/request_types/116 116
12 City of New Haven Public Space, Streets and Drains https://seeclickfix.com/api/v2/request_types/1249 1249
13 City of New Haven Private Property Issue https://seeclickfix.com/api/v2/request_types/1251 1251
14 City of New Haven Sidewalks and Curb damage https://seeclickfix.com/api/v2/request_types/117 117
15 City of New Haven Signs / Bus Shelters / Pavement Markings https://seeclickfix.com/api/v2/request_types/373 373
16 City of New Haven Street Lamp https://seeclickfix.com/api/v2/request_types/124 124
17 City of New Haven Street Sweeping https://seeclickfix.com/api/v2/request_types/5251 5251
18 City of New Haven Traffic/Road Safety https://seeclickfix.com/api/v2/request_types/2625 2625
19 City of New Haven Traffic Signal / Pedestrian Signal https://seeclickfix.com/api/v2/request_types/51 51
20 City of New Haven Trash & Recycling https://seeclickfix.com/api/v2/request_types/1966 1966
21 City of New Haven Tree Trimming https://seeclickfix.com/api/v2/request_types/1853 1853

read in the issue data from file (to speed things up)


In [7]:
readfile=False
writejson=False
scf_df = ri.get_issues(readfile=readfile, writejson=writejson)


Category: Bins for Trash & Recycling, id: 5743, readfile: False
now reading from url...
Category: Graffiti, id: 122, readfile: False
now reading from url...
Category: Hangers, id: 6215, readfile: False
now reading from url...
Category: Health Complaints, id: 5185, readfile: False
now reading from url...
Category: Illegal Dumping, id: 1250, readfile: False
now reading from url...
Category: Other, id: 374, readfile: False
now reading from url...
Category: Other - city responsibility, id: 3018, readfile: False
now reading from url...
Category: Parking Meter, id: 372, readfile: False
now reading from url...
Category: Parking Violation/Abandoned Auto, id: 121, readfile: False
now reading from url...
Category: Parks Request, id: 126, readfile: False
now reading from url...
Category: Policing Issue, id: 2626, readfile: False
now reading from url...
Category: Potholes, id: 116, readfile: False
now reading from url...
Category: Public Space, Streets and Drains, id: 1249, readfile: False
now reading from url...
Category: Private Property Issue, id: 1251, readfile: False
now reading from url...
Category: Sidewalks and Curb damage, id: 117, readfile: False
now reading from url...
Category: Signs / Bus Shelters / Pavement Markings, id: 373, readfile: False
now reading from url...
Category: Street Lamp, id: 124, readfile: False
now reading from url...
Category: Street Sweeping, id: 5251, readfile: False
now reading from url...
Category: Traffic/Road Safety, id: 2625, readfile: False
now reading from url...
Category: Traffic Signal / Pedestrian Signal, id: 51, readfile: False
now reading from url...
Category: Trash & Recycling, id: 1966, readfile: False
now reading from url...
Category: Tree Trimming, id: 1853, readfile: False
now reading from url...
Category: Request for volunteers, id: 5998, readfile: False
now reading from url...
Category: General Bus Request/Incident, id: 4947, readfile: False
now reading from url...
Category: Post to Neighbors, id: other, readfile: False
now reading from url...

now determine the neighborhoods for each issue using the get_neighborhoods routine:


In [8]:
hoods = get_ngbrhd.get_neighborhoods()


Category: Bins for Trash & Recycling, id: 5743, readfile: False
now reading from url...
Category: Graffiti, id: 122, readfile: False
now reading from url...
Category: Hangers, id: 6215, readfile: False
now reading from url...
Category: Health Complaints, id: 5185, readfile: False
now reading from url...
Category: Illegal Dumping, id: 1250, readfile: False
now reading from url...
Category: Other, id: 374, readfile: False
now reading from url...
Category: Other - city responsibility, id: 3018, readfile: False
now reading from url...
Category: Parking Meter, id: 372, readfile: False
now reading from url...
Category: Parking Violation/Abandoned Auto, id: 121, readfile: False
now reading from url...
Category: Parks Request, id: 126, readfile: False
now reading from url...
Category: Policing Issue, id: 2626, readfile: False
now reading from url...
Category: Potholes, id: 116, readfile: False
now reading from url...
Category: Public Space, Streets and Drains, id: 1249, readfile: False
now reading from url...
Category: Private Property Issue, id: 1251, readfile: False
now reading from url...
Category: Sidewalks and Curb damage, id: 117, readfile: False
now reading from url...
Category: Signs / Bus Shelters / Pavement Markings, id: 373, readfile: False
now reading from url...
Category: Street Lamp, id: 124, readfile: False
now reading from url...
Category: Street Sweeping, id: 5251, readfile: False
now reading from url...
Category: Traffic/Road Safety, id: 2625, readfile: False
now reading from url...
Category: Traffic Signal / Pedestrian Signal, id: 51, readfile: False
now reading from url...
Category: Trash & Recycling, id: 1966, readfile: False
now reading from url...
Category: Tree Trimming, id: 1853, readfile: False
now reading from url...
Category: Request for volunteers, id: 5998, readfile: False
now reading from url...
Category: General Bus Request/Incident, id: 4947, readfile: False
now reading from url...
Category: Post to Neighbors, id: other, readfile: False
now reading from url...

now add the neighborhoods to the DataFrame:


In [9]:
scf_df['neighborhood'] = hoods

Statistics to calculate:

There are a few things that it would be nice to calculate. Namely,

  • The time to acknowledgement
  • The time to completion
  • Acknowledgement Improvement
  • Completion Improvement

Computing the Time to Acknowledgement


In [10]:
scf_df.columns


Out[10]:
Index([u'acknowledged_at', u'address', u'category', u'closed_at', u'created_at', u'description', u'id', u'issue_id', u'lat', u'lng', u'reporter_id', u'reporter_name', u'reporter_role', u'shortened_url', u'status', u'summary', u'updated_at', u'int_issue_id', u'neighborhood'], dtype='object')

In [18]:
scf_df.loc[0:1, 'created_at']


Out[18]:
0    2015-04-12T14:58:29-04:00
1    2015-03-27T15:57:26-04:00
Name: created_at, dtype: object

In [15]:
pd.to_datetime(scf_df.loc[0, 'created_at'])


Out[15]:
Timestamp('2015-04-12 18:58:29')

In [29]:
scf_df['time_to_acknowledge'] = (pd.to_datetime(scf_df['acknowledged_at']) - pd.to_datetime(scf_df['created_at']))/pd.Timedelta('1d')

In [33]:
scf_df['time_to_close'] = (pd.to_datetime(scf_df['closed_at']) - pd.to_datetime(scf_df['created_at']))/pd.Timedelta('1d')

In [32]:
scf_df.loc[0:1, 'time_to_acknowledge']


Out[32]:
0         NaN
1    2.705278
Name: time_to_acknowledge, dtype: float64

In [35]:
np.median(scf_df['time_to_acknowledge'].values)


Out[35]:
63.65517361111111

In [36]:
np.median(scf_df['time_to_close'].values)


Out[36]:
nan

Computing Progress

In this section I compute the progress made since the previous time period.


In [ ]: