Postcards of Parliament From A Digital Flâneur

Flâneur, noun, A man who saunters around observing society.

The flâneur wandered in the shopping arcades, but he did not give in to the temptations of consumerism; the arcade was primarily a pathway to a rich sensory experience — and only then a temple of consumption. His goal was to observe, to bathe in the crowd, taking in its noises, its chaos, its heterogeneity, its cosmopolitanism. Occasionally, he would narrate what he saw — surveying both his private self and the world at large — in the form of short essays for daily newspapers.

The Death of the Cyberflâneur, Evgeny Morozov, New York Times, Sunday Review, February 4, 2012

APIs

Using the APIs

Packages Make Life Easier (?)



In [6]:

    
import mnis
import datetime

# Create a date for the analysis
d = datetime.date.today()

# Download the full data for MPs serving on the given date as a list
mnis.getCommonsMembersOn(d)[0]









    Out[6]:





{'@Dods_Id': '25790',
 '@Member_Id': '172',
 '@Pims_Id': '3572',
 'Constituencies': {'Constituency': [{'@Id': '3506',
    'Election': {'@Id': '377',
     'Date': '2017-06-08T00:00:00',
     'Name': '2017 General Election',
     'Type': 'General Election'},
    'EndDate': {'@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
     '@xsi:nil': 'true'},
    'EndReason': None,
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': 'Created at General Election',
    'StartDate': '2017-06-08T00:00:00',
    'SwearInOrder': '32',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '3506',
    'Election': {'@Id': '369',
     'Date': '2015-05-07T00:00:00',
     'Name': '2015 General Election',
     'Type': 'General Election'},
    'EndDate': '2017-05-03T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': 'Created at General Election',
    'StartDate': '2015-05-07T00:00:00',
    'SwearInOrder': None,
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '3506',
    'Election': {'@Id': '19',
     'Date': '2010-05-06T00:00:00',
     'Name': '2010 General Election',
     'Type': 'General Election'},
    'EndDate': '2015-03-30T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '2010-05-06T00:00:00',
    'SwearInOrder': '320',
    'SwornInForename': 'Diane',
    'SwornInMiddleNames': 'Julie',
    'SwornInSurname': 'Abbott',
    'SwornInTitle': None},
   {'@Id': '1333',
    'Election': {'@Id': '17',
     'Date': '2005-05-05T00:00:00',
     'Name': '2005 General Election',
     'Type': 'General Election'},
    'EndDate': '2010-05-06T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '2005-05-05T00:00:00',
    'SwearInOrder': '109',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '1333',
    'Election': {'@Id': '16',
     'Date': '2001-06-07T00:00:00',
     'Name': '2001 General Election',
     'Type': 'General Election'},
    'EndDate': '2005-05-05T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '2001-06-07T00:00:00',
    'SwearInOrder': '153',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '1333',
    'Election': {'@Id': '15',
     'Date': '1997-05-01T00:00:00',
     'Name': '1997 General Election',
     'Type': 'General Election'},
    'EndDate': '2001-06-07T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '1997-05-01T00:00:00',
    'SwearInOrder': '0',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '1332',
    'Election': {'@Id': '14',
     'Date': '1992-04-09T00:00:00',
     'Name': '1992 General Election',
     'Type': 'General Election'},
    'EndDate': '1997-05-01T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'Continuation',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '1992-04-09T00:00:00',
    'SwearInOrder': '0',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None},
   {'@Id': '1332',
    'Election': {'@Id': '13',
     'Date': '1987-06-11T00:00:00',
     'Name': '1987 General Election',
     'Type': 'General Election'},
    'EndDate': '1992-04-09T00:00:00',
    'EndReason': 'General Election',
    'EntryType': 'First entry',
    'Name': 'Hackney North and Stoke Newington',
    'Notes': None,
    'StartDate': '1987-06-11T00:00:00',
    'SwearInOrder': '63',
    'SwornInForename': None,
    'SwornInMiddleNames': None,
    'SwornInSurname': None,
    'SwornInTitle': None}]},
 'CurrentStatus': {'@Id': '0',
  '@IsActive': 'True',
  'Name': 'Current Member',
  'Reason': None,
  'StartDate': '2017-06-08T00:00:00'},
 'DateOfBirth': '1953-09-27T00:00:00',
 'DateOfDeath': {'@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
  '@xsi:nil': 'true'},
 'DisplayAs': 'Ms Diane Abbott',
 'FullTitle': 'Rt Hon Diane Abbott MP',
 'Gender': 'F',
 'House': 'Commons',
 'HouseEndDate': {'@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
  '@xsi:nil': 'true'},
 'HouseMemberships': {'HouseMembership': {'EndDate': {'@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
    '@xsi:nil': 'true'},
   'EndNotes': None,
   'EndReason': None,
   'House': 'Commons',
   'StartDate': '1987-06-11T00:00:00'}},
 'HouseStartDate': '1987-06-11T00:00:00',
 'LayingMinisterName': None,
 'ListAs': 'Abbott, Ms Diane',
 'MemberFrom': 'Hackney North and Stoke Newington',
 'Parties': {'Party': [{'@Id': '15',
    'EndDate': {'@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
     '@xsi:nil': 'true'},
    'Name': 'Labour',
    'Note': 'Created at General Election',
    'SitsOppositeSideToParty': 'False',
    'StartDate': '2017-06-08T00:00:00',
    'SubType': None},
   {'@Id': '15',
    'EndDate': '2017-05-03T00:00:00',
    'Name': 'Labour',
    'Note': 'Dissolution of Parliament',
    'SitsOppositeSideToParty': 'False',
    'StartDate': '2015-05-07T00:00:00',
    'SubType': None},
   {'@Id': '15',
    'EndDate': '2015-03-30T00:00:00',
    'Name': 'Labour',
    'Note': 'Dissolution of Parliament',
    'SitsOppositeSideToParty': 'False',
    'StartDate': '1987-06-11T00:00:00',
    'SubType': None}]},
 'Party': {'#text': 'Labour', '@Id': '15'}}

Creating Custom `pandas` Data Reader Packages



In [14]:

    
import pd_datareader_nhs.nhs_digital_ods as ods



In [15]:

    
ods.search(string='Prison', field='Label')









    Out[15]:







  
    
      
      Dataset
      Date
      Label
      Period
      Type
      URL
    
  
  
    
      45
      eprison
      25 Aug 2017
      Prisons in England and Wales
      Quarterly
      non-nhs
      https://digital.nhs.uk/media/401/eprison/zip/e...



In [12]:

    
dd=ods.download('eprison')
dd.head()









    Out[12]:







  
    
      
      Organisation Code
      Name
      Address Line 1
      Address Line 2
      Address Line 3
      Address Line 4
      Address Line 5
      Postcode
      Open Date
      Close Date
      Contact Telephone Number
      Amended Record Indicator
    
  
  
    
      0
      YDE01
      HMP NORTHUMBERLAND
      ACKLINGTON
      NaN
      NaN
      MORPETH
      NORTHUMBERLAND
      NE65 9XG
      2003-04-01
      NaT
      NaN
      0
    
    
      1
      YDE02
      HMP ISLE OF WIGHT
      CLISSOLD ROAD
      NaN
      NaN
      NEWPORT
      ISLE OF WIGHT
      PO30 5RS
      2003-04-01
      NaT
      NaN
      0
    
    
      2
      YDE03
      HMP ALTCOURSE
      HIGHER LANE
      FAZAKERLEY
      NaN
      LIVERPOOL
      MERSEYSIDE
      L9 7LH
      2003-04-01
      NaT
      NaN
      0
    
    
      3
      YDE05
      HMP BEDFORD
      ST. LOYES STREET
      NaN
      NaN
      BEDFORD
      BEDFORDSHIRE
      MK40 1HG
      2003-04-01
      NaT
      NaN
      0
    
    
      4
      YDE06
      HMP BELMARSH
      1 BELMARSH ROAD
      WESTERN WAY
      THAMESMEAD
      LONDON
      GREATER LONDON
      SE28 0EB
      2003-04-01
      NaT
      NaN
      0

Package Issues

development

building up example and reusable recipes

ownership and production quality (participation in development)

Notebooks as Open / Shared Recipes

BinderHub Build Sequence

"[P]hilosophically similar to Heroku Build Packs"

requirements.txt
- python packages
environment.yml
- conda environment specification
apt.txt
- debian packages that should be installed (latest version of Ubuntu)

postBuild
- arbitrary commands to be run after the whole repository has been built
REQUIRE
- Julia packages
Dockerfile
- treated as a regular Dockerfile. The presence of a Dockerfile will cause all other building behavior to not be triggered.

Building a Local Docker Image From a Github Repository

pip3 install jupyter-repo2docker

jupyter-repo2docker --image-name psychemedia/parlihacks --no-run https://github.com/psychemedia/parlihacks


docker push psychemedia/parlihacks

Creating Simple Service APIs

In terminal:

jupyter kernelgateway --KernelGatewayApp.api='kernel_gateway.notebook_http' --KernelGatewayApp.seed_uri='./SimpleAPI2.ipynb' --port 8899



In [24]:

    
import requests
requests.get('http://127.0.0.1:8899/demo/role/worker').json()









    Out[24]:





{'result': [{'id': '5', 'name': 'bobbie', 'role': 'worker'},
  {'id': '7', 'name': 'sam', 'role': 'worker'}]}



In [26]:

    
requests.get('http://127.0.0.1:8899/demo/name/jo').json()









    Out[26]:





{'result': [{'id': '3', 'name': 'jo', 'role': 'boss'}]}

Possible DIY Service Types

data servers: for example, API defined against a CSV file or simple SQLite3 database

known entity taggers: for example, identify members mentioned within a text

classifiers: for example, attempt to associate a text with a particular topic, or subject expert who can best handle a particular question, relative to a particular trained classifier / model

Reporting Patterns

generate report for a single MP about their constituency
- as a standalone item
- in comparison to all other consituencies nationally
- in comparison to a subset of other constitutencies eg neighbouring, similar

generate a report over all consituencies nationally

generate a report of wards within a particular constituency



In [ ]:

	Organisation Code	Name	Address Line 1	Address Line 2	Address Line 3	Address Line 4	Address Line 5	Postcode	Open Date	Close Date	Contact Telephone Number
0	YDE01	HMP NORTHUMBERLAND	ACKLINGTON	NaN	NaN	MORPETH	NORTHUMBERLAND	NE65 9XG	2003-04-01	NaT	NaN
1	YDE02	HMP ISLE OF WIGHT	CLISSOLD ROAD	NaN	NaN	NEWPORT	ISLE OF WIGHT	PO30 5RS	2003-04-01	NaT	NaN
2	YDE03	HMP ALTCOURSE	HIGHER LANE	FAZAKERLEY	NaN	LIVERPOOL	MERSEYSIDE	L9 7LH	2003-04-01	NaT	NaN
3	YDE05	HMP BEDFORD	ST. LOYES STREET	NaN	NaN	BEDFORD	BEDFORDSHIRE	MK40 1HG	2003-04-01	NaT	NaN
4	YDE06	HMP BELMARSH	1 BELMARSH ROAD	WESTERN WAY	THAMESMEAD	LONDON	GREATER LONDON	SE28 0EB	2003-04-01	NaT	NaN