Malaysian MP Statistics

A live notebook of working examples of using Sinar's Popit API and database of Malaysian MPs.

TODO

  • Detailed information of Persons should probably be appended to post memberships this would allow us to show post (Seat) information, not just details of person.

  • Refactor functions here into common library

Issues

  • Posts should check for role, eg. Member of Parliament. Speaker is a post in Dewan Rakyat, but is not an MP.

Author

Feel free to do pull requests, or contact me on issues with the data.

Khairil Yusof khairil.yusof@sinarproject.org


In [1]:
import requests
import json

#Dewan Rakyat MP Posts in Sinar Malaysia Popit Database
posts = []
for page in range(1,10):
    dewan_rakyat_request = requests.get('http://sinar-malaysia.popit.mysociety.org/api/v0.1/search/posts?q=organization_id:53633b5a19ee29270d8a9ecf'+'&page='+str(page))
    for post in (json.loads(dewan_rakyat_request.content)['result']):
        posts.append(post)

Now we will load up information on the MPs holding these posts


In [2]:
import datetime
from dateutil import parser


current = datetime.date(2013,5,5)

def has_end_date(member):
    
    if member.has_key('end_date') and member['end_date'] == '':
        return False
    elif not member.has_key('end_date'):
        return False
    else:
        return True
    
def current_MP(member):
    #Legislative tag term here would simply
    if (parser.parse(member['start_date'])).date() > current:
        if not has_end_date(member):
            return True
    else:
        return False
    
def person(person_id):
    #Load up information of persons from Popit database
    req = requests.get('https://sinar-malaysia.popit.mysociety.org/api/v0.1/persons/' + person_id)
    return json.loads(req.content)['result']


def age(str):
    #calculate age based on date strings stored in Popit
    born = parser.parse(str)
    today = datetime.date.today()
    age = today.year - born.year - ((today.month, today.day) < (born.month, born.day)) 
    return int(age)

#Current MPs will not have end dates, and have terms after 2013-05-05

MP_ids = []
for post in posts:
    for member in post['memberships']:
        if current_MP(member):
            MP_ids.append(member['person_id'])

In [4]:
#Pull down the data of current MPs from Popit Database add calculate age if there is birthdate
MPs = []

for id in MP_ids:
    MPs.append(person(id))

In [23]:
for MP in MPs:
    if MP.has_key('birth_date'):
        if MP['birth_date']:
            #add current age in addition to the values in Popit
            MP['age'] = age(MP['birth_date'])

Women MPs


In [6]:
WomenMPs = []
for MP in MPs:
    if MP.has_key('gender') and MP['gender'] == 'Female':
        WomenMPs.append(MP)

print "Number of Women MPs " + str(len(WomenMPs))

for MP in WomenMPs:
    print MP['name']


Number of Women MPs 24
Norah Abdul Rahman
Nurul Izzah Anwar
Teresa Kok Suh Sim
Mas Ermieyati Samsudin
Noraini Ahmad
Halimah Mohamed Sadique
Azalina Othman Said
Normala Abdul Samad
Teo Nie Ching
Rosnah Abdul Rashid Shirlin
Azizah Mohd Dun
Tsen Thau Lin
Yap Kain Ching
Julian Tan Kok Ping
Rubiah Wang
Nancy Shukri
Rohani Abdul Karim
Alice Lau Kiong Yieng
Wan Azizah Wan Ismail
Noriah Kasnon
Siti Zailah Binti Mohd Yusoff
Kasthuriraani Patto
Fuziah Binti Salleh
Zuraida Kamaruddin

Age of MPs


In [25]:
import numpy

#list of ages
ages = []
for MP in MPs:
    if MP.has_key('age'):
        ages.append(int(MP['age']))
    
print numpy.median(ages)
print numpy.max(ages)
print numpy.min(ages)


56.0
74
32

Pandas

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

If you're learning Python to work with data, it's worth getting used to this library, as it provides pretty much all you will need when working with data from importing and cleaning messy data, to exporting it, including working with very large data sets.

A lot of the earlier work, such as cleaning, getting unique values etc. could be done easily with built-in functions of pandas as a DataFrame.


In [26]:
import pandas

pandas.DataFrame(MPs)


Out[26]:
age birth_date contact_details death_date dissolution_date end_date founding_date gender html_url id ... initials links memberships name other_names proxy_image sources start_date summary url
0 39 1975-09-12 [{u'type': u'phone', u'id': u'558337eef3c3651a... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e48755222837c2c0596a7 ... NaN [{u'url': u'http://ongkianming.com', u'note': ... [{u'contact_details': [], u'links': [], u'end_... Ong Kian Ming [{u'name': u'王建民', u'id': u'558337eef3c3651a49... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://ongkianming.com', u'note': ... NaN Dr. Ong Kian Ming was elected as a first-time ... https://sinar-malaysia.popit.mysociety.org/api...
1 54 1960-11-01 [{u'type': u'phone number', u'id': u'5590d64dd... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e46ee5222837c2c0592eb ... NaN [{u'url': u'http://charlessantiago.org/', u'no... [{u'contact_details': [], u'links': [], u'end_... Charles Anthony A/L R. Santiago [{u'name': u'Charles Santiago', u'id': u'55918... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
2 46 1968-12-18 [{u'type': u'email', u'id': u'55b351e0f1faa736... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 53635149f1eab6270da6c8f6 ... NaN [{u'url': u'https://en.wikipedia.org/wiki/Wong... [{u'area_id': None, u'contact_details': [], u'... Wong Chen [{u'name': u'Wong Chen X', u'id': u'55b351e0f1... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Wong Chen is a Malaysian politician and is the... https://sinar-malaysia.popit.mysociety.org/api...
3 55 1959-10-04 [{u'type': u'address', u'id': u'55b6017353d566... NaN NaN NaN NaN Female https://sinar-malaysia.popit.mysociety.org/per... 545e43f15222837c2c058b76 ... NaN [{u'url': u'http://dtknorah.blogspot.com/', u'... [{u'contact_details': [], u'links': [], u'end_... Norah Abdul Rahman [{u'note': u'With honorifics', u'name': u'Datu... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Datuk Hajjah Norah Abdul Rahman (born 4 Octobe... https://sinar-malaysia.popit.mysociety.org/api...
4 37 1977-09-14 [{u'type': u'handphone number', u'id': u'5582d... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e429d5222837c2c058829 ... NaN [{u'url': u'https://ms-my.facebook.com/rafizir... [{u'contact_details': [], u'links': [], u'end_... Mohd Rafizi Bin Ramli [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
5 64 1951-07-16 [{u'type': u'facebook', u'id': u'5582d9c500912... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45785222837c2c058f42 ... NaN [{u'url': u'https://en.wikipedia.org/wiki/Che_... [{u'contact_details': [], u'links': [], u'end_... Che Rosli Bin Che Mat [{u'name': u'Dr. Che Rosli Che Mat', u'id': u'... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
6 42 1973-06-19 [{u'type': u'phone', u'id': u'5582db444a70d893... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e43a65222837c2c058abc ... NaN [{u'url': u'https://en.wikipedia.org/wiki/Gobi... [{u'contact_details': [], u'links': [], u'end_... Gobind Singh Deo [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Gobind Singh Deo (born 19 June 1973) is a prom... https://sinar-malaysia.popit.mysociety.org/api...
7 45 1970-06-18 [{u'type': u'facebook', u'id': u'55b5fcce9fcfa... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e44d15222837c2c058da1 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Hee Loy Sian [] https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://www.keadilandaily.com', u'n... NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
8 43 1972-08-01 [{u'type': u'twitter', u'id': u'55830e49f3c365... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e469b5222837c2c05921b ... NaN [{u'url': u'http://tonypua.blogspot.com', u'no... [{u'contact_details': [], u'links': [], u'end_... Tony Pua Kiam Wee [{u'name': u'Pua Kiam Wee', u'id': u'55830e49f... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
9 58 1956-12-08 [{u'type': u'phone', u'id': u'55b6edf00d1ed7f9... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e45805222837c2c058f56 ... NaN [{u'url': u'http://sivarasa.blogspot.com', u'n... [{u'contact_details': [], u'links': [], u'end_... Sivarasa Rasiah [{u'name': u'Sivarasa A/L K. Rasiah', u'id': u... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Sivarasa Rasiah (Tamil: சிவராசா ராசையா), or al... https://sinar-malaysia.popit.mysociety.org/api...
10 57 1957-08-14 [{u'type': u'twitter', u'id': u'55832f5e80ad1f... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e43335222837c2c05899c ... NaN [{u'url': u'http://www.khalidsamad.com', u'not... [{u'contact_details': [], u'links': [], u'end_... Khalid Abdul Samad [{u'name': u'Hj Khalid Abd Samad', u'id': u'55... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN A politician who is a member of PAS. He gradua... https://sinar-malaysia.popit.mysociety.org/api...
11 NaN NaN [{u'type': u'handphone number', u'id': u'55833... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e43145222837c2c058952 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... G Manivannan A/L Gowindasamy [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
12 57 1958-02-14 [{u'type': u'phone number', u'id': u'55833b42f... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e44185222837c2c058bd8 ... NaN [{u'url': u'drsitimariah.blogspot.com/', u'not... [{u'contact_details': [], u'links': [], u'end_... Siti Mariah Binti Mahmud [{u'name': u'Dr. Siti Mariah binti Mahmud', u'... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
13 NaN NaN [{u'type': u'address', u'id': u'5571b6775ccbb5... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e447a5222837c2c058cca ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Abdullah Sani Abdul Hamid [{u'name': u'Abdullah Sani Bin Abdul Hamid', u... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://www.mtuc.org.my/about-us/of... NaN Abdullah Sani Abdul Hamid is a Malaysian polit... https://sinar-malaysia.popit.mysociety.org/api...
14 NaN NaN [{u'type': u'twitter', u'id': u'5571b661e5ce8b... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45f95222837c2c059086 ... NaN [{u'url': u'https://ybhanipamaidin.wordpress.c... [{u'contact_details': [], u'links': [], u'end_... Mohamed Hanipa Maidin [{u'name': u'Hanipa Maidin', u'id': u'5571b661... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
15 73 1942-05-26 [{u'type': u'twitter', u'id': u'5571b64bba593e... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e47f15222837c2c05956b ... NaN [{u'url': u'http://tansenggiaw.blogspot.com/',... [{u'contact_details': [], u'links': [], u'end_... Tan Seng Giaw [{u'name': u'Tan Seng Giaw @ Tan Chun Tin', u'... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://www.thenutgraph.com/mp-watc... NaN Dr. Tan Seng Giaw (Chinese : 陈胜尧;) is a Malays... https://sinar-malaysia.popit.mysociety.org/api...
16 51 1963-12-21 [{u'type': u'twitter', u'id': u'5571b61f46ee15... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 556c804aaf36fc0760ed58cf ... [] [{u'contact_details': [], u'links': [], u'end_... Chua Tian Chang [{u'name': u'Tian Chua', u'id': u'5571b61f46ee... https://sinar-malaysia.popit.mysociety.org/ima... [] NaN Chua Tian Chang, better known as Tian Chua (Si... https://sinar-malaysia.popit.mysociety.org/api...
17 43 1972-07-25 [{u'type': u'facebook', u'id': u'5571b5ef46ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e472b5222837c2c059384 ... NaN [{u'url': u'http://limlipeng.blogspot.com/', u... [{u'contact_details': [], u'links': [], u'end_... Lim Lip Eng [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Lim Lip Eng is a Malaysian politician and is t... https://sinar-malaysia.popit.mysociety.org/api...
18 52 1963-06-02 [{u'type': u'address', u'id': u'5571b5cfe5ce8b... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45d25222837c2c059023 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Ahmad Fauzi Zahari [{u'note': u'With honorifics', u'name': u'Dato... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
19 NaN NaN [{u'type': u'facebook', u'id': u'5571b5b6e5ce8... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e470b5222837c2c059333 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Johari Abdul Ghani [{u'note': u'With honorifics', u'name': u'Datu... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://uda.com.my/about/#lead', u'... NaN Dato' Haji Johari Abdul Ghani is Barisan Nasio... https://sinar-malaysia.popit.mysociety.org/api...
20 NaN NaN [{u'type': u'address', u'id': u'5571b58d46ee15... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e42945222837c2c058813 ... NaN [{u'url': u'http://fongkuilun.blogspot.com/', ... [{u'contact_details': [], u'links': [], u'end_... Fong Kui Lun [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Fong Kui Lun (Chinese: 方贵伦) is a Malaysian pol... https://sinar-malaysia.popit.mysociety.org/api...
21 34 1980-11-19 [{u'type': u'address', u'id': u'55b5fd3c9fcfab... NaN NaN NaN NaN Female https://sinar-malaysia.popit.mysociety.org/per... 545e469e5222837c2c059223 ... NaN [{u'url': u'http://www.nurulizzah.com', u'note... [{u'contact_details': [], u'links': [], u'end_... Nurul Izzah Anwar [{u'name': u'Nurul Izzah Binti Anwar', u'id': ... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN A Malaysian politician from Parti Keadilan Rak... https://sinar-malaysia.popit.mysociety.org/api...
22 51 1964-03-31 [{u'type': u'facebook', u'id': u'55b5fd569fcfa... NaN NaN NaN NaN Female https://sinar-malaysia.popit.mysociety.org/per... 545e46185222837c2c0590d5 ... NaN [{u'url': u'http://teresakok.com/', u'note': u... [{u'contact_details': [], u'links': [], u'end_... Teresa Kok Suh Sim [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Teresa Kok Suh Sim (Chinese: 郭素沁) is a Malaysi... https://sinar-malaysia.popit.mysociety.org/api...
23 57 1957-10-07 [{u'type': u'facebook', u'id': u'5571b54a46ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e425c5222837c2c058787 ... NaN [{u'url': u'Official Website', u'note': u'http... [{u'contact_details': [], u'links': [], u'end_... Tan Kok Wai [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Tan Kok Wai (born 7 October 1957) is a Malaysi... https://sinar-malaysia.popit.mysociety.org/api...
24 68 1946-12-14 [{u'type': u'twitter', u'id': u'5571b52b46ee15... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e444f5222837c2c058c61 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Abdul Khalid Ibrahim [{u'name': u'Khalid Ibrahim', u'id': u'5571b52... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Tan Sri Dato' Seri Abdul Khalid bin Ibrahim is... https://sinar-malaysia.popit.mysociety.org/api...
25 NaN NaN [{u'type': u'facebook', u'id': u'5571b4fe46ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e44215222837c2c058bef ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Zainudin Ismail [{u'note': u'With honorifics', u'name': u'Dato... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
26 NaN NaN [{u'type': u'facebook', u'id': u'5571b4e246ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e46c35222837c2c05927f ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Mohamed Isa Abdul Samad [{u'note': u'With honorifics', u'name': u'Tan ... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN YB Tan Sri Haji Mohd Isa Dato' Haji Abdul Sama... https://sinar-malaysia.popit.mysociety.org/api...
27 38 1977-04-28 [{u'type': u'facebook', u'id': u'5571b4a946ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e448c5222837c2c058cf5 ... NaN [{u'url': u'http://lokesiewfook.blogspot.com/'... [{u'contact_details': [], u'links': [], u'end_... Loke Siew Fook [{u'name': u'Anthony Loke Siew Fook', u'id': u... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Anthony Loke Siew Fook is a Malaysian politici... https://sinar-malaysia.popit.mysociety.org/api...
28 69 1946-04-24 [{u'type': u'facebook', u'id': u'5571b48c46ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45905222837c2c058f7f ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Hasan Malek [{u'note': u'With honorifics', u'name': u'Dato... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Dato' Sri Haji Hasan Malek is a Malaysian poli... https://sinar-malaysia.popit.mysociety.org/api...
29 NaN NaN [{u'type': u'facebook', u'id': u'5571b47646ee1... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e47505222837c2c0593e3 ... NaN [{u'url': u'http://teokokseong.blogspot.com/',... [{u'contact_details': [], u'links': [], u'end_... Teo Kok Seong [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Teo Kok Seong is a Malaysian politician who is... https://sinar-malaysia.popit.mysociety.org/api...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
192 50 1964-10-25 [{u'note': u'MyMP Survey', u'type': u'phone nu... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e46ac5222837c2c059246 ... NaN [{u'url': u'http://drmujahid.com', u'note': u'... [{u'contact_details': [], u'links': [], u'end_... Dr. Mujahid Yusof Rawa [{u'name': u'Mujahid Yusof Rawa', u'id': u'55a... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
193 NaN NaN [{u'type': u'phone number', u'id': u'557bf846d... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e42c25222837c2c058883 ... NaN [{u'url': u'https://www.facebook.com/dapthomas... [{u'contact_details': [], u'links': [], u'end_... Su Keong Siong [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
194 NaN NaN [{u'type': u'phone number', u'id': u'5575c5775... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e42d85222837c2c0588bc ... NaN [{u'url': u'https://www.facebook.com/reezal.me... [{u'contact_details': [], u'links': [], u'end_... Reezal Merican Bin Naina Merican [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
195 58 1957-03-12 [{u'type': u'phone number', u'id': u'55a36cb47... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e475c5222837c2c0593ff ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Hamzah Zainuddin [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
196 48 1967-01-06 [{u'type': u'phone number', u'id': u'557b34987... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e425e5222837c2c05878d ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Shamsul Anuar Bin Nasarah [{u'name': u'Datuk Samsul Anuar Nasarah', u'id... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
197 NaN NaN [{u'type': u'phone number', u'id': u'557b0ada7... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e47375222837c2c0593a3 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Hasbullah Bin Osman [{u'name': u'Datuk Hasbullah Osman', u'id': u'... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
198 NaN NaN [{u'type': u'handphone', u'id': u'557a9c0d435e... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45725222837c2c058f33 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... N Surendran A/L K Nagarajan [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
199 63 1952-07-21 [{u'type': u'phone number', u'id': u'55bdb7fc1... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e42e25222837c2c0588d4 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Ahmad Husni Mohamad Hanadzlah [{u'name': u'Datuk Ahmad Husni Hanadzlah', u'i... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://www.bloomberg.com/research/... NaN Dato’ Seri Ahmad Husni Mohamad Hanadzlah was a... https://sinar-malaysia.popit.mysociety.org/api...
200 60 1955-03-28 [{u'type': u'phone number', u'id': u'55aa504af... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e44e35222837c2c058dcd ... NaN [{u'url': u'http://en.wikipedia.org/wiki/Micha... [{u'contact_details': [], u'links': [], u'end_... Michael Jeyakumar Devaraj [{u'name': u'Dr Michael Jeyakumar Devaraj', u'... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN Michael Jeyakumar Devaraj is a Malaysian polit... https://sinar-malaysia.popit.mysociety.org/api...
201 NaN None [{u'type': u'phone number', u'id': u'55a37b76b... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 557b39bdeb081bf345655519 ... [{u'url': u'https://idrisahmad.wordpress.com/a... [{u'contact_details': [], u'links': [], u'url'... Idris Ahmad [{u'name': u'Idris Ahmad', u'id': u'55a37b76ba... https://sinar-malaysia.popit.mysociety.org/ima... [] NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
202 61 1954-05-15 [{u'type': u'phone number', u'id': u'55a37dd5f... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e43835222837c2c058a67 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Mohd Nazri Abdul Aziz [{u'name': u'Mohamed Nazri Bin Abdul Aziz', u'... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
203 53 1961-09-29 [{u'type': u'phone number', u'id': u'557c57e5d... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e44005222837c2c058b9d ... NaN [{u'url': u' ngehkooham.blogspot.com', u'note'... [{u'contact_details': [], u'links': [], u'url'... Ngeh Koo Ham [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
204 NaN NaN [] NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e489c5222837c2c059705 ... NaN [] [{u'contact_details': [], u'links': [], u'post... Sivakumar A/L Varatharaju Naidu [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
205 42 1972-11-11 [{u'type': u'phone number', u'id': u'558f1e0eb... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45745222837c2c058f39 ... NaN [{u'url': u'https://twitter.com/ngakorming?lan... [{u'contact_details': [], u'links': [], u'end_... Nga Kor Ming [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
206 53 1961-11-28 [{u'type': u'phone number', u'id': u'55a9c7453... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e423f5222837c2c058741 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Noor Azmi Bin Ghazali [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
207 55 1960-01-16 [{u'type': u'phone number', u'id': u'557c554b7... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e43b45222837c2c058adf ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Wan Mohammad Khair-Il Anuar Bin Wan Ahmad [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN An architect by profession. He owns an archite... https://sinar-malaysia.popit.mysociety.org/api...
208 61 1954-07-10 [{u'type': u'phone number', u'id': u'55af09332... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e48c25222837c2c059765 ... NaN [] [{u'contact_details': [], u'links': [], u'url'... Mohamad Imran Bin Abd Hamid [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN A retired naval officer, now a politician and ... https://sinar-malaysia.popit.mysociety.org/api...
209 47 1968 [{u'type': u'phone number', u'id': u'55ac6af20... NaN NaN NaN NaN Male https://sinar-malaysia.popit.mysociety.org/per... 545e42955222837c2c058816 ... NaN [{u'url': u'https://www.facebook.com/KoChungSe... [{u'contact_details': [], u'links': [], u'end_... Ko Chung Sen [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
210 67 1948-01-17 [{u'type': u'phone number', u'id': u'557c6fe3d... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e47b85222837c2c0594e2 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Tajuddin Bin Abdul Rahman [{u'name': u'Datuk Tajuddin Abdul Rahman', u'i... https://sinar-malaysia.popit.mysociety.org/ima... [{u'url': u'http://www.freemalaysiatoday.com/w... NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
211 NaN NaN [] NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e452b5222837c2c058e80 ... NaN [] [{u'contact_details': [], u'links': [], u'post... Madhi Bin Hasan [] NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
212 NaN NaN [{u'type': u'phone number', u'id': u'557c5bc0d... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e461b5222837c2c0590dc ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Mohd Zaim Bin Abu Hasan [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
213 55 1959-09-26 [{u'type': u'phone', u'id': u'55ad8b0a5ebd2964... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e48fe5222837c2c0597f8 ... NaN [{u'url': u'https://www.facebook.com/leeboonch... [{u'contact_details': [], u'links': [], u'end_... Lee Boon Chye [{u'name': u'Dr Lee Boon Chye', u'id': u'55ad8... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
214 NaN NaN [{u'type': u'phone number', u'id': u'557c690e2... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e48a75222837c2c059720 ... NaN [] [{u'contact_details': [], u'links': [], u'url'... Saravanan A/L Murugan [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
215 NaN NaN [{u'type': u'phone number', u'id': u'557db9903... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e44d65222837c2c058dac ... NaN [{u'url': u'https://www.facebook.com/datomahsi... [{u'contact_details': [], u'links': [], u'end_... Mah Siew Keong [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
216 66 1949-03-01 [{u'type': u'phone number', u'id': u'557df20ec... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45e35222837c2c05904e ... NaN [{u'url': u'https://www.facebook.com/DatukSeri... [{u'contact_details': [], u'links': [], u'end_... Palanivel A/L K. Govindasamy [{u'name': u'Datuk G. Palanivel', u'id': u'558... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
217 57 1957-08-10 [{u'type': u'phone number', u'id': u'557c03bbb... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e48ea5222837c2c0597c7 ... NaN [{u'url': u'https://twitter.com/mkula', u'note... [{u'contact_details': [], u'links': [], u'end_... M. Kulasegaran A/L V. Murugeson [{u'name': u'M Kulasegaran', u'id': u'558f214b... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
218 61 1954-05-29 [{u'type': u'phone number', u'id': u'557df0bbc... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e47565222837c2c0593f2 ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Ong Ka Chuan [{u'name': u'Dato' Seri Ong Ka Chuan', u'id': ... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
219 58 1957-01-27 [{u'type': u'phone number', u'id': u'5582a65e4... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e44855222837c2c058ce4 ... NaN [{u'url': u'http://williamleongjeekeen.blogspo... [{u'contact_details': [], u'links': [], u'end_... William Leong Jee Keen [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
220 NaN NaN [{u'type': u'phone number', u'id': u'557df40fc... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e45635222837c2c058f0f ... NaN [] [{u'contact_details': [], u'links': [], u'end_... Abdul Rahman Bin Mohamad [] https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...
221 NaN NaN [{u'type': u'phone number', u'id': u'5580ff404... NaN NaN NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/per... 545e46f25222837c2c0592f6 ... NaN [{u'url': u'https://www.facebook.com/pages/Dat... [{u'contact_details': [], u'links': [], u'end_... Abdul Aziz Sheikh Fadzir [{u'name': u'Datuk Abdul Aziz Sheikh Fadzir', ... https://sinar-malaysia.popit.mysociety.org/ima... NaN NaN NaN https://sinar-malaysia.popit.mysociety.org/api...

222 rows × 23 columns


In [27]:
df = pandas.DataFrame(MPs)


print df['age'].median()
print df['age'].max()
print df['age'].min()


56.0
74.0
32.0

We could have dropped duplicates from bad data:

df.drop_duplicates('id')

Parse and set birth_date column as datetime to calculate age without parsing it manually:

df['birth_date']= pandas.to_datetime(df['birth_date'])

Best of all after cleaning up the data, we can easily export it to CSV format where it is more easily usable by normal users in spreadsheets or plotting charts.


In [28]:
MP_source = {'name':df['name'],'birth_date':df['birth_date'],'age':df['age']}

In [29]:
MP_Names = pandas.DataFrame(MP_source)

MP_Names.sort('age')


Out[29]:
age birth_date name
191 32 1982-10-17 Zairil Khir Johari
167 33 1982-05-13 Sim Chee Keong
65 34 1981-01-27 Teo Nie Ching
21 34 1980-11-19 Nurul Izzah Anwar
168 35 1979-08-09 Kasthuriraani Patto
171 37 1977-12-20 Ng Wei Aik
4 37 1977-09-14 Mohd Rafizi Bin Ramli
47 37 1977-11-27 Liew Chin Tong
43 37 1977-10-19 Chua Tee Yong
36 38 1976-11-13 Mas Ermieyati Samsudin
27 38 1977-04-28 Loke Siew Fook
178 39 1976-02-06 Sim Tze Tzin
30 39 1976-01-10 Khairy Jamaluddin Abu Bakar
0 39 1975-09-12 Ong Kian Ming
37 40 1974-12-29 Shamsul Iskandar Mohd Akin
53 42 1972-12-17 Er Teck Hwa
133 42 1972-12-28 Gooi Hsiao-Leung
6 42 1973-06-19 Gobind Singh Deo
205 42 1972-11-11 Nga Kor Ming
17 43 1972-07-25 Lim Lip Eng
76 43 1972-07-10 Rosnah Abdul Rashid Shirlin
8 43 1972-08-01 Tony Pua Kiam Wee
54 44 1971-07-24 Liang Teck Meng
98 44 1971-02-12 Chong Chieng Jen
42 44 1970-10-14 Razali Ibrahim
144 45 1970-03-22 Nik Mohamad Abduh Bin Nik Abdul Aziz
7 45 1970-06-18 Hee Loy Sian
82 45 1970-03-08 Raime Unggi
2 46 1968-12-18 Wong Chen
51 46 1968-10-20 Wee Ka Siong
... ... ... ...
156 NaN NaN Wan Abd Rahim Bin Wan Abdullah
160 NaN NaN Wan Hassan Mohd Ramli
161 NaN NaN Mohd Khairuddin Bin Aman Razali
164 NaN NaN Che Mohamad Zulkifly Bin Jusoh
165 NaN NaN Shabudin Yahaya
166 NaN None Ramkarpal Singh A/L Karpal Singh
170 NaN NaN Mansor Bin Othman
173 NaN NaN Jeff Ooi Chun Aun
174 NaN NaN Hilmi Bin Yahaya
176 NaN NaN Fuziah Binti Salleh
179 NaN NaN Mohd Ariff Sabri Bin Abdul Aziz
180 NaN NaN Ahmad Nazlan Bin Idris
181 NaN NaN Ismail Bin Abd. Muttalib
182 NaN NaN Shahril Azman Bin Abd Halim
184 NaN NaN Noh Bin Omar
187 NaN NaN Liow Tiong Lai
188 NaN NaN Irmohizam Bin Ibrahim
189 NaN NaN Nasrudin Bin Hassan
193 NaN NaN Su Keong Siong
194 NaN NaN Reezal Merican Bin Naina Merican
197 NaN NaN Hasbullah Bin Osman
198 NaN NaN N Surendran A/L K Nagarajan
201 NaN None Idris Ahmad
204 NaN NaN Sivakumar A/L Varatharaju Naidu
211 NaN NaN Madhi Bin Hasan
212 NaN NaN Mohd Zaim Bin Abu Hasan
214 NaN NaN Saravanan A/L Murugan
215 NaN NaN Mah Siew Keong
220 NaN NaN Abdul Rahman Bin Mohamad
221 NaN NaN Abdul Aziz Sheikh Fadzir

222 rows × 3 columns


In [30]:
%matplotlib inline 
grouped = MP_Names.groupby('age')
grouped.age.count().plot(kind='bar',figsize=(15,15))


Out[30]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f80c1142c90>

Previous example tried to massage python data structures into Pandas DataFrame which works, but isn't very pretty.

Ng Swee Meng sweester@sinarproject.org has contributed proper way of building up data structures for Pandas DataFrames in the following example:


In [32]:
import pandas
data = { "age": [], "birth_date": []}
data_index = { "age": [], "birth_date": []}

for entry in MPs:
    if entry.has_key('age'):
        data["age"].append(entry["age"])
        data_index["age"].append(entry["name"])
        data["birth_date"].append(entry["birth_date"])
        data_index["birth_date"].append(entry["name"])

final_data = { "age": pandas.Series(data["age"], index=data_index["age"]),
                "birth_date": pandas.Series(data["birth_date"], index=data_index["birth_date"])
             }
mp_age_df = pandas.DataFrame(final_data)
mp_age_df.sort("age")


Out[32]:
age birth_date
Zairil Khir Johari 32 1982-10-17
Sim Chee Keong 33 1982-05-13
Teo Nie Ching 34 1981-01-27
Nurul Izzah Anwar 34 1980-11-19
Kasthuriraani Patto 35 1979-08-09
Ng Wei Aik 37 1977-12-20
Mohd Rafizi Bin Ramli 37 1977-09-14
Liew Chin Tong 37 1977-11-27
Chua Tee Yong 37 1977-10-19
Mas Ermieyati Samsudin 38 1976-11-13
Loke Siew Fook 38 1977-04-28
Sim Tze Tzin 39 1976-02-06
Khairy Jamaluddin Abu Bakar 39 1976-01-10
Ong Kian Ming 39 1975-09-12
Shamsul Iskandar Mohd Akin 40 1974-12-29
Er Teck Hwa 42 1972-12-17
Gooi Hsiao-Leung 42 1972-12-28
Gobind Singh Deo 42 1973-06-19
Nga Kor Ming 42 1972-11-11
Lim Lip Eng 43 1972-07-25
Rosnah Abdul Rashid Shirlin 43 1972-07-10
Tony Pua Kiam Wee 43 1972-08-01
Liang Teck Meng 44 1971-07-24
Chong Chieng Jen 44 1971-02-12
Razali Ibrahim 44 1970-10-14
Nik Mohamad Abduh Bin Nik Abdul Aziz 45 1970-03-22
Hee Loy Sian 45 1970-06-18
Raime Unggi 45 1970-03-08
Wong Chen 46 1968-12-18
Wee Ka Siong 46 1968-10-20
... ... ...
Kamarudin Bin Jaffar 63 1951-11-03
Richard Riot Jaem 63 1951-12-01
Che Rosli Bin Che Mat 64 1951-07-16
Tengku Adnan Tengku Mansor 64 1950-12-20
Wahab Dolah 64 1950-11-15
Wong Sze Phin 65 1950-03-01
William Nyallau Badak 65 1950-07-22
Shahrir Abdul Samad 65 1949-11-22
Ahmad Lai Bujang 65 1949-11-26
Muhammad Leo Michael Toyad Abdullah 65 1950-04-11
Joseph Salang Gandum 65 1950-01-01
Palanivel A/L K. Govindasamy 66 1949-03-01
James Dawos Mamit 66 1948-11-04
Sim Tong Him 66 1948-09-08
Ahmad Hamzah 66 1948-09-27
Abdul Manan Ismail 67 1948-06-28
Tajuddin Bin Abdul Rahman 67 1948-01-17
Abdul Hadi Bin Awang 67 1947-10-20
Kamarul Baharin Abbas 68 1947-06-12
Abdul Khalid Ibrahim 68 1946-12-14
Muhyiddin Yassin 68 1947-05-15
Fauzi Bin Abdul Rahman 68 1946-08-27
Hasan Malek 69 1946-04-24
Wan Junaidi Tuanku Jaafar 69 1946-02-01
Henry Sum Agong 69 1946-02-18
Joseph Kurup 70 1944-12-15
Abdul Ghapur Salleh 72 1943-03-21
Tan Seng Giaw 73 1942-05-26
Joseph Pairin Kitingan 74 1940-08-17
Lim Kit Siang 74 1941-02-20

144 rows × 2 columns


In [34]:
mp_age_df["age"].plot(kind="hist",figsize=(15,15))


Out[34]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f80c0a96710>

In [ ]: