Web Scraping: Extracting Data from the Web

Some Import


In [56]:
%matplotlib inline 
from selenium import webdriver
import time,re,json,numpy as np
import pandas as pd
from collections import defaultdict,Counter
import matplotlib.pyplot as plt

Initial Setup and Launch the browser to open the URL


In [57]:
url = "http://www.imdb.com/list/ls061683439/"
with open('./img/filmfare.json',encoding="utf-8") as f:
    datatbl = json.load(f)
driver = webdriver.Chrome(datatbl['data']['chromedriver'])
driver.get(url)

Beautiful Soup


In [60]:
from bs4 import BeautifulSoup
soup = BeautifulSoup(driver.page_source, 'lxml')
lstelem = soup.findAll("div", attrs={"class" : 'info'})
Movielist = []
for lst in lstelem:
    Movielist.append((lst.find('b').contents[0]).text)
print("First 10 Movies in the list")
Movielist[0:10]


First 10 Movies in the list
Out[60]:
['Bajirao Mastani',
 'Queen',
 'Bhaag Milkha Bhaag',
 'Barfi!',
 'Zindagi Na Milegi Dobara',
 'Dabangg',
 '3 Idiots',
 'Jodhaa Akbar',
 'Like Stars on Earth',
 'Rang De Basanti']

Getting Data

Function to extract the data from Web using Selenium


In [61]:
def ExtractText(Xpath):
    textlist=[]
    if(Xpath=='Movies_Director_Xpath'):
        for item in range(1,123,2):
            textlist.append(driver.find_element_by_xpath(datatbl['data'][Xpath]+'[%d]'%item).text)
    else:
        [textlist.append(item.text) for item in driver.find_elements_by_xpath(datatbl['data'][Xpath])]
    return textlist

Let's extract all the required data like Ratings,Votes,Genre, Year of Release for the Best Movies


In [62]:
#Extracting Data from Web
Movies_Votes,Movies_Name,Movies_Ratings,Movies_RunTime=[[] for i in range(4)]
datarepo = [[]]*5
Xpath_list = ['Movies_Name_Xpath','Movies_Rate_Xpath','Movies_Runtime_Xpath','Movies_Votes_Xpath',
              'Movies_Director_Xpath']

for i in range(5):
    if(i==3):
        driver.find_element_by_xpath(datatbl['data']['listview']).click()
    if(i==4):
        driver.find_element_by_xpath(datatbl['data']['detailview']).click()
    datarepo[i] = ExtractText(Xpath_list[i])

driver.quit()

Data in List


In [63]:
# Movie Name List & Ratings
print(datarepo[0][:5])
print(datarepo[1][:5])
print(datarepo[3][:5])
print(datarepo[4][:5])
print("")
print(datarepo[3][:5])


['Bajirao Mastani', 'Queen', 'Bhaag Milkha Bhaag', 'Barfi!', 'Zindagi Na Milegi Dobara']
['7.2', '8.4', '8.3', '8.2', '8.1']
['17,622', '39,681', '39,919', '52,476', '41,890']
['Director: Sanjay Leela Bhansali', 'Director: Vikas Bahl', 'Director: Rakeysh Omprakash Mehra', 'Director: Anurag Basu', 'Director: Zoya Akhtar']

['17,622', '39,681', '39,919', '52,476', '41,890']

Store Data in a Python Dictionary


In [64]:
# Result in a Python Dictionary
Years=range(2015,1954,-1)
result = defaultdict(dict)
for i in range(0,len(datarepo[0])):
    result[i]['Movie Name']= datarepo[0][i]
    result[i]['Year']= Years[i]
    result[i]['Rating']= datarepo[1][i]
    result[i]['Votes']= datarepo[3][i]
    result[i]['RunTime']= datarepo[2][i]
    result[i]['Genre']= datatbl['data']['Genre'][i]
    result[i]['Director']= datarepo[4][i]

Let's see now how the data in dictionary looks like?


In [66]:
print(json.dumps((result[0]),indent=4))


{
    "Votes": "17,622",
    "Movie Name": "Bajirao Mastani",
    "Director": "Director: Sanjay Leela Bhansali",
    "Rating": "7.2",
    "Year": 2015,
    "Genre": "Drama",
    "RunTime": "A historical account of the romance between the Maratha general, Baji Rao I and Mastani, a Muslim princess. (158 mins.)"
}

In [67]:
print(json.dumps((result),indent=4))


{
    "0": {
        "Votes": "17,622",
        "Movie Name": "Bajirao Mastani",
        "Director": "Director: Sanjay Leela Bhansali",
        "Rating": "7.2",
        "Year": 2015,
        "Genre": "Drama",
        "RunTime": "A historical account of the romance between the Maratha general, Baji Rao I and Mastani, a Muslim princess. (158 mins.)"
    },
    "1": {
        "Votes": "39,681",
        "Movie Name": "Queen",
        "Director": "Director: Vikas Bahl",
        "Rating": "8.4",
        "Year": 2014,
        "Genre": "Drama",
        "RunTime": "A Delhi girl from a traditional family sets out on a solo honeymoon after her marriage gets cancelled. (146 mins.)"
    },
    "2": {
        "Votes": "39,919",
        "Movie Name": "Bhaag Milkha Bhaag",
        "Director": "Director: Rakeysh Omprakash Mehra",
        "Rating": "8.3",
        "Year": 2013,
        "Genre": "Drama",
        "RunTime": "The truth behind the ascension of Milkha \"The Flying Sikh\" Singh who was scarred because of the India-Pakistan partition. (186 mins.)"
    },
    "3": {
        "Votes": "52,476",
        "Movie Name": "Barfi!",
        "Director": "Director: Anurag Basu",
        "Rating": "8.2",
        "Year": 2012,
        "Genre": "Family",
        "RunTime": "Three young people learn that love can neither be defined nor contained by society's norms of normal and abnormal. (151 mins.)"
    },
    "4": {
        "Votes": "41,890",
        "Movie Name": "Zindagi Na Milegi Dobara",
        "Director": "Director: Zoya Akhtar",
        "Rating": "8.1",
        "Year": 2011,
        "Genre": "Drama",
        "RunTime": "Three friends decide to turn their fantasy vacation into reality after one of their number becomes engaged. (155 mins.)"
    },
    "5": {
        "Votes": "19,817",
        "Movie Name": "Dabangg",
        "Director": "Director: Abhinav Singh Kashyap",
        "Rating": "6.3",
        "Year": 2010,
        "Genre": "Family",
        "RunTime": "A corrupt police officer faces challenges from his family, gangsters and politicians. (126 mins.)"
    },
    "6": {
        "Votes": "202,476",
        "Movie Name": "3 Idiots",
        "Director": "Director: Rajkumar Hirani",
        "Rating": "8.4",
        "Year": 2009,
        "Genre": "Musical",
        "RunTime": "Two friends are searching for their long lost companion. They revisit their college days and recall the memories of their friend who inspired them to think differently, even as the rest of the world called them \"idiots\". (170 mins.)"
    },
    "7": {
        "Votes": "17,961",
        "Movie Name": "Jodhaa Akbar",
        "Director": "Director: Ashutosh Gowariker",
        "Rating": "7.6",
        "Year": 2008,
        "Genre": "Drama",
        "RunTime": "A sixteenth century love story about a marriage of alliance that gave birth to true love between a great Mughal emperor, Akbar, and a Rajput princess, Jodha. (213 mins.)"
    },
    "8": {
        "Votes": "82,920",
        "Movie Name": "Like Stars on Earth",
        "Director": "Director: Aamir Khan",
        "Rating": "8.5",
        "Year": 2007,
        "Genre": "Family",
        "RunTime": "An eight-year-old boy is thought to be lazy and a trouble-maker, until the new art teacher has the patience and compassion to discover the real problem behind his struggles in school. (165 mins.)"
    },
    "9": {
        "Votes": "68,800",
        "Movie Name": "Rang De Basanti",
        "Director": "Director: Rakesh Omprakash Mehra",
        "Rating": "8.4",
        "Year": 2006,
        "Genre": "Musical",
        "RunTime": "The story of six young Indians who assist an English Woman to film a documentary on the extremist freedom fighters from their past, and the events that lead them to relive the long forgotten saga of freedom. (157 mins.)"
    },
    "10": {
        "Votes": "22,924",
        "Movie Name": "Black",
        "Director": "Director: Sanjay Leela Bhansali",
        "Rating": "8.3",
        "Year": 2005,
        "Genre": "Musical",
        "RunTime": "The cathartic tale of a deaf, mute and blind girl, and her teacher who brings a ray of light into her world of BLACK. (122 mins.)"
    },
    "11": {
        "Votes": "33,616",
        "Movie Name": "Veer-Zaara",
        "Director": "Director: Yash Chopra",
        "Rating": "7.9",
        "Year": 2004,
        "Genre": "Mystery",
        "RunTime": "The story of the love between Veer Pratap Singh, an Indian, and Zaara Hayaat Khan, a Pakistani...a love so great it knows no boundaries... (192 mins.)"
    },
    "12": {
        "Votes": "12,063",
        "Movie Name": "Koi... Mil Gaya",
        "Director": "Director: Rakesh Roshan",
        "Rating": "7.1",
        "Year": 2003,
        "Genre": "Romance",
        "RunTime": "A developmentally disabled young man tries to continue the work his father did in communicating with extra-terrestrials from outer space, which leads to something miraculous and wonderful. (171 mins.)"
    },
    "13": {
        "Votes": "25,260",
        "Movie Name": "Devdas",
        "Director": "Director: Sanjay Leela Bhansali",
        "Rating": "7.6",
        "Year": 2002,
        "Genre": "Drama",
        "RunTime": "After his wealthy family prohibits him from marrying the woman he is in love with, Devdas Mukherjee's life spirals further and further out of control as he takes up alcohol and a life of vice to numb the pain. (185 mins.)"
    },
    "14": {
        "Votes": "69,447",
        "Movie Name": "Lagaan: Once Upon a Time in India",
        "Director": "Director: Ashutosh Gowariker",
        "Rating": "8.2",
        "Year": 2001,
        "Genre": "Romance",
        "RunTime": "The people of a small village in Victorian India stake their future on a game of cricket against their ruthless British rulers. (224 mins.)"
    },
    "15": {
        "Votes": "7,904",
        "Movie Name": "Kaho Naa... Pyaar Hai",
        "Director": "Director: Rakesh Roshan",
        "Rating": "6.9",
        "Year": 2000,
        "Genre": "Drama",
        "RunTime": "Rich girl loves poor singer and vice versa. But fate has two surprises in store. (172 mins.)"
    },
    "16": {
        "Votes": "9,823",
        "Movie Name": "Straight from the Heart",
        "Director": "Director: Sanjay Leela Bhansali",
        "Rating": "7.6",
        "Year": 1999,
        "Genre": "Romance",
        "RunTime": "Nandini has to choose between Sameer, the man who taught her to fall in love; and Vanraj, the man from whom she learnt how to abide and fulfill promises of love. (188 mins.)"
    },
    "17": {
        "Votes": "31,607",
        "Movie Name": "Kuch Kuch Hota Hai",
        "Director": "Director: Karan Johar",
        "Rating": "7.8",
        "Year": 1998,
        "Genre": "War",
        "RunTime": "During their college years, Anjali was in love with her best-friend Rahul, but he had eyes only for Tina. Years later, Rahul and the now-deceased Tina's eight-year-old daughter attempts to reunite her father and Anjali. (177 mins.)"
    },
    "18": {
        "Votes": "13,960",
        "Movie Name": "Dil To Pagal Hai",
        "Director": "Director: Yash Chopra",
        "Rating": "7.1",
        "Year": 1997,
        "Genre": "Crime",
        "RunTime": "The story of four lovers: Rahul, Ajay, Pooja, and Nisha. This Yash Chopra fantasy explores romance as the days go on, and who live in a really nice dream house while their dreams come true. (179 mins.)"
    },
    "19": {
        "Votes": "4,816",
        "Movie Name": "Raja Hindustani",
        "Director": "Director: Dharmesh Darshan",
        "Rating": "6.1",
        "Year": 1996,
        "Genre": "Drama",
        "RunTime": "India's biggest box office hit of 1996, Raja Hindustani is a musical drama about a poor cab driver who marries a rich woman and the struggles they face after marriage. (165 mins.)"
    },
    "20": {
        "Votes": "42,249",
        "Movie Name": "Dilwale Dulhania Le Jayenge",
        "Director": "Director: Aditya Chopra",
        "Rating": "8.3",
        "Year": 1995,
        "Genre": "Musical",
        "RunTime": "A young man and woman - both of Indian descent but born and raised in Britain - fall in love during a trip to Switzerland. However, the girl's traditional father takes her back to India to fulfill a betrothal promise. (189 mins.)"
    },
    "21": {
        "Votes": "10,981",
        "Movie Name": "Hum Aapke Hain Koun...!",
        "Director": "Director: Sooraj R. Barjatya",
        "Rating": "7.7",
        "Year": 1994,
        "Genre": "Drama",
        "RunTime": "Prem, a top student, is learning the ropes of business under his elder brother Rajesh and his uncle Kailashnath... (206 mins.)"
    },
    "22": {
        "Votes": "3,460",
        "Movie Name": "Hum Hain Rahi Pyar Ke",
        "Director": "Director: Mahesh Bhatt",
        "Rating": "7.5",
        "Year": 1993,
        "Genre": "Musical",
        "RunTime": "Rahul Malhotra (Aamir Khan) is the manager of the heavily in debt family business. He is also the guardian of his dead sister's mischievous kids... (163 mins.)"
    },
    "23": {
        "Votes": "12,347",
        "Movie Name": "Jo Jeeta Wohi Sikandar",
        "Director": "Director: Mansoor Khan",
        "Rating": "8.3",
        "Year": 1992,
        "Genre": "Drama",
        "RunTime": "A rich brat and a poor chap try to woo a girl, with their rivalry culminating in the most prestigious college event - the marathon cycle race. (174 mins.)"
    },
    "24": {
        "Votes": "1,930",
        "Movie Name": "Lamhe",
        "Director": "Director: Yash Chopra",
        "Rating": "7.4",
        "Year": 1991,
        "Genre": "Musical",
        "RunTime": "Viren, on a visit to India, falls in love with Pallavi, who happens to be older than he is. Her age doesn't bother him but as Dai Maa points out... (187 mins.)"
    },
    "25": {
        "Votes": "2,658",
        "Movie Name": "Ghayal",
        "Director": "Director: Rajkumar Santoshi",
        "Rating": "7.6",
        "Year": 1990,
        "Genre": "Romance",
        "RunTime": "Ajay Mehra, an amateur boxer, takes revenge against Balwant Rai, a crooked businessman, after his elder brother is killed. (163 mins.)"
    },
    "26": {
        "Votes": "5,827",
        "Movie Name": "Maine Pyar Kiya",
        "Director": "Director: Sooraj R. Barjatya",
        "Rating": "7.5",
        "Year": 1989,
        "Genre": "Drama",
        "RunTime": "Kishen Kumar Chaudhary and Karan are family like friends, when Kishen's wife Mrs. Kaushalya gives birth to a baby boy Prem... (192 mins.)"
    },
    "27": {
        "Votes": "6,175",
        "Movie Name": "Qayamat Se Qayamat Tak",
        "Director": "Director: Mansoor Khan",
        "Rating": "7.6",
        "Year": 1988,
        "Genre": "Musical",
        "RunTime": "A young man and young woman fall in love against their feuding fathers' wishes; Romeo and Juliet set in contemporary India. (162 mins.)"
    },
    "28": {
        "Votes": "688",
        "Movie Name": "Ram Teri Ganga Maili",
        "Director": "Director: Raj Kapoor",
        "Rating": "6.8",
        "Year": 1987,
        "Genre": "Family",
        "RunTime": "Ganga Singh lives near Gangotri with her brother, Karam. One day she comes to the assistance of a young man... (178 mins.)"
    },
    "29": {
        "Votes": "384",
        "Movie Name": "Sparsh",
        "Director": "Director: Sai Paranjape",
        "Rating": "8.1",
        "Year": 1986,
        "Genre": "Adventure",
        "RunTime": "Anirudh Parmar is blind, but he does not let this impairment bind him down. He runs a school for the blind as a Principal... (145 mins.)"
    },
    "30": {
        "Votes": "1,084",
        "Movie Name": "Ardh Satya",
        "Director": "Director: Govind Nihalani",
        "Rating": "8.2",
        "Year": 1985,
        "Genre": "Drama",
        "RunTime": "A newly appointed police rookie deals with corruption, romance, and brutality. (130 mins.)"
    },
    "31": {
        "Votes": "1,354",
        "Movie Name": "Shakti",
        "Director": "Director: Ramesh Sippy",
        "Rating": "7.8",
        "Year": 1984,
        "Genre": "Family",
        "RunTime": "A scrupulously honest cop refuses kidnappers' demands at grave risk to the life of his son. The son... (166 mins.)"
    },
    "32": {
        "Votes": "370",
        "Movie Name": "Kalyug",
        "Director": "Director: Shyam Benegal",
        "Rating": "7.8",
        "Year": 1983,
        "Genre": "Drama",
        "RunTime": "This movie tells story of fierce and total competition of two related business families in the days... (152 mins.)"
    },
    "33": {
        "Votes": "947",
        "Movie Name": "Khubsoorat",
        "Director": "Director: Hrishikesh Mukherjee",
        "Rating": "7.8",
        "Year": 1982,
        "Genre": "Musical",
        "RunTime": "Nirmala Gupta runs her household and family in a very strict and authoritarian manner. While most of her family resent this... (126 mins.)"
    },
    "34": {
        "Votes": "340",
        "Movie Name": "Junoon",
        "Director": "Director: Shyam Benegal",
        "Rating": "7.6",
        "Year": 1981,
        "Genre": "Romance",
        "RunTime": "Set during the 1857 War of Independence, an obsessed Indian Nawab desires to wed a young Anglo-Indian woman, but the girl's obstinate mother stands between them. (141 mins.)"
    },
    "35": {
        "Votes": "75",
        "Movie Name": "Main Tulsi Tere Aangan Ki",
        "Director": "Director: Raj Khosla",
        "Rating": "7.4",
        "Year": 1980,
        "Genre": "Drama",
        "RunTime": "Thakur Rajnath Singh Chouhan, an aristocrat loves dearly a former prostitute, Tulsi and wishes to marry her... (151 mins.)"
    },
    "36": {
        "Votes": "314",
        "Movie Name": "Bhumika",
        "Director": "Director: Shyam Benegal",
        "Rating": "7.6",
        "Year": 1979,
        "Genre": "Drama",
        "RunTime": "Coming from a poor and destitute family, Urvashi is encouraged by her mother, Shanta (Sualbha Deshpande)... (142 mins.)"
    },
    "37": {
        "Votes": "565",
        "Movie Name": "Mausam",
        "Director": "Director: Gulzar",
        "Rating": "8.1",
        "Year": 1978,
        "Genre": "Musical",
        "RunTime": "A medical student, Amarnath meets and falls in love with the daughter of a herbal medicine healer, Chanda while preparing for his exams in Darjeeling... (156 mins.)"
    },
    "38": {
        "Votes": "5,589",
        "Movie Name": "Deewaar",
        "Director": "Director: Yash Chopra",
        "Rating": "8.2",
        "Year": 1977,
        "Genre": "Comedy",
        "RunTime": "As a youth, Vijay struggles as a dockworker. Eventually, he becomes a leading figure of the underworld, while younger brother, Ravi, is an educated, upright policeman. But in the end, it all comes down to, who does mother love more? (174 mins.)"
    },
    "39": {
        "Votes": "318",
        "Movie Name": "Rajnigandha",
        "Director": "Director: Basu Chatterjee",
        "Rating": "7.5",
        "Year": 1976,
        "Genre": "Drama",
        "RunTime": "A Delhi-based woman in a steady relationship with her boyfriend is forced to come to terms with her true feelings when a job interview in Mumbai brings an old flame back in her life. (110 mins.)"
    },
    "40": {
        "Votes": "44",
        "Movie Name": "Anuraag",
        "Director": "Director: Shakti Samanta",
        "Rating": "7.5",
        "Year": 1975,
        "Genre": "Musical",
        "RunTime": "Bombay-based Rajesh lives a fairly wealthy lifestyle with his parents, mom - a housewife and dad - a rather parsimonious businessman..."
    },
    "41": {
        "Votes": "57",
        "Movie Name": "Be-Imaan",
        "Director": "Director: Sohanlal Kanwar",
        "Rating": "7.4",
        "Year": 1974,
        "Genre": "Romance",
        "RunTime": "Mohan has abandoned his love, on the eve of his engagement. On the run, he turns into a safe-breaker and thief... (133 mins.)"
    },
    "42": {
        "Votes": "10,778",
        "Movie Name": "Anand",
        "Director": "Director: Hrishikesh Mukherjee",
        "Rating": "8.9",
        "Year": 1973,
        "Genre": "Drama",
        "RunTime": "The classic story of a terminally ill man who wishes to live life to the full before the inevitable occurs, as told by his best friend. (122 mins.)"
    },
    "43": {
        "Votes": "232",
        "Movie Name": "Toy",
        "Director": "Director: Chander Vohra",
        "Rating": "7.3",
        "Year": 1972,
        "Genre": "Musical",
        "RunTime": "Thakur Suraj Singh lives a wealthy lifestyle with his wife; three sons - Kishore, his wife Laxmi & two children Pappu and Laali; Vijaykamal... (160 mins.)"
    },
    "44": {
        "Votes": "1,097",
        "Movie Name": "Aradhana",
        "Director": "Director: Shakti Samanta",
        "Rating": "7.7",
        "Year": 1971,
        "Genre": "Romance",
        "RunTime": "Dashing pilot Arun romances Vandana Tripathi, and both in love with each other, become very intimate... (169 mins.)"
    },
    "45": {
        "Votes": "239",
        "Movie Name": "Brahmachari",
        "Director": "Director: Bhappi Sonie",
        "Rating": "6.8",
        "Year": 1970,
        "Genre": "Drama",
        "RunTime": "Of unknown parentage, as he was abandoned as a child, calling himself Brahmachari (Shammi Kapoor) lives in a heavily mortgaged house... (157 mins.)"
    },
    "46": {
        "Votes": "364",
        "Movie Name": "Upkar",
        "Director": "Director: Manoj Kumar",
        "Rating": "7.7",
        "Year": 1969,
        "Genre": "Drama",
        "RunTime": "Cult classic highlighting the contrast between Indian and Western values in the newly independent state. (175 mins.)"
    },
    "47": {
        "Votes": "3,976",
        "Movie Name": "Guide",
        "Director": "Director: Vijay Anand",
        "Rating": "8.6",
        "Year": 1968,
        "Genre": "Family",
        "RunTime": "A Guide falls in love with a married woman, then is jailed for forgery and stealing money from her. (183 mins.)"
    },
    "48": {
        "Votes": "57",
        "Movie Name": "Himalay Ki Godmein",
        "Director": "Director: Vijay Bhatt",
        "Rating": "7.2",
        "Year": 1967,
        "Genre": "Drama",
        "RunTime": "Sunil, son of D.I.G. Shyamlal Mehra, is back in India after successful completion of his medical studies..."
    },
    "49": {
        "Votes": "1,012",
        "Movie Name": "Dosti",
        "Director": "Director: Satyen Bose",
        "Rating": "8.5",
        "Year": 1966,
        "Genre": "Family",
        "RunTime": "Mr. Gupta passes away after an accident while on duty, leaving behind his ailing wife and school-going son... (163 mins.)"
    },
    "50": {
        "Votes": "548",
        "Movie Name": "Bandini",
        "Director": "Director: Bimal Roy",
        "Rating": "7.8",
        "Year": 1965,
        "Genre": "Drama",
        "RunTime": "During the British Raj of the 1930s, a prison-doctor falls in love with a convict who eventually reveals the story of her past and her connection to a freedom fighter. (157 mins.)"
    },
    "51": {
        "Votes": "1,083",
        "Movie Name": "Sahib Bibi Aur Ghulam",
        "Director": "Director: Abrar Alvi",
        "Rating": "8.4",
        "Year": 1964,
        "Genre": "Romance",
        "RunTime": "A lowly servant, Bhootnath gets close to the wife of his employer, and narrates her story through his eyes and perspective. (152 mins.)"
    },
    "52": {
        "Votes": "301",
        "Movie Name": "Jis Desh Men Ganga Behti Hai",
        "Director": "Director: Radhu Karmakar",
        "Rating": "7.3",
        "Year": 1963,
        "Genre": "Action",
        "RunTime": "Orphaned lives a poor lifestyle in Central India and makes living singing songs. One day he comes to the assistance of an injured man... (167 mins.)"
    },
    "53": {
        "Votes": "3,882",
        "Movie Name": "Mughal-E-Azam",
        "Director": "Director: K. Asif",
        "Rating": "8.4",
        "Year": 1962,
        "Genre": "Crime",
        "RunTime": "Inspired by true events, a 16th century prince falls in love with a court dancer and battles with his emperor father. (197 mins.)"
    },
    "54": {
        "Votes": "191",
        "Movie Name": "Sujata",
        "Director": "Director: Bimal Roy",
        "Rating": "7.5",
        "Year": 1961,
        "Genre": "Drama",
        "RunTime": "The movie basically revolves around the caste system that is prevelant in India even to this day. It also raises interesting questions regarding family ties and how deep they can be. (161 mins.)"
    },
    "55": {
        "Votes": "796",
        "Movie Name": "Madhumati",
        "Director": "Director: Bimal Roy",
        "Rating": "8.1",
        "Year": 1960,
        "Genre": "Musical",
        "RunTime": "While driving his car on a rainy night, Anand's car breaks down, and he goes to seek shelter in a nearby house... (110 mins.)"
    },
    "56": {
        "Votes": "4,865",
        "Movie Name": "Mother India",
        "Director": "Director: Mehboob",
        "Rating": "8.1",
        "Year": 1959,
        "Genre": "Romance",
        "RunTime": "In this melodrama, a poverty-stricken woman raises her sons through many trials and tribulations. But no matter the struggles, always sticks to her own moral code. (172 mins.)"
    },
    "57": {
        "Votes": "69",
        "Movie Name": "Jhanak Jhanak Payal Baaje",
        "Director": "Director: V. Shantaram",
        "Rating": "7.3",
        "Year": 1958,
        "Genre": "Drama",
        "RunTime": "Dancer maestro Mangal Maharaj wants his son, Girdhar, who is a dancer par excellence, to enter the once-in-ten-years Nateshwar Utsav dance competition... (143 mins.)"
    },
    "58": {
        "Votes": "82",
        "Movie Name": "Jagriti",
        "Director": "Director: Satyen Bose",
        "Rating": "7.8",
        "Year": 1957,
        "Genre": "Drama",
        "RunTime": "Sent to a hostel as a punishment, a rebellious lad leads a revolt and creates havoc in the institution."
    },
    "59": {
        "Votes": "500",
        "Movie Name": "Boot Polish",
        "Director": "Director: Prakash Arora",
        "Rating": "8.1",
        "Year": 1956,
        "Genre": "Family",
        "RunTime": "Preteen brother and sister, abandoned by their father to care of an abusive prostitute aunt, survive life in an Indian squatters' camp, aided by a friendly cripple, by learning to shine shoes instead of begging. (149 mins.)"
    },
    "60": {
        "Votes": "1,107",
        "Movie Name": "Do Bigha Zamin",
        "Director": "Director: Bimal Roy",
        "Rating": "8.4",
        "Year": 1955,
        "Genre": "Drama",
        "RunTime": "In the hope of earning enough money to pay off his debts and save his land, a poor farmer becomes a rickshaw puller in the Calcutta and faces many difficulties. (131 mins.)"
    }
}

Let's clean the data

  • Replace the comma(,) in Vote Value and change the data type to int
  • Change the Data type for Rating and RunTime

In [68]:
for key,values in result.items():
    values['Votes'] = int(values['Votes'].replace(",",""))
    values['Rating']= float(values['Rating'])
    values['Director']= values['Director'].replace('Director: ','')
    try:
        values['RunTime'] = int(re.findall(r'\d+',values['RunTime'])[-1])
    except TypeError:
        values['RunTime'] = np.NaN
    except IndexError:
        values['RunTime'] = np.NaN

Now let's look at the data and see how it looks like


In [69]:
print(json.dumps((result[0]),indent=4))


{
    "Votes": 17622,
    "Movie Name": "Bajirao Mastani",
    "Director": "Sanjay Leela Bhansali",
    "Rating": 7.2,
    "Year": 2015,
    "Genre": "Drama",
    "RunTime": 158
}

Movie details in dictionary


In [70]:
print(json.dumps((result),indent=4))


{
    "0": {
        "Votes": 17622,
        "Movie Name": "Bajirao Mastani",
        "Director": "Sanjay Leela Bhansali",
        "Rating": 7.2,
        "Year": 2015,
        "Genre": "Drama",
        "RunTime": 158
    },
    "1": {
        "Votes": 39681,
        "Movie Name": "Queen",
        "Director": "Vikas Bahl",
        "Rating": 8.4,
        "Year": 2014,
        "Genre": "Drama",
        "RunTime": 146
    },
    "2": {
        "Votes": 39919,
        "Movie Name": "Bhaag Milkha Bhaag",
        "Director": "Rakeysh Omprakash Mehra",
        "Rating": 8.3,
        "Year": 2013,
        "Genre": "Drama",
        "RunTime": 186
    },
    "3": {
        "Votes": 52476,
        "Movie Name": "Barfi!",
        "Director": "Anurag Basu",
        "Rating": 8.2,
        "Year": 2012,
        "Genre": "Family",
        "RunTime": 151
    },
    "4": {
        "Votes": 41890,
        "Movie Name": "Zindagi Na Milegi Dobara",
        "Director": "Zoya Akhtar",
        "Rating": 8.1,
        "Year": 2011,
        "Genre": "Drama",
        "RunTime": 155
    },
    "5": {
        "Votes": 19817,
        "Movie Name": "Dabangg",
        "Director": "Abhinav Singh Kashyap",
        "Rating": 6.3,
        "Year": 2010,
        "Genre": "Family",
        "RunTime": 126
    },
    "6": {
        "Votes": 202476,
        "Movie Name": "3 Idiots",
        "Director": "Rajkumar Hirani",
        "Rating": 8.4,
        "Year": 2009,
        "Genre": "Musical",
        "RunTime": 170
    },
    "7": {
        "Votes": 17961,
        "Movie Name": "Jodhaa Akbar",
        "Director": "Ashutosh Gowariker",
        "Rating": 7.6,
        "Year": 2008,
        "Genre": "Drama",
        "RunTime": 213
    },
    "8": {
        "Votes": 82920,
        "Movie Name": "Like Stars on Earth",
        "Director": "Aamir Khan",
        "Rating": 8.5,
        "Year": 2007,
        "Genre": "Family",
        "RunTime": 165
    },
    "9": {
        "Votes": 68800,
        "Movie Name": "Rang De Basanti",
        "Director": "Rakesh Omprakash Mehra",
        "Rating": 8.4,
        "Year": 2006,
        "Genre": "Musical",
        "RunTime": 157
    },
    "10": {
        "Votes": 22924,
        "Movie Name": "Black",
        "Director": "Sanjay Leela Bhansali",
        "Rating": 8.3,
        "Year": 2005,
        "Genre": "Musical",
        "RunTime": 122
    },
    "11": {
        "Votes": 33616,
        "Movie Name": "Veer-Zaara",
        "Director": "Yash Chopra",
        "Rating": 7.9,
        "Year": 2004,
        "Genre": "Mystery",
        "RunTime": 192
    },
    "12": {
        "Votes": 12063,
        "Movie Name": "Koi... Mil Gaya",
        "Director": "Rakesh Roshan",
        "Rating": 7.1,
        "Year": 2003,
        "Genre": "Romance",
        "RunTime": 171
    },
    "13": {
        "Votes": 25260,
        "Movie Name": "Devdas",
        "Director": "Sanjay Leela Bhansali",
        "Rating": 7.6,
        "Year": 2002,
        "Genre": "Drama",
        "RunTime": 185
    },
    "14": {
        "Votes": 69447,
        "Movie Name": "Lagaan: Once Upon a Time in India",
        "Director": "Ashutosh Gowariker",
        "Rating": 8.2,
        "Year": 2001,
        "Genre": "Romance",
        "RunTime": 224
    },
    "15": {
        "Votes": 7904,
        "Movie Name": "Kaho Naa... Pyaar Hai",
        "Director": "Rakesh Roshan",
        "Rating": 6.9,
        "Year": 2000,
        "Genre": "Drama",
        "RunTime": 172
    },
    "16": {
        "Votes": 9823,
        "Movie Name": "Straight from the Heart",
        "Director": "Sanjay Leela Bhansali",
        "Rating": 7.6,
        "Year": 1999,
        "Genre": "Romance",
        "RunTime": 188
    },
    "17": {
        "Votes": 31607,
        "Movie Name": "Kuch Kuch Hota Hai",
        "Director": "Karan Johar",
        "Rating": 7.8,
        "Year": 1998,
        "Genre": "War",
        "RunTime": 177
    },
    "18": {
        "Votes": 13960,
        "Movie Name": "Dil To Pagal Hai",
        "Director": "Yash Chopra",
        "Rating": 7.1,
        "Year": 1997,
        "Genre": "Crime",
        "RunTime": 179
    },
    "19": {
        "Votes": 4816,
        "Movie Name": "Raja Hindustani",
        "Director": "Dharmesh Darshan",
        "Rating": 6.1,
        "Year": 1996,
        "Genre": "Drama",
        "RunTime": 165
    },
    "20": {
        "Votes": 42249,
        "Movie Name": "Dilwale Dulhania Le Jayenge",
        "Director": "Aditya Chopra",
        "Rating": 8.3,
        "Year": 1995,
        "Genre": "Musical",
        "RunTime": 189
    },
    "21": {
        "Votes": 10981,
        "Movie Name": "Hum Aapke Hain Koun...!",
        "Director": "Sooraj R. Barjatya",
        "Rating": 7.7,
        "Year": 1994,
        "Genre": "Drama",
        "RunTime": 206
    },
    "22": {
        "Votes": 3460,
        "Movie Name": "Hum Hain Rahi Pyar Ke",
        "Director": "Mahesh Bhatt",
        "Rating": 7.5,
        "Year": 1993,
        "Genre": "Musical",
        "RunTime": 163
    },
    "23": {
        "Votes": 12347,
        "Movie Name": "Jo Jeeta Wohi Sikandar",
        "Director": "Mansoor Khan",
        "Rating": 8.3,
        "Year": 1992,
        "Genre": "Drama",
        "RunTime": 174
    },
    "24": {
        "Votes": 1930,
        "Movie Name": "Lamhe",
        "Director": "Yash Chopra",
        "Rating": 7.4,
        "Year": 1991,
        "Genre": "Musical",
        "RunTime": 187
    },
    "25": {
        "Votes": 2658,
        "Movie Name": "Ghayal",
        "Director": "Rajkumar Santoshi",
        "Rating": 7.6,
        "Year": 1990,
        "Genre": "Romance",
        "RunTime": 163
    },
    "26": {
        "Votes": 5827,
        "Movie Name": "Maine Pyar Kiya",
        "Director": "Sooraj R. Barjatya",
        "Rating": 7.5,
        "Year": 1989,
        "Genre": "Drama",
        "RunTime": 192
    },
    "27": {
        "Votes": 6175,
        "Movie Name": "Qayamat Se Qayamat Tak",
        "Director": "Mansoor Khan",
        "Rating": 7.6,
        "Year": 1988,
        "Genre": "Musical",
        "RunTime": 162
    },
    "28": {
        "Votes": 688,
        "Movie Name": "Ram Teri Ganga Maili",
        "Director": "Raj Kapoor",
        "Rating": 6.8,
        "Year": 1987,
        "Genre": "Family",
        "RunTime": 178
    },
    "29": {
        "Votes": 384,
        "Movie Name": "Sparsh",
        "Director": "Sai Paranjape",
        "Rating": 8.1,
        "Year": 1986,
        "Genre": "Adventure",
        "RunTime": 145
    },
    "30": {
        "Votes": 1084,
        "Movie Name": "Ardh Satya",
        "Director": "Govind Nihalani",
        "Rating": 8.2,
        "Year": 1985,
        "Genre": "Drama",
        "RunTime": 130
    },
    "31": {
        "Votes": 1354,
        "Movie Name": "Shakti",
        "Director": "Ramesh Sippy",
        "Rating": 7.8,
        "Year": 1984,
        "Genre": "Family",
        "RunTime": 166
    },
    "32": {
        "Votes": 370,
        "Movie Name": "Kalyug",
        "Director": "Shyam Benegal",
        "Rating": 7.8,
        "Year": 1983,
        "Genre": "Drama",
        "RunTime": 152
    },
    "33": {
        "Votes": 947,
        "Movie Name": "Khubsoorat",
        "Director": "Hrishikesh Mukherjee",
        "Rating": 7.8,
        "Year": 1982,
        "Genre": "Musical",
        "RunTime": 126
    },
    "34": {
        "Votes": 340,
        "Movie Name": "Junoon",
        "Director": "Shyam Benegal",
        "Rating": 7.6,
        "Year": 1981,
        "Genre": "Romance",
        "RunTime": 141
    },
    "35": {
        "Votes": 75,
        "Movie Name": "Main Tulsi Tere Aangan Ki",
        "Director": "Raj Khosla",
        "Rating": 7.4,
        "Year": 1980,
        "Genre": "Drama",
        "RunTime": 151
    },
    "36": {
        "Votes": 314,
        "Movie Name": "Bhumika",
        "Director": "Shyam Benegal",
        "Rating": 7.6,
        "Year": 1979,
        "Genre": "Drama",
        "RunTime": 142
    },
    "37": {
        "Votes": 565,
        "Movie Name": "Mausam",
        "Director": "Gulzar",
        "Rating": 8.1,
        "Year": 1978,
        "Genre": "Musical",
        "RunTime": 156
    },
    "38": {
        "Votes": 5589,
        "Movie Name": "Deewaar",
        "Director": "Yash Chopra",
        "Rating": 8.2,
        "Year": 1977,
        "Genre": "Comedy",
        "RunTime": 174
    },
    "39": {
        "Votes": 318,
        "Movie Name": "Rajnigandha",
        "Director": "Basu Chatterjee",
        "Rating": 7.5,
        "Year": 1976,
        "Genre": "Drama",
        "RunTime": 110
    },
    "40": {
        "Votes": 44,
        "Movie Name": "Anuraag",
        "Director": "Shakti Samanta",
        "Rating": 7.5,
        "Year": 1975,
        "Genre": "Musical",
        "RunTime": NaN
    },
    "41": {
        "Votes": 57,
        "Movie Name": "Be-Imaan",
        "Director": "Sohanlal Kanwar",
        "Rating": 7.4,
        "Year": 1974,
        "Genre": "Romance",
        "RunTime": 133
    },
    "42": {
        "Votes": 10778,
        "Movie Name": "Anand",
        "Director": "Hrishikesh Mukherjee",
        "Rating": 8.9,
        "Year": 1973,
        "Genre": "Drama",
        "RunTime": 122
    },
    "43": {
        "Votes": 232,
        "Movie Name": "Toy",
        "Director": "Chander Vohra",
        "Rating": 7.3,
        "Year": 1972,
        "Genre": "Musical",
        "RunTime": 160
    },
    "44": {
        "Votes": 1097,
        "Movie Name": "Aradhana",
        "Director": "Shakti Samanta",
        "Rating": 7.7,
        "Year": 1971,
        "Genre": "Romance",
        "RunTime": 169
    },
    "45": {
        "Votes": 239,
        "Movie Name": "Brahmachari",
        "Director": "Bhappi Sonie",
        "Rating": 6.8,
        "Year": 1970,
        "Genre": "Drama",
        "RunTime": 157
    },
    "46": {
        "Votes": 364,
        "Movie Name": "Upkar",
        "Director": "Manoj Kumar",
        "Rating": 7.7,
        "Year": 1969,
        "Genre": "Drama",
        "RunTime": 175
    },
    "47": {
        "Votes": 3976,
        "Movie Name": "Guide",
        "Director": "Vijay Anand",
        "Rating": 8.6,
        "Year": 1968,
        "Genre": "Family",
        "RunTime": 183
    },
    "48": {
        "Votes": 57,
        "Movie Name": "Himalay Ki Godmein",
        "Director": "Vijay Bhatt",
        "Rating": 7.2,
        "Year": 1967,
        "Genre": "Drama",
        "RunTime": NaN
    },
    "49": {
        "Votes": 1012,
        "Movie Name": "Dosti",
        "Director": "Satyen Bose",
        "Rating": 8.5,
        "Year": 1966,
        "Genre": "Family",
        "RunTime": 163
    },
    "50": {
        "Votes": 548,
        "Movie Name": "Bandini",
        "Director": "Bimal Roy",
        "Rating": 7.8,
        "Year": 1965,
        "Genre": "Drama",
        "RunTime": 157
    },
    "51": {
        "Votes": 1083,
        "Movie Name": "Sahib Bibi Aur Ghulam",
        "Director": "Abrar Alvi",
        "Rating": 8.4,
        "Year": 1964,
        "Genre": "Romance",
        "RunTime": 152
    },
    "52": {
        "Votes": 301,
        "Movie Name": "Jis Desh Men Ganga Behti Hai",
        "Director": "Radhu Karmakar",
        "Rating": 7.3,
        "Year": 1963,
        "Genre": "Action",
        "RunTime": 167
    },
    "53": {
        "Votes": 3882,
        "Movie Name": "Mughal-E-Azam",
        "Director": "K. Asif",
        "Rating": 8.4,
        "Year": 1962,
        "Genre": "Crime",
        "RunTime": 197
    },
    "54": {
        "Votes": 191,
        "Movie Name": "Sujata",
        "Director": "Bimal Roy",
        "Rating": 7.5,
        "Year": 1961,
        "Genre": "Drama",
        "RunTime": 161
    },
    "55": {
        "Votes": 796,
        "Movie Name": "Madhumati",
        "Director": "Bimal Roy",
        "Rating": 8.1,
        "Year": 1960,
        "Genre": "Musical",
        "RunTime": 110
    },
    "56": {
        "Votes": 4865,
        "Movie Name": "Mother India",
        "Director": "Mehboob",
        "Rating": 8.1,
        "Year": 1959,
        "Genre": "Romance",
        "RunTime": 172
    },
    "57": {
        "Votes": 69,
        "Movie Name": "Jhanak Jhanak Payal Baaje",
        "Director": "V. Shantaram",
        "Rating": 7.3,
        "Year": 1958,
        "Genre": "Drama",
        "RunTime": 143
    },
    "58": {
        "Votes": 82,
        "Movie Name": "Jagriti",
        "Director": "Satyen Bose",
        "Rating": 7.8,
        "Year": 1957,
        "Genre": "Drama",
        "RunTime": NaN
    },
    "59": {
        "Votes": 500,
        "Movie Name": "Boot Polish",
        "Director": "Prakash Arora",
        "Rating": 8.1,
        "Year": 1956,
        "Genre": "Family",
        "RunTime": 149
    },
    "60": {
        "Votes": 1107,
        "Movie Name": "Do Bigha Zamin",
        "Director": "Bimal Roy",
        "Rating": 8.4,
        "Year": 1955,
        "Genre": "Drama",
        "RunTime": 131
    }
}

Data in Pandas Dataframe


In [71]:
# create dataframe
df = pd.DataFrame.from_dict(result,orient='index')
df = df[['Year', 'Movie Name', 'Rating', 'Votes','Genre','RunTime','Director']]
df.index = np.arange(1, 62)
df.head(10)


Out[71]:
Year Movie Name Rating Votes Genre RunTime Director
1 2015 Bajirao Mastani 7.2 17622 Drama 158 Sanjay Leela Bhansali
2 2014 Queen 8.4 39681 Drama 146 Vikas Bahl
3 2013 Bhaag Milkha Bhaag 8.3 39919 Drama 186 Rakeysh Omprakash Mehra
4 2012 Barfi! 8.2 52476 Family 151 Anurag Basu
5 2011 Zindagi Na Milegi Dobara 8.1 41890 Drama 155 Zoya Akhtar
6 2010 Dabangg 6.3 19817 Family 126 Abhinav Singh Kashyap
7 2009 3 Idiots 8.4 202476 Musical 170 Rajkumar Hirani
8 2008 Jodhaa Akbar 7.6 17961 Drama 213 Ashutosh Gowariker
9 2007 Like Stars on Earth 8.5 82920 Family 165 Aamir Khan
10 2006 Rang De Basanti 8.4 68800 Musical 157 Rakesh Omprakash Mehra

Let's use some of the Pandas functions now and start the Analysis

No. of rows with missing values(NaN)


In [72]:
df


Out[72]:
Year Movie Name Rating Votes Genre RunTime Director
1 2015 Bajirao Mastani 7.2 17622 Drama 158 Sanjay Leela Bhansali
2 2014 Queen 8.4 39681 Drama 146 Vikas Bahl
3 2013 Bhaag Milkha Bhaag 8.3 39919 Drama 186 Rakeysh Omprakash Mehra
4 2012 Barfi! 8.2 52476 Family 151 Anurag Basu
5 2011 Zindagi Na Milegi Dobara 8.1 41890 Drama 155 Zoya Akhtar
6 2010 Dabangg 6.3 19817 Family 126 Abhinav Singh Kashyap
7 2009 3 Idiots 8.4 202476 Musical 170 Rajkumar Hirani
8 2008 Jodhaa Akbar 7.6 17961 Drama 213 Ashutosh Gowariker
9 2007 Like Stars on Earth 8.5 82920 Family 165 Aamir Khan
10 2006 Rang De Basanti 8.4 68800 Musical 157 Rakesh Omprakash Mehra
11 2005 Black 8.3 22924 Musical 122 Sanjay Leela Bhansali
12 2004 Veer-Zaara 7.9 33616 Mystery 192 Yash Chopra
13 2003 Koi... Mil Gaya 7.1 12063 Romance 171 Rakesh Roshan
14 2002 Devdas 7.6 25260 Drama 185 Sanjay Leela Bhansali
15 2001 Lagaan: Once Upon a Time in India 8.2 69447 Romance 224 Ashutosh Gowariker
16 2000 Kaho Naa... Pyaar Hai 6.9 7904 Drama 172 Rakesh Roshan
17 1999 Straight from the Heart 7.6 9823 Romance 188 Sanjay Leela Bhansali
18 1998 Kuch Kuch Hota Hai 7.8 31607 War 177 Karan Johar
19 1997 Dil To Pagal Hai 7.1 13960 Crime 179 Yash Chopra
20 1996 Raja Hindustani 6.1 4816 Drama 165 Dharmesh Darshan
21 1995 Dilwale Dulhania Le Jayenge 8.3 42249 Musical 189 Aditya Chopra
22 1994 Hum Aapke Hain Koun...! 7.7 10981 Drama 206 Sooraj R. Barjatya
23 1993 Hum Hain Rahi Pyar Ke 7.5 3460 Musical 163 Mahesh Bhatt
24 1992 Jo Jeeta Wohi Sikandar 8.3 12347 Drama 174 Mansoor Khan
25 1991 Lamhe 7.4 1930 Musical 187 Yash Chopra
26 1990 Ghayal 7.6 2658 Romance 163 Rajkumar Santoshi
27 1989 Maine Pyar Kiya 7.5 5827 Drama 192 Sooraj R. Barjatya
28 1988 Qayamat Se Qayamat Tak 7.6 6175 Musical 162 Mansoor Khan
29 1987 Ram Teri Ganga Maili 6.8 688 Family 178 Raj Kapoor
30 1986 Sparsh 8.1 384 Adventure 145 Sai Paranjape
... ... ... ... ... ... ... ...
32 1984 Shakti 7.8 1354 Family 166 Ramesh Sippy
33 1983 Kalyug 7.8 370 Drama 152 Shyam Benegal
34 1982 Khubsoorat 7.8 947 Musical 126 Hrishikesh Mukherjee
35 1981 Junoon 7.6 340 Romance 141 Shyam Benegal
36 1980 Main Tulsi Tere Aangan Ki 7.4 75 Drama 151 Raj Khosla
37 1979 Bhumika 7.6 314 Drama 142 Shyam Benegal
38 1978 Mausam 8.1 565 Musical 156 Gulzar
39 1977 Deewaar 8.2 5589 Comedy 174 Yash Chopra
40 1976 Rajnigandha 7.5 318 Drama 110 Basu Chatterjee
41 1975 Anuraag 7.5 44 Musical NaN Shakti Samanta
42 1974 Be-Imaan 7.4 57 Romance 133 Sohanlal Kanwar
43 1973 Anand 8.9 10778 Drama 122 Hrishikesh Mukherjee
44 1972 Toy 7.3 232 Musical 160 Chander Vohra
45 1971 Aradhana 7.7 1097 Romance 169 Shakti Samanta
46 1970 Brahmachari 6.8 239 Drama 157 Bhappi Sonie
47 1969 Upkar 7.7 364 Drama 175 Manoj Kumar
48 1968 Guide 8.6 3976 Family 183 Vijay Anand
49 1967 Himalay Ki Godmein 7.2 57 Drama NaN Vijay Bhatt
50 1966 Dosti 8.5 1012 Family 163 Satyen Bose
51 1965 Bandini 7.8 548 Drama 157 Bimal Roy
52 1964 Sahib Bibi Aur Ghulam 8.4 1083 Romance 152 Abrar Alvi
53 1963 Jis Desh Men Ganga Behti Hai 7.3 301 Action 167 Radhu Karmakar
54 1962 Mughal-E-Azam 8.4 3882 Crime 197 K. Asif
55 1961 Sujata 7.5 191 Drama 161 Bimal Roy
56 1960 Madhumati 8.1 796 Musical 110 Bimal Roy
57 1959 Mother India 8.1 4865 Romance 172 Mehboob
58 1958 Jhanak Jhanak Payal Baaje 7.3 69 Drama 143 V. Shantaram
59 1957 Jagriti 7.8 82 Drama NaN Satyen Bose
60 1956 Boot Polish 8.1 500 Family 149 Prakash Arora
61 1955 Do Bigha Zamin 8.4 1107 Drama 131 Bimal Roy

61 rows × 7 columns


In [73]:
df[df['RunTime'].isnull()==True]


Out[73]:
Year Movie Name Rating Votes Genre RunTime Director
41 1975 Anuraag 7.5 44 Musical NaN Shakti Samanta
49 1967 Himalay Ki Godmein 7.2 57 Drama NaN Vijay Bhatt
59 1957 Jagriti 7.8 82 Drama NaN Satyen Bose

In [74]:
nans = df.shape[0] - df.dropna().shape[0]
print('%d rows have missing values' % nans)


3 rows have missing values

Replace NaN with Mean


In [75]:
df=df.fillna(int(df['RunTime'].mean()))
df.ix[[41, 49,59]]


Out[75]:
Year Movie Name Rating Votes Genre RunTime Director
41 1975 Anuraag 7.5 44 Musical 162 Shakti Samanta
49 1967 Himalay Ki Godmein 7.2 57 Drama 162 Vijay Bhatt
59 1957 Jagriti 7.8 82 Drama 162 Satyen Bose

In [76]:
df.info()


<class 'pandas.core.frame.DataFrame'>
Int64Index: 61 entries, 1 to 61
Data columns (total 7 columns):
Year          61 non-null int64
Movie Name    61 non-null object
Rating        61 non-null float64
Votes         61 non-null int64
Genre         61 non-null object
RunTime       61 non-null float64
Director      61 non-null object
dtypes: float64(2), int64(2), object(3)
memory usage: 3.8+ KB

Movies with Highest Ratings

The top five movies with Maximum Rating since 1955


In [77]:
#Highest Rating Movies
df1=df.sort_values('Rating',ascending=[False]).head(5)
df1.index = np.arange(1, 6)
df1


Out[77]:
Year Movie Name Rating Votes Genre RunTime Director
1 1973 Anand 8.9 10778 Drama 122 Hrishikesh Mukherjee
2 1968 Guide 8.6 3976 Family 183 Vijay Anand
3 1966 Dosti 8.5 1012 Family 163 Satyen Bose
4 2007 Like Stars on Earth 8.5 82920 Family 165 Aamir Khan
5 1955 Do Bigha Zamin 8.4 1107 Drama 131 Bimal Roy

Rating Trend for Best Movies from last 65 years


In [93]:
df.plot(x=df.Year,y=['Rating']);


Movies with Lowest Ratings


In [79]:
df1=df.sort_values('Rating',ascending=[True]).head(5)
df1.index = np.arange(1, 6)
df1


Out[79]:
Year Movie Name Rating Votes Genre RunTime Director
1 1996 Raja Hindustani 6.1 4816 Drama 165 Dharmesh Darshan
2 2010 Dabangg 6.3 19817 Family 126 Abhinav Singh Kashyap
3 1987 Ram Teri Ganga Maili 6.8 688 Family 178 Raj Kapoor
4 1970 Brahmachari 6.8 239 Drama 157 Bhappi Sonie
5 2000 Kaho Naa... Pyaar Hai 6.9 7904 Drama 172 Rakesh Roshan

Movies with Maximum Run time

Top 10 movies with maximum Run time


In [80]:
#Movies with maximum Run Time
df1=df.sort_values(['RunTime'],ascending=[False]).head(10)
df1.index = np.arange(1, 11)
df1


Out[80]:
Year Movie Name Rating Votes Genre RunTime Director
1 2001 Lagaan: Once Upon a Time in India 8.2 69447 Romance 224 Ashutosh Gowariker
2 2008 Jodhaa Akbar 7.6 17961 Drama 213 Ashutosh Gowariker
3 1994 Hum Aapke Hain Koun...! 7.7 10981 Drama 206 Sooraj R. Barjatya
4 1962 Mughal-E-Azam 8.4 3882 Crime 197 K. Asif
5 1989 Maine Pyar Kiya 7.5 5827 Drama 192 Sooraj R. Barjatya
6 2004 Veer-Zaara 7.9 33616 Mystery 192 Yash Chopra
7 1995 Dilwale Dulhania Le Jayenge 8.3 42249 Musical 189 Aditya Chopra
8 1999 Straight from the Heart 7.6 9823 Romance 188 Sanjay Leela Bhansali
9 1991 Lamhe 7.4 1930 Musical 187 Yash Chopra
10 2013 Bhaag Milkha Bhaag 8.3 39919 Drama 186 Rakeysh Omprakash Mehra

Best Movie Run time

Let's plot a graph to see the movie run time trend from 1955 thru 2015


In [81]:
df.plot(x=df.Year,y=['RunTime']);


Mean of the Movie Run Time


In [82]:
df['RunTime'].mean()


Out[82]:
162.2295081967213

Perform some analysis on the ratings of all the Best won movies

  • No. of Movies Greater than IMDB 7 ratings

In [83]:
df[(df['Rating']>=7)]['Rating'].count()


Out[83]:
56

Movie Ratings Visualization using Bar Graph


In [84]:
import seaborn as sns
sns.set_style("white")
Rating_Histdic = defaultdict(dict)

Rating_Histdic['Btwn 6&7'] = df[(df['Rating']>=6)&(df['Rating']<7)]['Rating'].count()
Rating_Histdic['GTEQ 8'] = df[(df['Rating']>=8)]['Rating'].count()
Rating_Histdic['Btwn 7 & 8'] = df[(df['Rating']>=7)&(df['Rating']<8)]['Rating'].count()


plt.bar(range(len(Rating_Histdic)), Rating_Histdic.values(), align='center',color='b',width=0.4)
plt.xticks(range(len(Rating_Histdic)), Rating_Histdic.keys(), rotation=25);


Percentage distribution of the Ratings in a Pie-Chart


In [85]:
sns.set_style("white")
sns.set_context("notebook")

Rating_Hist = []
import numpy as np
    
Rating_Hist.append(Rating_Histdic['Btwn 6&7'])
Rating_Hist.append(Rating_Histdic['GTEQ 8'])
Rating_Hist.append(Rating_Histdic['Btwn 7 & 8'])

labels = ['Btwn 6&7', 'GTEQ 8', 'Btwn 7 & 8']
colors = ['red', 'c', 'lightgreen']

plt.pie(Rating_Hist,labels=labels, colors=colors,autopct='%1.1f%%', shadow=True, startangle=90);
plt.figure(figsize=(8, 6));


<matplotlib.figure.Figure at 0x10fd604e0>

Best Picture by Genre

Let's analyze the Genre for the best won movies


In [86]:
Category=Counter(datatbl['data']['Genre'])
df1 = pd.DataFrame.from_dict(Category,orient='index')
df1 = df1.sort_values([0],ascending=[False]).head(5)
df1.plot(kind='barh',color=['g','c','m']);


Directors whose movie won more than once for Best Film

Movies for which they won best film award


In [87]:
df['freq']= df.groupby('Director')['Director'].transform('count')
df2=df[df['freq']>1]
del df2['freq']

In [88]:
df2.groupby(['Director','Year', 'Movie Name',
             'Rating', 'Genre','Votes','RunTime']).count()[0:100]


Out[88]:
Director Year Movie Name Rating Genre Votes RunTime
Ashutosh Gowariker 2001 Lagaan: Once Upon a Time in India 8.2 Romance 69447 224
2008 Jodhaa Akbar 7.6 Drama 17961 213
Bimal Roy 1955 Do Bigha Zamin 8.4 Drama 1107 131
1960 Madhumati 8.1 Musical 796 110
1961 Sujata 7.5 Drama 191 161
1965 Bandini 7.8 Drama 548 157
Hrishikesh Mukherjee 1973 Anand 8.9 Drama 10778 122
1982 Khubsoorat 7.8 Musical 947 126
Mansoor Khan 1988 Qayamat Se Qayamat Tak 7.6 Musical 6175 162
1992 Jo Jeeta Wohi Sikandar 8.3 Drama 12347 174
Rakesh Roshan 2000 Kaho Naa... Pyaar Hai 6.9 Drama 7904 172
2003 Koi... Mil Gaya 7.1 Romance 12063 171
Sanjay Leela Bhansali 1999 Straight from the Heart 7.6 Romance 9823 188
2002 Devdas 7.6 Drama 25260 185
2005 Black 8.3 Musical 22924 122
2015 Bajirao Mastani 7.2 Drama 17622 158
Satyen Bose 1957 Jagriti 7.8 Drama 82 162
1966 Dosti 8.5 Family 1012 163
Shakti Samanta 1971 Aradhana 7.7 Romance 1097 169
1975 Anuraag 7.5 Musical 44 162
Shyam Benegal 1979 Bhumika 7.6 Drama 314 142
1981 Junoon 7.6 Romance 340 141
1983 Kalyug 7.8 Drama 370 152
Sooraj R. Barjatya 1989 Maine Pyar Kiya 7.5 Drama 5827 192
1994 Hum Aapke Hain Koun...! 7.7 Drama 10981 206
Yash Chopra 1977 Deewaar 8.2 Comedy 5589 174
1991 Lamhe 7.4 Musical 1930 187
1997 Dil To Pagal Hai 7.1 Crime 13960 179
2004 Veer-Zaara 7.9 Mystery 33616 192

Conclusion :

  • Movies with Ratings greater than 7
  • Run time more than 2hrs
  • Category Drama & Musical are most likely to be selcted for Best Picture