- Selecting a dataset
- Exploring the dataset to identify what kinds of questions can be answered using the dataset
- Identifying one research question
- Using pandas methods to explore the dataset - this also involves using visualization techniques using matplotlib
- Reporting findings/analyses
- Presenting the work in the given presentation template
The dataset is available for download here - https://grouplens.org/datasets/movielens/20m/
Description about the dataset, as shown on the website is below:
This dataset (ml-20m) describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 20000263 ratings and 465564 tag applications across 27278 movies. These data were created by 138493 users between January 09, 1995 and March 31, 2015. This dataset was generated on October 17, 2016.
Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.
The data are contained in six files, genome-scores.csv, genome-tags.csv, links.csv, movies.csv, ratings.csv and tags.csv. More details about the contents and use of all these files follows.
This and other GroupLens data sets are publicly available for download at http://grouplens.org/datasets/.
In [2]:
# The first step is to import the dataset into a pandas dataframe.
import pandas as pd
#path = 'C:/Users/hrao/Documents/Personal/HK/Python/ml-20m/ml-20m/'
path = '/Users/Harish/Documents/HK_Work/Python/ml-20m/'
movies = pd.read_csv(path+'movies.csv')
movies.shape
Out[2]:
In [3]:
tags = pd.read_csv(path+'tags.csv')
tags.shape
Out[3]:
In [4]:
ratings = pd.read_csv(path+'ratings.csv')
ratings.shape
Out[4]:
In [5]:
links = pd.read_csv(path+'links.csv')
links.shape
Out[5]:
In [6]:
movies.head()
Out[6]:
In [7]:
tags.head()
Out[7]:
In [8]:
ratings.head()
Out[8]:
In [9]:
links.head()
Out[9]:
- Is there a correlation or a trend between the year of release of a movie and the genre?
- Which genres were more dominant in each decade of the range available in the dataset?
- Do science fiction movies tend to be rated more highly than other movie genres?
In [10]:
# List of genres as a Python list
genres = ['Action','Adventure','Animation','Children','Comedy','Crime','Documentary','Drama','Fantasy','Film-Noir','Horror','Musical','Mystery','Romance','Sci-Fi','Thriller','War','Western']
In [11]:
genres_rating_list = []
In [12]:
# The loop reads each element of the above list
# For each iteration, one genre is selected from the movies data frame
# This selection of the data frame is then merged with the rating data frame to get the rating for that genre
# Once the new merged data frame is created, we use the mean function to get the mean rating for the genre
# The genre and the corresponding mean rating are then appended to the genres_rating Data Frame
# The entire looping takes long - can certainly be optimized for performance
for i in range(len(genres)):
fil = genres[i]+'_filter'
mov = genres[i]+'_movies'
rat = genres[i]+'_ratings'
rat_mean = rat+'_mean'
fil = movies['genres'].str.contains(genres[i])
mov = movies[fil]
rat = mov.merge(ratings, on='movieId', how='inner')
rat_mean = round(rat['rating'].mean(), 2)
#print(genres[i], round(rat_mean,2))
genres_rating_list.append(rat_mean)
In [13]:
df = {'Genre':genres, 'Genres Mean Rating':genres_rating_list}
In [14]:
genres_rating = pd.DataFrame(df)
In [15]:
genres_rating
Out[15]:
In [16]:
genres_rating['Genres Standard Deviation'] = genres_rating['Genres Mean Rating'].std()
In [17]:
genres_rating['Mean'] = genres_rating['Genres Mean Rating'].mean()
genres_rating['Zero'] = 0
In [18]:
genres_rating
Out[18]:
In [19]:
overall_mean = round(genres_rating['Genres Mean Rating'].mean(), 2)
overall_std = round(genres_rating['Genres Mean Rating'].std(),2)
scifi_rating = genres_rating[genres_rating['Genre'] == 'Sci-Fi']['Genres Mean Rating']
In [20]:
print(overall_mean)
In [21]:
print(overall_std)
In [22]:
print(scifi_rating)
In [23]:
genres_rating['Diff from Mean'] = genres_rating['Genres Mean Rating'] - overall_mean
In [24]:
genres_rating
Out[24]:
In [25]:
genre_list = list(genres_rating['Genre'])
In [26]:
genres_rating_list = list(genres_rating['Genres Mean Rating'])
genres_diff_list = list(genres_rating['Diff from Mean'])
In [27]:
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 10))
ax1 = plt.subplot(2,1,1)
x = [x for x in range(0, 18)]
xticks_genre_list = genre_list
y = genres_rating_list
plt.xticks(range(len(x)), xticks_genre_list)
plt.scatter(x,y, color='g')
plt.plot(x, genres_rating['Mean'], color="red")
plt.autoscale(tight=True)
#plt.rcParams["figure.figsize"] = (10,2)
plt.title('Movie ratings by genre')
plt.xlabel('Genre')
plt.ylabel('Rating')
plt.ylim(ymax = 4, ymin = 3)
plt.grid(True)
plt.savefig(r'movie-ratings-by-genre.png')
plt.annotate("Sci-Fi Rating",
xy=(14.25,3.5), xycoords='data',
xytext=(14.20, 3.7), textcoords='data',
arrowprops=dict(arrowstyle="->",
connectionstyle="arc3"),
)
for i,j in enumerate( y ):
ax1.annotate( j, ( x[i] + 0.03, y[i] + 0.02))
ax2 = plt.subplot(2,1,2)
x = [x for x in range(0, 18)]
xticks_genre_list = genre_list
y = genres_rating['Diff from Mean']
plt.xticks(range(len(x)), xticks_genre_list)
plt.plot(x,y)
plt.plot(x, genres_rating['Zero'])
plt.autoscale(tight=True)
#plt.rcParams["figure.figsize"] = (10,2)
plt.title('Deviation of each genre\'s rating from the overall mean rating')
plt.xlabel('Genre')
plt.ylabel('Deviation from mean rating')
plt.grid(True)
plt.savefig(r'deviation-from-mean-rating.png')
plt.annotate("Sci-Fi Rating",
xy=(14,-0.13), xycoords='data',
xytext=(14.00, 0.0), textcoords='data',
arrowprops=dict(arrowstyle="->",
connectionstyle="arc3"),
)
plt.show()
The scatter plot shows the mean rating value for each genre. Each genre has a value on the scatter plot for the mean rating value for that genre. Let us now see if the plot is able to help us answer the question above.
The mean rating for Sci-Fi genre is about 3.45. When looking at the plot, we see that there are only three other genres out of 18 genres in total, that have lesser mean ratings than Sci-Fi - Horror, Children and Comedy. The remaining 10 genres have mean ratings higher than Science Fiction.
This gives us enough information to answer the question. Sci-Fi movies do not tend to be rated higher than other genres.
The second plot, a bar plot, shows how much each genre's ratings deviate from the overall mean of ratings. Science Fiction is around -0.13 lower than the mean rating of 3.58, showing lesser deviation than Horror at the lower end and Film-Noir at the higher end.
- Is there a correlation or a trend between the year of release of a movie and the genre?
- Which genres were more dominant in each decade of the range available in the dataset?
In [160]:
# extract year of release of each movie from the title column
# convert the data type of the movie_year column to numeric (from str)
import numpy as np
import re
movies['movie_year'] = movies['title']
movies['movie_year'] = movies['movie_year'].str.extract(r"\(([0-9]+)\)", expand=False)
# creating a new column with just the movie titles
movies['title_only'] = movies['title']
movies['title_only'] = movies['title_only'].str.extract('(.*?)\s*\(', expand=False)
In [161]:
movies['movie_year'].fillna(0, inplace=True)
In [162]:
#Drop all rows containing incorrect year values - such as 0, 6, 69, 500 and -2147483648
movies.drop(movies[movies.movie_year == '0'].index, inplace=True)
movies.drop(movies[movies.movie_year == '6'].index, inplace=True)
movies.drop(movies[movies.movie_year == '06'].index, inplace=True)
movies.drop(movies[movies.movie_year == '69'].index, inplace=True)
movies.drop(movies[movies.movie_year == '500'].index, inplace=True)
movies.drop(movies[movies.movie_year == '-2147483648'].index, inplace=True)
movies.drop(movies[movies.movie_year == 0].index, inplace=True)
movies.drop(movies[movies.movie_year == 6].index, inplace=True)
movies.drop(movies[movies.movie_year == 69].index, inplace=True)
movies.drop(movies[movies.movie_year == 500].index, inplace=True)
movies.drop(movies[movies.movie_year == -2147483648].index, inplace=True)
In [163]:
#convert the string values to numeric
movies['movie_year'] = pd.to_datetime(movies['movie_year'], format='%Y')
Now that we have a move year column, let us list the data types of the columns in the movies data frame.
movie_year is of float64 datat type. We must convert the data type of the movie_year column to int64. Before we go ahead and do that, we must replace all NULL and inifinite entries in the column with zero. If we do not perform this step, we will get the following errror message.
In [164]:
movie_year = pd.DataFrame(movies['title_only'].groupby(movies['movie_year']).count())
In [165]:
movie_year.reset_index(inplace=True)
In [166]:
X=movie_year['movie_year']
Y=movie_year['title_only']
In [167]:
plt.plot_date(X,Y,'bo-')
plt.grid(True)
plt.rcParams["figure.figsize"] = (15,5)
plt.title('Number of movies per year')
plt.xlabel('Years')
plt.ylabel('Number of movies')
plt.xlim('1885-01-01','2020-01-01')
plt.show()
The above plot provides some interesting insight:
In [264]:
movies.head()
Out[264]:
In [415]:
list(movies)
Out[415]:
In [410]:
a = pd.Series(movies.iloc[0])
In [411]:
a
Out[411]:
In [418]:
def flat(str1):
c = pd.DataFrame(columns=list(movies))
for i in range(len(str1)):
#print(str1[i])
if i == 2:
a = str1[i].split('|')
for j in range(len(a)):
c.loc[j] = [str1[0], str1[1], a[j], str1[3], str1[4]]
return c
In [419]:
c = flat(a)
In [420]:
c
Out[420]:
In [ ]: