In [4]:
%matplotlib inline
In [5]:
import datetime
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
In [3]:
!unzip ml-100k.zip
In [6]:
%load ml-100k/README
MovieLens data sets were collected by the GroupLens Research Project at the University of Minnesota.
This data set consists of:
* 100,000 ratings (1-5) from 943 users on 1682 movies.
* Each user has rated at least 20 movies.
* Simple demographic info for the users (age, gender, occupation, zip)
The data was collected through the MovieLens web site (movielens.umn.edu) during the seven-month period from September 19th, 1997 through April 22nd, 1998. This data has been cleaned up - users who had less than 20 ratings or did not have complete demographic information were removed from this data set. Detailed descriptions of the data file can be found at the end of this file.
Neither the University of Minnesota nor any of the researchers involved can guarantee the correctness of the data, its suitability for any particular purpose, or the validity of results based on the use of the data set. The data set may be used for any research purposes under the following conditions:
* The user may not state or imply any endorsement from the
University of Minnesota or the GroupLens Research Group.
* The user must acknowledge the use of the data set in
publications resulting from the use of the data set, and must
send us an electronic or paper copy of those publications.
* The user may not redistribute the data without separate
permission.
* The user may not use this information for any commercial or
revenue-bearing purposes without first obtaining permission
from a faculty member of the GroupLens Research Project at the
University of Minnesota.
If you have any further questions or comments, please contact Jon Herlocker herlocke@cs.umn.edu.
Thanks to Al Borchers for cleaning up this data and writing the accompanying scripts.
Herlocker, J., Konstan, J., Borchers, A., Riedl, J.. An Algorithmic Framework for Performing Collaborative Filtering. Proceedings of the 1999 Conference on Research and Development in Information Retrieval. Aug. 1999.
The GroupLens Research Project is a research group in the Department of Computer Science and Engineering at the University of Minnesota. Members of the GroupLens Research Project are involved in many research projects related to the fields of information filtering, collaborative filtering, and recommender systems. The project is lead by professors John Riedl and Joseph Konstan. The project began to explore automated collaborative filtering in 1992, but is most well known for its world wide trial of an automated collaborative filtering system for Usenet news in 1996. The technology developed in the Usenet trial formed the base for the formation of Net Perceptions, Inc., which was founded by members of GroupLens Research. Since then the project has expanded its scope to research overall information filtering solutions, integrating in content-based methods as well as improving current collaborative filtering technology.
Further information on the GroupLens Research project, including research publications, can be found at the following web site:
http://www.grouplens.org/
GroupLens Research currently operates a movie recommender based on collaborative filtering:
http://www.movielens.org/
Here are brief descriptions of the data.
ml-data.tar.gz -- Compressed tar file. To rebuild the u data files do this: gunzip ml-data.tar.gz tar xvf ml-data.tar mku.sh
u.data -- The full u data set, 100000 ratings by 943 users on 1682 items. Each user has rated at least 20 movies. Users and items are numbered consecutively from 1. The data is randomly ordered. This is a tab separated list of user id | item id | rating | timestamp. The time stamps are unix seconds since 1/1/1970 UTC
u.info -- The number of users, items, and ratings in the u data set.
u.item -- Information about the items (movies); this is a tab separated list of movie id | movie title | release date | video release date | IMDb URL | unknown | Action | Adventure | Animation | Children's | Comedy | Crime | Documentary | Drama | Fantasy | Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | Thriller | War | Western | The last 19 fields are the genres, a 1 indicates the movie is of that genre, a 0 indicates it is not; movies can be in several genres at once. The movie ids are the ones used in the u.data data set.
u.genre -- A list of the genres.
u.user -- Demographic information about the users; this is a tab separated list of user id | age | gender | occupation | zip code The user ids are the ones used in the u.data data set.
u.occupation -- A list of the occupations.
u1.base -- The data sets u1.base and u1.test through u5.base and u5.test u1.test are 80%/20% splits of the u data into training and test data. u2.base Each of u1, ..., u5 have disjoint test sets; this if for u2.test 5 fold cross validation (where you repeat your experiment u3.base with each training and test set and average the results). u3.test These data sets can be generated from u.data by mku.sh. u4.base u4.test u5.base u5.test
ua.base -- The data sets ua.base, ua.test, ub.base, and ub.test ua.test split the u data into a training set and a test set with ub.base exactly 10 ratings per user in the test set. The sets ub.test ua.test and ub.test are disjoint. These data sets can be generated from u.data by mku.sh.
allbut.pl -- The script that generates training and test sets where all but n of a users ratings are in the training data.
mku.sh -- A shell script to generate all the u data sets from u.data.
In [8]:
ls ml-100k/
In [9]:
!head -n 2 ml-100k/u.data
In [13]:
# user id | item id | rating | timestamp
data_cols = ['user_id', 'item_id', 'rating', 'timestamp']
ratings = pd.read_csv('ml-100k/u.data', sep='\t', names=data_cols)
ratings.head()
Out[13]:
Select by row labels .loc[] or position with .iloc.
In [30]:
ratings.loc[0]
Out[30]:
Select columns with __getitem__.
In [28]:
ratings[['user_id', 'item_id']].head()
Out[28]:
In [31]:
ratings.loc[0:10, ['user_id', 'rating']]
Out[31]:
In [33]:
ratings['timestamp'] = pd.to_datetime(ratings.timestamp, unit='s')
ratings.dtypes
Out[33]:
In [34]:
ratings.describe()
Out[34]:
In [37]:
topids = pd.value_counts(ratings.item_id).head(10).index
topids
Out[37]:
In [40]:
top_movies = ratings[ratings.item_id.isin(topids)]
top_movies.head()
Out[40]:
In [44]:
top_movies.describe()
Out[44]:
In [46]:
pd.value_counts(top_movies.rating).sort_index().plot(kind='bar')
Out[46]:
Are more reviewed movies better liked?
Use groupby:
item_id)count, mean)
In [58]:
ratings.groupby('item_id')['rating'].agg(['count', 'mean']).head()
Out[58]:
In [73]:
ax = (ratings.groupby('item_id')['rating']
.agg(['count', 'mean'])
.plot(kind='scatter', x='count', y='mean'))
In [60]:
import statsmodels.api as sm
In [82]:
mat = ratings.groupby('item_id')['rating'].agg(['mean', 'count'])
mod = sm.OLS.from_formula('mean ~ count', mat)
res = mod.fit()
res.summary()
Out[82]:
In [87]:
def plot_reg(res):
ax = (ratings.groupby('item_id')['rating']
.agg(['count', 'mean'])
.plot(kind='scatter', x='count', y='mean'))
xx = np.linspace(0, 500)
y = res.params['Intercept'] + xx * res.params['count']
ax.plot(xx, y)
return ax
In [88]:
plot_reg(res)
Out[88]:
In [13]:
pd.value_counts(ratings.item_id).hist(bins=80)
Out[13]:
Bring in some other data
In [17]:
!head -n 3 ml-100k/u.genre
In [11]:
genres = (pd.read_csv('ml-100k/u.genre', sep='|', index_col=1)
.squeeze()
.to_dict())
genres
Out[11]:
In [12]:
!head -n 3 ml-100k/u.item
movie id | movie title | release date | video release date | IMDb URL | unknown | Action | Adventure | Animation | Children's | Comedy | Crime | Documentary | Drama | Fantasy | Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | Thriller | War | Western |
In [93]:
# Cp from readme
names = "movie id | movie title | release date | video release date | IMDb URL | unknown | Action | Adventure | Animation | Children's | Comedy | Crime | Documentary | Drama | Fantasy | Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi | Thriller | War | Western"
names = names.replace(' ', '_').split('_|_')
items = pd.read_csv('ml-100k/u.item', names=names, encoding='latin1', sep='|')
items['release_date'] = pd.to_datetime(items.release_date)
items.head()
Out[93]:
In [63]:
ratings.head()
Out[63]:
In [94]:
top_raters = pd.value_counts(ratings.user_id).head(10)
top_rater = top_raters.max()
In [86]:
pd.merge(ratings.query('user_id == @top_rater'), items,
left_on='item_id', right_on='movie_id')
Out[86]:
In [10]:
gen
Out[10]:
In [13]:
!head -n 2 ml-100k/u.user
In [22]:
user_cols = ['user_id', 'age', 'gender', 'occupation', 'zip']
users = pd.read_csv('ml-100k/u.user', sep='|', names=user_cols,
index_col='user_id')
users.head()
Out[22]:
In [23]:
sns.kdeplot(users.age)
Out[23]:
In [24]:
users.head()
Out[24]:
In [25]:
ratings.groupby('user_id')
Out[25]: