Have you ever been watching a music video and stop and ask yourself: "Man, I wish I knew which genre this song is." Just weeks ago, this was us. Four college students with no sense of harmonic direction, just trying to find some musical fusion in our lives. We set out to find solution to this problem, and build a tool that analyzes music videos from YouTube and classifies them by genre.
We wanted to find out if it was possible to intelligently determine the genre of a video by its youtube metadata (likes, dislikes, view count, etc.) and its average frame colors throughout the video.
After thorough analysis, Pixal emerged: a web interface that takes a YouTube URL as input, and outputs a predicted genre. This is done using a combination of OpenCV color analysis and scikit-learn machine learning, and makes intelligent assumptions on our data to output a substantially accurate genre classification for the input video.
Our motivation primarily comes from GitHub user Sacert's 'Colors of Film' analysis, in which he analyzed popular Hollywood films and extracted avereage frame colors. He used OpenCV and some built-in python libraries to generate his output, which is an image with colors from each frame of the movie. Below is the output image for Harry Potter and the Prizoner of Azkaban:
When we began our analysis, the question we asked was, "Can we accurately predict music genre by analyzing various aspects of a music video". This question was obviously extremely broad, and we quickly realized that we needed to narrow the scope and focus in on a few important/relevant features.
Thus, we revised our question to, "Can we accurately predict music genre by analyzing average frame color and youtube's video metadata (likes, dislikes, viewcount, etc.)". This question was much more manageable, but was more geared towards a final product than the actual data science process.
We then divided our project into two separate questions, with connected, but different goals. Firstly, "What are the most defining/relevant features for a music video when it comes to genre", and, "Is it possible to leverage these features to accurately predict the genre of a music video". These are the two questions that our project really sought to answer.
We employed several techniques to gather and aggregate the data we needed.
Date Acquisition
Firstly, we chose five of the most common genres to focus on and found a YouTube playlist for each of them. We used a chrome extension called scraper to download the links to the top 200 videos in each playlist. Then, we used Pafy, a python library, to download each of the videos. Pafy also allowed us to access the metadata (view count, rating, etc.) that we needed.
Compressing Videos
In order to speed up our frame-by-frame analysis of the videos, we wrote a script using 'ffmpeg', a video conversion tool, to lower the quality of videos, and therefore decrease the number of frames to analyze.
Analyzing Frame Colors
With skvideo, we iterated over the frames of each video. Originally, we took the average color of all the pixels in the frames. However, we quickly found that almost every frame's color averaged out to gray. Because of this, we decided to look at the most commonly appearing (mode) color of each frame. We then took the 10 most common frame colors throughout the video. These became the RGB values that we used in our analysis.
In [32]:
import pandas as pd
from os import path
from sklearn.ensemble import RandomForestClassifier
import numpy as np
from sklearn.ensemble import ExtraTreesClassifier
import sklearn
# Edit path if need be (shouldn't need to b/c we all have the same folder structure)
CSV_PATH_1 = '../Videos/all_data'
CSV_PATH_2 = '../Videos2/all_data2'
FILE_EXTENSION = '_all.csv'
GENRES = ['country', 'edm', 'pop', 'rap', 'rock']
# Containers for the data frames
genre_dfs = {}
all_genres = None
# Read in the 5 genre's of CV's
for genre in GENRES:
genre_csv_path_1 = path.join(CSV_PATH_1, genre) + FILE_EXTENSION
genre_csv_path_2 = path.join(CSV_PATH_2, genre) + FILE_EXTENSION
df_1 = pd.read_csv(genre_csv_path_1)
df_2 = pd.read_csv(genre_csv_path_2)
df_1 = df_1.drop('Unnamed: 0',1)
df_2 = df_2.drop('Unnamed: 0',1)
df_combined = pd.concat([df_1,df_2],ignore_index=True)
genre_dfs[genre] = df_combined
all_genres = pd.concat(genre_dfs.values())
all_genres.head(3)
# genre_dfs is now a dictionary that contains the 5 different data frames
# all_genres is a dataframe that contains all of the data
Out[32]:
In accordance with scikit-learn, we had to make the genres ordinal to fit in the random forest classifiers. We add a new column to our dataframe to do so, write a function to populate it, and run it across the dataframe. We also create binary genre columns for each genre for additional analysis.
We create our training and test sets by splitting all_genres by genre, and making ~100 of each genre train and ~100 test. We aggregate by genre to make our full train and full test sets, each containing ~500 records of various genres.
In [33]:
def genre_to_ordinal(genre_in):
if(genre_in == "country"):
return 0
elif(genre_in == "pop"):
return 1
elif(genre_in == "rock"):
return 2
elif(genre_in == "edm"):
return 3
elif(genre_in == "rap"):
return 4
else:
return genre_in
all_genres['genre_ordinal'] = all_genres.genre.apply(genre_to_ordinal)
# Adding is_country flag
def is_country(genre_in):
if(genre_in == "country"):
return 1
else:
return 0
all_genres['is_country'] = all_genres.genre.apply(is_country)
# Adding is_country flag
def is_rock(genre_in):
if(genre_in == "rock"):
return 1
else:
return 0
all_genres['is_rock'] = all_genres.genre.apply(is_rock)
# Adding is_edm flag
def is_edm(genre_in):
if(genre_in == "edm"):
return 1
else:
return 0
all_genres['is_edm'] = all_genres.genre.apply(is_edm)
# Adding is_rap flag
def is_rap(genre_in):
if(genre_in == "rap"):
return 1
else:
return 0
all_genres['is_rap'] = all_genres.genre.apply(is_rap)
# Adding is_country flag
def is_pop(genre_in):
if(genre_in == "pop"):
return 1
else:
return 0
all_genres['is_pop'] = all_genres.genre.apply(is_pop)
# Subset all_genres to group by individual genres
country_records = all_genres[all_genres["genre"] == "country"]
rock_records = all_genres[all_genres["genre"] == "rock"]
pop_records = all_genres[all_genres["genre"] == "pop"]
edm_records = all_genres[all_genres["genre"] == "edm"]
rap_records = all_genres[all_genres["genre"] == "rap"]
# From the subsets above, create train and test sets from each
country_train = country_records.head(len(country_records) / 2)
country_test = country_records.tail(len(country_records) / 2)
rock_train = rock_records.head(len(rock_records) / 2)
rock_test = rock_records.tail(len(rock_records) / 2)
pop_train = pop_records.head(len(pop_records) / 2)
pop_test = pop_records.tail(len(pop_records) / 2)
edm_train = edm_records.head(len(edm_records) / 2)
edm_test = edm_records.tail(len(edm_records) / 2)
rap_train = rap_records.head(len(rap_records) / 2)
rap_test = rap_records.tail(len(rap_records) / 2)
# Create big training and big test set for analysis
training_set = pd.concat([country_train,rock_train,pop_train,edm_train,rap_train])
test_set = pd.concat([country_test,rock_test,pop_test,edm_test,rap_test])
training_set = training_set.fillna(0)
test_set = test_set.fillna(0)
print "Training Records:\t" , len(training_set)
print "Test Records:\t\t" , len(test_set)
training_set.head()
Out[33]:
The above training_set and test_set were the two dataframes that we use for analysis over the course of the project.
We start generating our random forests, and output a relative accuracy and a confusion matrix. In this first one, we simply factor in non-color variables (rating, likes, dislikes, length and viewcount), and run it across all records to predict an ordinal genre value.
As you will see, this method yields relatively poor results. This is because there's no distinct clusters being created by our random forest, and simple viewer statistics tell us nothing about what kind of video we're watching. However, we see that country, rap and pop are initially somewhat distinct (diagonal is the highest value), and rock and edm are getting mistaken for one another. Let's see if we can't make something of this.
In [34]:
# Predicting based solely on non-color features, using RF
clf = RandomForestClassifier(n_estimators=11)
meta_data_features = ['rating', 'likes','dislikes','length','viewcount']
y, _ = pd.factorize(training_set['genre_ordinal'])
clf = clf.fit(training_set[meta_data_features], y)
z, _ = pd.factorize(test_set['genre_ordinal'])
print clf.score(test_set[meta_data_features],z)
pd.crosstab(test_set.genre_ordinal, clf.predict(test_set[meta_data_features]),rownames=["Actual"], colnames=["Predicted"])
Out[34]:
Below, we do the same random forest as above, but strictly off of average frame color for the video.
As shown below, this actually yields worse results than just the viewer statistics, because the color of a video by itself does not determine the genre. If rappers only had red in their videos and rockers only had black this might be somewhat accurate, but that's just not the case. But, what if we pair these findings with our initial viewer statistics?
In [35]:
def gen_new_headers(old_headers):
headers = ['colors_' + str(x+1) + '_' for x in range(10)]
h = []
for x in headers:
h.append(x + 'red')
h.append(x + 'blue')
h.append(x + 'green')
return old_headers + h + ['genre']
clf = RandomForestClassifier(n_estimators=11)
color_features = gen_new_headers([])[:-1]
# Predicting based solely on colors
y, _ = pd.factorize(training_set['genre_ordinal'])
clf = clf.fit(training_set[color_features], y)
z, _ = pd.factorize(test_set['genre_ordinal'])
print clf.score(test_set[color_features],z)
pd.crosstab(test_set.genre_ordinal, clf.predict(test_set[color_features]),rownames=["Actual"], colnames=["Predicted"])
Out[35]:
In [36]:
clf = RandomForestClassifier(n_estimators=11)
all_features = meta_data_features + color_features
# Predicting based on colors and non-color features
y, _ = pd.factorize(training_set['genre_ordinal'])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set['genre_ordinal'])
print clf.score(test_set[all_features],z)
pd.crosstab(test_set.genre_ordinal, clf.predict(test_set[all_features]),rownames=["Actual"], colnames=["Predicted"])
Out[36]:
In [37]:
clf = RandomForestClassifier(n_estimators=11)
all_features = meta_data_features + color_features
# Predicting based on colors and non-color features
y, _ = pd.factorize(training_set['is_pop'])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set['is_pop'])
print clf.score(test_set[all_features],z)
pd.crosstab(test_set.is_pop, clf.predict(test_set[all_features]),rownames=["Actual"], colnames=["Predicted"])
Out[37]:
In [38]:
clf = RandomForestClassifier(n_estimators=11)
all_features = meta_data_features + color_features
# Predicting based on colors and non-color features
y, _ = pd.factorize(training_set['is_rap'])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set['is_rap'])
print clf.score(test_set[all_features],z)
pd.crosstab(test_set.is_rap, clf.predict(test_set[all_features]),rownames=["Actual"], colnames=["Predicted"])
Out[38]:
What we're seeing above is a confusion matrix that, based on our training data, predicts whether or not a video in the test set is a pop video or not. You can see that it predicted 299 + 25 = 324 videos correctly, versus 60 + 21 = 81 videos incorrectly.
The confusion matrix above is our first effort at utilizing these binary classifiers. The classifier does a relatively good job with a success rate of 80%. However, we could use some improvement in the realm of "false negatives", where the model classified a video as not pop when it actually was.
We attempt this same test for each genre, and run the test multiple times to see what our average success rate is for each genre.
In [39]:
training_set = pd.concat([country_train,rock_train,pop_train,edm_train,rap_train])
test_set = pd.concat([country_test,rock_test,pop_test,edm_test,rap_test])
def multi_RF_averages(is_genre,num_iterations):
clf = RandomForestClassifier(n_estimators=11)
loop_indices = range(0,num_iterations)
cumsum = 0
for i in loop_indices:
y, _ = pd.factorize(training_set[is_genre])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set[is_genre])
cumsum = cumsum + clf.score(test_set[all_features],z)
print "Average Score for",len(loop_indices),is_genre,"iterations:", cumsum/len(loop_indices)
NUM_TIMES = 20
pop_class = multi_RF_averages("is_pop", NUM_TIMES)
rap_class = multi_RF_averages("is_rap", NUM_TIMES)
rock_class = multi_RF_averages("is_rock", NUM_TIMES)
edm_class = multi_RF_averages("is_edm", NUM_TIMES)
country_class = multi_RF_averages("is_country", NUM_TIMES)
Now, let's look at certain genres.
In [40]:
# Removing EDM for better analysis - makes is_pop and is_rap much more accurate
# training_set = pd.concat([rock_train,edm_train,country_train,pop_train])
# test_set = pd.concat([rock_test,edm_test,country_test,pop_test])
training_set = pd.concat([country_train,pop_train])
test_set = pd.concat([country_test,pop_test])
multi_RF_averages("is_country",50)
multi_RF_averages("is_pop",50)
print '--------------------------------'
training_set = pd.concat([edm_train,rock_train])
test_set = pd.concat([edm_test,rock_test])
multi_RF_averages("is_rock",50)
multi_RF_averages("is_edm",50)
So, what does this tell us? It seems that country and pop are easy to distinguish between while rock and edm are significantly harder to differentiate.
Next let's look at which genres seem to be the best at identifying themselves.
In [41]:
training_set = pd.concat([country_train,rock_train,edm_train,rap_train,pop_train])
test_set = pd.concat([rock_test])
multi_RF_averages("is_rock",50)
test_set = pd.concat([rap_test])
multi_RF_averages("is_rap",50)
test_set = pd.concat([country_test])
multi_RF_averages("is_country",50)
test_set = pd.concat([pop_test])
multi_RF_averages("is_pop",50)
test_set = pd.concat([edm_test])
multi_RF_averages("is_edm",50)
Something seems to be going on with Country. Let's also take a look at EDM which seems to be very successful at differentiating itself from other genres.
In [42]:
training_set = pd.concat([country_train,rock_train,pop_train,edm_train,rap_train])
test_set = pd.concat([country_test,rock_test,pop_test,edm_test,rap_test])
clf = RandomForestClassifier(n_estimators=11)
all_features = meta_data_features + color_features
y, _ = pd.factorize(training_set['is_edm'])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set['is_edm'])
pd.crosstab(test_set.is_edm, clf.predict(test_set[all_features]),rownames=["Actual"], colnames=["Predicted"])
Out[42]:
In [43]:
training_set = pd.concat([country_train,rock_train,pop_train,edm_train,rap_train])
test_set = pd.concat([country_test,rock_test,pop_test,edm_test,rap_test])
clf = RandomForestClassifier(n_estimators=11)
all_features = meta_data_features + color_features
y, _ = pd.factorize(training_set['is_country'])
clf = clf.fit(training_set[all_features], y)
z, _ = pd.factorize(test_set['is_country'])
pd.crosstab(test_set.is_country, clf.predict(test_set[all_features]),rownames=["Actual"], colnames=["Predicted"])
Out[43]:
Here we see some interesting insights. It seems that when it comes to classifying for country, everything looks like country. You'll notice that the country classifier incorrectly labels a large number (~300) non-country videos as country. This is opposed to the EDM classifier which does not suffer from these issues. This makes sense, as our production system that we developed after all of this analysis seems to categorize videos as country extremely often.
Where EDM suffers, is that it has a hard time correctly identifying itself as EDM.
Essentially, this means that neither EDM or Country are very distinguishable. No videos seem to look like EDM videos (including EDM videos themselves), while every video seems to look like a country video.
Now let's look at which feature seems to be the most helpful in classifying videos.
In [44]:
model = ExtraTreesClassifier()
training_set = pd.concat([country_train,pop_train,rap_train,rock_train,edm_train])
y, _ = pd.factorize(training_set['is_rock'])
model.fit(training_set[all_features], y)
df = pd.DataFrame()
df['index'] = all_features
y, _ = pd.factorize(training_set['is_rap'])
model.fit(training_set[all_features], y)
df['rap'] = model.feature_importances_
y, _ = pd.factorize(training_set['is_rock'])
model.fit(training_set[all_features], y)
df['rock'] = model.feature_importances_
y, _ = pd.factorize(training_set['is_country'])
model.fit(training_set[all_features], y)
df['country'] = model.feature_importances_
y, _ = pd.factorize(training_set['is_edm'])
model.fit(training_set[all_features], y)
df['edm'] = model.feature_importances_
y, _ = pd.factorize(training_set['is_pop'])
model.fit(training_set[all_features], y)
df['pop'] = model.feature_importances_
df.head()
Out[44]:
In [45]:
import plotly.offline as py
import plotly.graph_objs as go
df = df.set_index('index')
df = df.transpose()
df.head()
lol = df.values.tolist()
cols = []
for x in df.columns:
cols.append(x)
py.init_notebook_mode()
title = 'Feature Importance By Genre'
labels = ['rap','rock','country','edm','pop']
x_data = cols
y_data = df.values.tolist()
traces = []
for i in range(0, 5):
traces.append(go.Scatter(
x=x_data,
y=y_data[i],
mode='lines',
connectgaps=True,
name = labels[i]
))
layout = go.Layout(
yaxis=dict(
showgrid=False,
zeroline=False,
showline=False,
showticklabels=False,
),
autosize=False,
margin=dict(
autoexpand=True,
l=100,
r=20,
t=110,
),
showlegend=False,
)
layout = dict(title = 'Feature Importance by Genre',
xaxis = dict(title = 'Feature'),
yaxis = dict(title = 'Percent Importance (All Features Sum to 1.0)',
showgrid=False),
margin=go.Margin(
l=80,
r=50,
b=170,
t=100,
pad=8
),
)
fig = go.Figure(data=traces, layout=layout)
py.iplot(fig, filename='news-source')
Here we notice a few things. Perhaps unsurprisingly, pop classification seems to place heavy emphasis on the likes and viewcount feautres. This seems logical, as often pop videos have a much larger number of likes and views than other videos.
The other take away here is that, aside from a small increase in importance of viewcount over all genres, none of the features, be they metatdata or color data, seem to be particularly more important than others. This could mean that frame color data and view/like data is not the right information for classifying genre, however, some more analysis would need to be done to confirm that.
Unforunately, it appears that we have failed in our mission to classify videos by metadata and frame color data. There are three main issues that we discovered when analyzing that dta.
Overall, it seems that using frame color data and basic metadata is not the best for identifying music video genre.