Michaël Defferrard, PhD student, EPFL LTS2
Theme of the exercise: understand the impact of your communication on social networks. A real life situation: the marketing team needs help in identifying which were the most engaging posts they made on social platforms to prepare their next AdWords campaign.
This notebook is the second part of the exercise. Given the data we collected from Facebook an Twitter in the last exercise, we will construct an ML model and evaluate how good it is to predict the number of likes of a post / tweet given the content.
pandas to import the facebook.sqlite and twitter.sqlite databases.The facebook.sqlite and twitter.sqlite SQLite databases can be created by running the data acquisition and exploration exercise.
In [1]:
import pandas as pd
import numpy as np
from IPython.display import display
import os.path
folder = os.path.join('..', 'data', 'social_media')
fb = pd.read_sql('facebook', 'sqlite:///' + os.path.join(folder, 'facebook.sqlite'), index_col='index')
tw = pd.read_sql('twitter', 'sqlite:///' + os.path.join(folder, 'twitter.sqlite'), index_col='index')
display(fb[:5])
display(tw[:5])
First step: transform the data into a format understandable by the machine. What to do with text ? A common choice is the so-called bag-of-word model, where we represent each word a an integer and simply count the number of appearances of a word into a document.
Example
Let's say we have a vocabulary represented by the following correspondance table.
| Integer | Word |
|---|---|
| 0 | unknown |
| 1 | dog |
| 2 | school |
| 3 | cat |
| 4 | house |
| 5 | work |
| 6 | animal |
Then we can represent the following document
I have a cat. Cats are my preferred animals.
by the vector $x = [6, 0, 0, 2, 0, 0, 1]^T$.
Tasks
Tip: the natural language modeling libraries nltk and gensim are useful for advanced operations. You don't need them here.
Arise a first data cleaning question. We may have some text in french and other in english. What do we do ?
In [2]:
from sklearn.feature_extraction.text import CountVectorizer
nwords = 200 # 100
def compute_bag_of_words(text, nwords):
vectorizer = CountVectorizer(max_features=nwords)
vectors = vectorizer.fit_transform(text)
vocabulary = vectorizer.get_feature_names()
return vectors, vocabulary
fb_bow, fb_vocab = compute_bag_of_words(fb.text, nwords)
#fb_p = pd.Panel({'orig': fb, 'bow': fb_bow})
display(fb_bow)
display(fb_vocab[100:110])
tw_bow, tw_vocab = compute_bag_of_words(tw.text, nwords)
display(tw_bow)
Exploration question: what are the 5 most used words ? Exploring your data while playing with it is a useful sanity check.
In [3]:
def print_most_frequent(bow, vocab, n=10):
idx = np.argsort(bow.sum(axis=0))
for i in range(10):
j = idx[0, -i]
print(vocab[j])
print_most_frequent(tw_bow, tw_vocab)
print('---')
print_most_frequent(fb_bow, fb_vocab)
In [4]:
X = tw_bow
y = tw['likes'].values
n, d = X.shape
assert n == y.size
print(X.shape)
print(y.shape)
In [5]:
# Training and testing sets.
test_size = n // 2
print('Split: {} testing and {} training samples'.format(test_size, y.size - test_size))
perm = np.random.permutation(y.size)
X_test = X[perm[:test_size]]
X_train = X[perm[test_size:]]
y_test = y[perm[:test_size]]
y_train = y[perm[test_size:]]
Using numpy, fit and evaluate the linear model $$\hat{w}, \hat{b} = \operatorname*{arg min}_{w,b} \| Xw + b - y \|_2^2.$$
Please define a class LinearRegression with two methods:
fit learn the parameters $w$ and $b$ of the model given the training examples.predict gives the estimated number of likes of a post / tweet. That will be used to evaluate the model on the testing set.To evaluate the classifier, create an accuracy(y_pred, y_true) function which computes the mean squared error $\frac1n \| \hat{y} - y \|_2^2$.
Hint: you may want to use the function scipy.sparse.linalg.spsolve().
If solve and spsolve tells you that your matrix is singular, please read this good comment. Potential solutions:
In [13]:
import scipy.sparse
class LinearRegression(object):
def predict(self, X):
"""Return the predicted class given the features."""
return X.dot(self.w) + self.b
def fit(self, X, y):
"""Learn the model's parameters given the training data, the closed-form way."""
n, d = X.shape
self.b = y.mean()
A = X.T.dot(X)
b = X.T.dot(y - self.b)
#self.w = np.linalg.solve(A, b)
self.w = scipy.sparse.linalg.spsolve(A, b)
def evaluate(y_pred, y_true):
return np.linalg.norm(y_pred - y_true, ord=2)**2 / y_true.size
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
mse = evaluate(y_pred, y_test)
print('mse: {:.4f}'.format(mse))
Interpretation: what are the most important words a post / tweet should include ?
In [7]:
idx = np.argsort(abs(model.w))
for i in range(20):
j = idx[-1-i]
print('weight: {:5.2f}, word: {}'.format(model.w[j], tw_vocab[j]))
In [8]:
import ipywidgets
from IPython.display import clear_output
slider = ipywidgets.widgets.IntSlider(
value=1,
min=1,
max=nwords,
step=1,
description='nwords',
)
def handle(change):
"""Handler for value change: fit model and print performance."""
nwords = change['new']
clear_output()
print('nwords = {}'.format(nwords))
model = LinearRegression()
model.fit(X_train[:, :nwords], y_train)
y_pred = model.predict(X_test[:, :nwords])
mse = evaluate(y_pred, y_test)
print('mse: {:.4f}'.format(mse))
slider.observe(handle, names='value')
display(slider)
slider.value = nwords # As if someone moved the slider.
In [9]:
from sklearn import linear_model, metrics
model = linear_model.LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
mse = metrics.mean_squared_error(y_test, y_pred)
assert np.allclose(evaluate(y_pred, y_test), mse)
print('mse: {:.4f}'.format(mse))
In [10]:
import os
os.environ['KERAS_BACKEND'] = 'theano' # tensorflow
import keras
model = keras.models.Sequential()
model.add(keras.layers.Dense(output_dim=50, input_dim=nwords, activation='relu'))
model.add(keras.layers.Dense(output_dim=20, activation='relu'))
model.add(keras.layers.Dense(output_dim=1, activation='relu'))
model.compile(loss='mse', optimizer='sgd')
model.fit(X_train.toarray(), y_train, nb_epoch=20, batch_size=100)
y_pred = model.predict(X_test.toarray(), batch_size=32)
mse = evaluate(y_test, y_pred.squeeze())
print('mse: {:.4f}'.format(mse))
Use matplotlib to plot a performance visualization. E.g. the true number of likes and the real number of likes for all posts / tweets.
What do you observe ? What are your suggestions to improve the performance ?
In [11]:
from matplotlib import pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
n = 100
plt.figure(figsize=(15, 5))
plt.plot(y_test[:n], '.', alpha=.7, markersize=10, label='ground truth')
plt.plot(y_pred[:n], '.', alpha=.7, markersize=10, label='prediction')
plt.legend()
plt.show()