This notebook demonstrates the basic contruction of a vandalism classification system using the revscoring library that we have developed specifically for classification models of MediaWiki stuff.
The basic process that we'll follow is this:
And then we'll have some fun applying the model to some edits using RCStream. The following diagram gives a good sense for the whole process of training and evaluating a model.
Regretfully, running SQL queries isn't something we can do directly from the notebook yet. So, we'll use Quarry to generate a nice random sample of edits. 20,000 observations should do just fine. Here's the query I want to run:
USE enwiki_p;
SELECT rev_id
FROM revision
WHERE rev_timestamp BETWEEN "20150201" AND "20160201"
ORDER BY RAND()
LIMIT 20000;
See http://quarry.wmflabs.org/query/7530. By clicking around the UI, I can see that this URL will download my tab-separated file: http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true
In [1]:
# Magical ipython notebook stuff puts the result of this command into a variable
revids_f = !wget http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true -qO-
revids = [int(line) for line in revids_f[1:]]
len(revids)
Out[1]:
OK. Now that we have a set of revisions, we need to label them. In this case, we're going to label them as reverted/not. We want to exclude a few different types of reverts -- e.g. when a user reverts themself or when an edit is reverted back to by someone else. For this, we'll use the mwreverts and mwapi libraries.
In [2]:
import sys, traceback
import mwreverts.api
import mwapi
# We'll use the mwreverts API check. In order to do that, we need an API session
session = mwapi.Session("https://en.wikipedia.org",
user_agent="Revert detection demo <ahalfaker@wikimedia.org>")
# For each revision, find out if it was "reverted" and label it so.
rev_reverteds = []
for rev_id in revids[:20]: # NOTE: Limiting to the first 20!!!!
try:
_, reverted, reverted_to = mwreverts.api.check(
session, rev_id, radius=5, # most reverts within 5 edits
window=48*60*60, # 2 days
rvprop={'user', 'ids'}) # Some properties we'll make use of
except (RuntimeError, KeyError) as e:
sys.stderr.write(str(e))
continue
if reverted is not None:
reverted_doc = [r for r in reverted.reverteds
if r['revid'] == rev_id][0]
if 'user' not in reverted_doc or 'user' not in reverted.reverting:
continue
# self-reverts
self_revert = \
reverted_doc['user'] == reverted.reverting['user']
# revisions that are reverted back to by others
reverted_back_to = \
reverted_to is not None and \
'user' in reverted_to.reverting and \
reverted_doc['user'] != \
reverted_to.reverting['user']
# If we are reverted, not by self or reverted back to by someone else,
# then, let's assume it was damaging.
damaging_reverted = not (self_revert or reverted_back_to)
else:
damaging_reverted = False
rev_reverteds.append((rev_id, damaging_reverted))
sys.stderr.write("r" if damaging_reverted else ".")
Eeek! This takes too long. You get the idea. So, I uploaded dataset that has already been labeled here @ ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2
In [3]:
rev_reverteds_f = !bzcat ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2
rev_reverteds = [line.strip().split("\t") for line in rev_reverteds_f[1:]]
rev_reverteds = [(int(rev_id), reverted == "True") for rev_id, reverted in rev_reverteds]
len(rev_reverteds)
Out[3]:
OK. It looks like we got an error when trying to extract the reverted status of ~132 edits, which is an acceptable loss. Now just to make sure we haven't gone crazy, let's check some of the reverted edits:
OK. Looks like we are doing pretty good. :)
In this section, we'll both split the training and testing set and gather prective features for each of the labeled observations.
In [4]:
train_set = rev_reverteds[:15000]
test_set = rev_reverteds[15000:]
print("training:", len(train_set))
print("testing:", len(test_set))
OK. In order to train the machine learning model, we'll need to give it a source of signal. This is where "features" come into play. A feature represents a simple numerical statistic that we can extract from our observations that we think will be predictive of our outcome. Luckily, revscoring provides a whole suite of features that work well for damage detection. In this case, we'll be looking at features of the edit diff.
In [5]:
from revscoring.features import wikitext, revision_oriented, temporal
from revscoring.languages import english
features = [
# Catches long key mashes like kkkkkkkkkkkk
wikitext.revision.diff.longest_repeated_char_added,
# Measures the size of the change in added words
wikitext.revision.diff.words_added,
# Measures the size of the change in removed words
wikitext.revision.diff.words_removed,
# Measures the proportional change in "badwords"
english.badwords.revision.diff.match_prop_delta_sum,
# Measures the proportional change in "informals"
english.informals.revision.diff.match_prop_delta_sum,
# Measures the proportional change meaningful words
english.stopwords.revision.diff.non_stopword_prop_delta_sum,
# Is the user anonymous
revision_oriented.revision.user.is_anon,
# Is the user a bot or a sysop
revision_oriented.revision.user.in_group({'bot', 'sysop'}),
# How long ago did the user register?
temporal.revision.user.seconds_since_registration
]
Now, we'll need to turn to revscorings feature extractor to help us get us feature values for each revision.
In [6]:
from revscoring.extractors import api
api_extractor = api.Extractor(session)
revisions = [695071713, 667375206]
for rev_id in revisions:
print("https://en.wikipedia.org/wiki/?diff={0}".format(rev_id))
print(list(api_extractor.extract(rev_id, features)))
In [10]:
# Now for the whole set!
training_features_reverted = []
for rev_id, reverted in train_set[:20]:
try:
feature_values = list(api_extractor.extract(rev_id, features))
observation = {"rev_id": rev_id, "cache": feature_values, "reverted": reverted}
except RuntimeError as e:
sys.stderr.write(str(e))
continue
sys.stderr.write(".")
training_features_reverted.append(observation)
In [54]:
# Uncomment to regenerate the observations file.
#import bz2
#from revscoring.utilities.util import dump_observation
#
#f = bz2.open("../datasets/demo/enwiki.features_reverted.training.20k_2015.json.bz2", "wt")
#for observation in training_features_reverted:
# dump_observation(observation, f)
#f.close()
Eeek! Again this takes too long, so again, I uploaded a dataset with features already extracted @ ../datasets/demo/enwiki.features_reverted.training.20k_2015.tsv.bz2
In [56]:
from revscoring.utilities.util import read_observations
training_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.training.20k_2015.json.bz2
training_features_reverted = list(read_observations(training_features_reverted_f))
len(training_features_reverted)
Out[56]:
Now that we have a set of features extracted for our training set, it's time to train a model. revscoring provides a set of different classifier algorithms. From past experience, I know a gradient boosting classifier works well, so we'll use that.
In [58]:
from revscoring.scoring.models import GradientBoosting
is_reverted = GradientBoosting(features, labels=[True, False], version="live demo!",
learning_rate=0.01, max_features="log2",
n_estimators=700, max_depth=5,
population_rates={False: 0.5, True: 0.5}, scale=True, center=True)
training_unpacked = [(o["cache"], o["reverted"]) for o in training_features_reverted]
is_reverted.train(training_unpacked)
Out[58]:
We now have a trained model that we can play around with. Let's try a few edits from our test set.
In [13]:
reverted_obs = [rev_id for rev_id, reverted in test_set if reverted]
non_reverted_obs = [rev_id for rev_id, reverted in test_set if not reverted]
for rev_id in reverted_obs[:10]:
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
print(True, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
score['prediction'], round(score['probability'][True], 2))
for rev_id in non_reverted_obs[:10]:
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
print(False, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
score['prediction'], round(score['probability'][True], 2))
So, the above analysis can help give us a sense for whether the model is working or not, but it's hard to standardize between models. So, we can apply some metrics that are specially crafted for machine learning models.
But first, I'll need to load the pre-generated feature values.
In [14]:
testing_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.testing.20k_2015.json.bz2
testing_features_reverted = list(read_observations(testing_features_reverted_f))
testing_unpacked = [(o["cache"], o["reverted"]) for o in testing_features_reverted]
len(testing_unpacked)
Out[14]:
We'll use revscoring statistics to measure these against the test set.
In [65]:
is_reverted.test(testing_unpacked)
print(is_reverted.info.format())
So we don't have the most powerful damage detection classifier, but then again, we're only including 9 features. Usually we run with ~60 features and get to much higher levels of fitness. but this model is still useful and it should help us detect the most egregious vandalism in Wikipedia. In order to listen to Wikipedia, we'll need to connect to RCStream -- the same live feed that powers listen to Wikipedia.
In [18]:
import json
from sseclient import SSEClient as EventSource
url = 'https://stream.wikimedia.org/v2/stream/recentchange'
for event in EventSource(url):
if event.event == 'message':
try:
change = json.loads(event.data)
if change['type'] not in ('new', 'edit'):
continue
rev_id = change['revision']['new']
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
if score['prediction']:
print("!!!Please review", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
round(score['probability'][True], 2), flush=True)
else:
print("Good edit", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
round(score['probability'][True], 2), flush=True)
except ValueError:
pass
In [ ]: