Score an edit!

In this notebook, we'll look at the the facilities that revscoring gives you for loading in a model and scoring some edits with it.

Step 1: Load the scorer model

In this step, we'll load a scorer model file from the disk.



In [1]:

    
from revscoring import Model
sm = Model.load(open("../models/enwiki.damaging.gradient_boosting.model"), env_check=False)
print(sm.info.format())









    



Model Information:
	 - type: GradientBoosting
	 - version: 0.4.0
	 - params: {'init': None, 'verbose': 0, 'loss': 'deviance', 'labels': [True, False], 'label_weights': OrderedDict([(True, 10)]), 'criterion': 'friedman_mse', 'learning_rate': 0.01, 'min_samples_split': 2, 'population_rates': None, 'random_state': None, 'subsample': 1.0, 'warm_start': False, 'min_samples_leaf': 1, 'min_impurity_decrease': 0.0, 'min_weight_fraction_leaf': 0.0, 'max_features': 'log2', 'center': True, 'max_leaf_nodes': None, 'scale': True, 'min_impurity_split': None, 'presort': 'auto', 'max_depth': 7, 'n_estimators': 700, 'multilabel': False}
	Environment:
	 - revscoring_version: '2.2.2'
	 - platform: 'Linux-4.9.0-6-amd64-x86_64-with-debian-9.4'
	 - machine: 'x86_64'
	 - version: '#1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02)'
	 - system: 'Linux'
	 - processor: ''
	 - python_build: ('default', 'Jan 19 2017 14:11:04')
	 - python_compiler: 'GCC 6.3.0 20170118'
	 - python_branch: ''
	 - python_implementation: 'CPython'
	 - python_revision: ''
	 - python_version: '3.5.3'
	 - release: '4.9.0-6-amd64'
	
	Statistics:
	counts (n=19455):
		label        n         ~True    ~False
		-------  -----  ---  -------  --------
		True       751  -->      422       329
		False    18704  -->      731     17973
	rates:
		              True    False
		----------  ------  -------
		sample       0.039    0.961
		population   0.034    0.966
	match_rate (micro=0.913, macro=0.5):
		  False    True
		-------  ------
		  0.943   0.057
	filter_rate (micro=0.087, macro=0.5):
		  False    True
		-------  ------
		  0.057   0.943
	recall (micro=0.947, macro=0.761):
		  False    True
		-------  ------
		  0.961   0.562
	!recall (micro=0.576, macro=0.761):
		  False    True
		-------  ------
		  0.562   0.961
	precision (micro=0.962, macro=0.661):
		  False    True
		-------  ------
		  0.984   0.337
	!precision (micro=0.359, macro=0.661):
		  False    True
		-------  ------
		  0.337   0.984
	f1 (micro=0.954, macro=0.697):
		  False    True
		-------  ------
		  0.972   0.421
	!f1 (micro=0.44, macro=0.697):
		  False    True
		-------  ------
		  0.421   0.972
	accuracy (micro=0.947, macro=0.947):
		  False    True
		-------  ------
		  0.947   0.947
	fpr (micro=0.424, macro=0.239):
		  False    True
		-------  ------
		  0.438   0.039
	roc_auc (micro=0.924, macro=0.924):
		  False    True
		-------  ------
		  0.924   0.924
	pr_auc (micro=0.978, macro=0.722):
		  False    True
		-------  ------
		  0.997   0.447
	
	 - score_schema: {'properties': {'prediction': {'type': 'bool', 'description': 'The most likely label predicted by the estimator'}, 'probability': {'properties': {'false': 'number', 'true': 'number'}, 'type': 'object', 'description': 'A mapping of probabilities onto each of the potential output labels'}}, 'type': 'object', 'title': 'Scikit learn-based classifier score with probability'}

Step 2: Prepare an extractor

We're using a model that was trained on English Wikipedia. We'll build an extractor to pull features from English Wikipedia.



In [2]:

    
import mwapi
from revscoring.extractors import api

extractor = api.Extractor(mwapi.Session("https://en.wikipedia.org", user_agent="Score edit demo in editquality"))

Step 3: Extract features and score an edit

Now, we'll use the extractor to extract features for a revision ID and then ask the scorer model to generate a prediction based on those features.



In [3]:

    
rev_to_score = 71076450
feature_values = list(extractor.extract(rev_to_score, sm.features))
list(zip(sm.features, feature_values))[:10]









    Out[3]:





[(<feature.revision.page.is_articleish>, True),
 (<feature.revision.page.is_mainspace>, True),
 (<feature.revision.page.is_draftspace>, False),
 (<feature.log((wikitext.revision.parent.chars + 1))>, 10.618934240590841),
 (<feature.log((len(<datasource.tokenized(datasource.revision.parent.text)>) + 1))>,
  9.45234493093131),
 (<feature.log((len(<datasource.wikitext.revision.parent.words>) + 1))>,
  8.372860820526318),
 (<feature.log((len(<datasource.wikitext.revision.parent.uppercase_words>) + 1))>,
  5.976350909297934),
 (<feature.log((wikitext.revision.parent.headings + 1))>, 3.091042453358316),
 (<feature.log((wikitext.revision.parent.wikilinks + 1))>, 5.846438775057725),
 (<feature.log((wikitext.revision.parent.external_links + 1))>,
  4.442651256490317)]



In [4]:

    
sm.score(feature_values)









    Out[4]:





{'prediction': True,
 'probability': {False: 0.23860839638483888, True: 0.7613916036151611}}



In [ ]: