notebook.community

Edit and run

prelim_month_human - confusion matrix

old file name: 2017.10.21 - work log - prelim_month_human - confusion matrix

Confusion matrix for data where coder 1 is ground truth, coder 2 is uncorrected human coding.

Table of Contents

Setup





In [2]:



    


# set the label we'll be looking at throughout
current_label = "prelim_month_human"

Setup - Imports





In [3]:



    


import datetime
import math
import pandas
import pandas_ml
import sklearn
import sklearn.metrics
import six
import statsmodels
import statsmodels.api

print( "packages imported at " + str( datetime.datetime.now() ) )



    


















    






packages imported at 2017-10-26 00:17:11.975377










    






/home/jonathanmorgan/.virtualenvs/sourcenet/lib/python3.5/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools

















In [4]:



    


%pwd



    


















    
Out[4]:








'/home/jonathanmorgan/work/django/research/work/phd_work'

Setup - Initialize Django

First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.

You need to have installed your virtualenv with django as a kernel, then select that kernel for this notebook.





In [5]:



    


%run ../django_init.py



    


















    






django initialized at 2017-10-26 04:17:19.206314

Import any sourcenet or context_analysis models or classes.





In [6]:



    


# python_utilities
from python_utilities.analysis.statistics.confusion_matrix_helper import ConfusionMatrixHelper
from python_utilities.analysis.statistics.stats_helper import StatsHelper
from python_utilities.dictionaries.dict_helper import DictHelper

# context_analysis models.
from context_analysis.models import Reliability_Names

print( "sourcenet and context_analysis packages imported at " + str( datetime.datetime.now() ) )



    


















    






sourcenet and context_analysis packages imported at 2017-10-26 04:17:22.776195

Setup - Tools

Write functions here to do math, so that we can reuse said tools below.

Build Confusion Matrix Data

A basic confusion matrix ( https://en.wikipedia.org/wiki/Confusion_matrix ) contains counts of true positives, true negatives, false positives, and false negatives for a given binary or boolean (yes/no) classification decision you are asking someone or something to make.

To create a confusion matrix, you need two associated vectors containing classification decisions (0s and 1s), one that contains ground truth, and one that contains values predicted by whatever coder you are testing. For each associated pair of values:

  • Start with the predicted value: positive (1) or negative (0).
  • Look at the corresponding ground truth value. If they match, it is "true". If not, it is "false".
  • So, predicted 1 and ground_truth 1 is a "true positive".
  • Add one to the counter for the class of prediction: "true positive", "true negative", "false positive", "false negative".

Once you have your basic confusion matrix, the counts of true positives, true negatives, false positives, and false negatives can then be used to calculate a set of different scores and values one can use to assess the quality of predictive models. These scores include "precision and recall", "accuracy", an "F1 score" (a harmonic mean), and a "diagnostic odds ratio", among many others.

Person detection

For each person detected across the set of articles, look at whether the automated coder correctly detected the person, independent of eventual lookup or person type.

Person detection - build value lists

First, build lists of ground truth and predicted values per person.





In [7]:



    


# declare variables
reliability_names_label = None
label_in_list = []
reliability_names_qs = None
ground_truth_coder_index = 1
predicted_coder_index = 2

# processing
column_name = ""
predicted_value = -1
predicted_list = []
ground_truth_value = -1
ground_truth_list = []
reliability_names_instance = None

# set label
reliability_names_label = current_label  # "prelim_month_human"

# lookup Reliability_Names for selected label
label_in_list.append( reliability_names_label )
reliability_names_qs = Reliability_Names.objects.filter( label__in = label_in_list )

print( "Found " + str( reliability_names_qs.count() ) + " rows with label in " + str( label_in_list ) )

# loop over records
predicted_value = -1
predicted_list = []
ground_truth_value = -1
ground_truth_list = []
ground_truth_positive_count = 0
predicted_positive_count = 0
true_positive_count = 0
false_positive_count = 0
ground_truth_negative_count = 0
predicted_negative_count = 0
true_negative_count = 0
false_negative_count = 0
for reliability_names_instance in reliability_names_qs:
    
    # get detected flag from ground truth and predicted columns and add them to list.
    
    # ==> ground truth
    column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
    column_name += str( ground_truth_coder_index )
    column_name += "_" + Reliability_Names.FIELD_NAME_SUFFIX_DETECTED
    ground_truth_value = getattr( reliability_names_instance, column_name )
    ground_truth_list.append( ground_truth_value )
    
    # ==> predicted
    column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
    column_name += str( predicted_coder_index )
    column_name += "_" + Reliability_Names.FIELD_NAME_SUFFIX_DETECTED
    predicted_value = getattr( reliability_names_instance, column_name )
    predicted_list.append( predicted_value )
    
#-- END loop over Reliability_Names instances. --#

print( "==> population values count: " + str( len( ground_truth_list ) ) )
print( "==> predicted values count: " + str( len( predicted_list ) ) )
print( "==> percentage agreement = " + str( StatsHelper.percentage_agreement( ground_truth_list, predicted_list ) ) )



    


















    






Found 2421 rows with label in ['prelim_month_human']
==> population values count: 2421
==> predicted values count: 2421
==> percentage agreement = 0.953738124742

















In [8]:



    


print( "==> population values: " + str( len( ground_truth_list ) ) )
list_name = "ACTUAL_VALUE_LIST"
string_list = map( str, ground_truth_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )

print( "==> predicted values count: " + str( len( predicted_list ) ) )
list_name = "PREDICTED_VALUE_LIST"
string_list = map( str, predicted_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )



    


















    






==> population values: 2421
ACTUAL_VALUE_LIST = [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ]
==> predicted values count: 2421
PREDICTED_VALUE_LIST = [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ]

Person detection - confusion matrix





In [9]:



    


confusion_matrix = pandas_ml.ConfusionMatrix( ground_truth_list, predicted_list )
print("Confusion matrix:\n%s" % confusion_matrix)
confusion_matrix.print_stats()
stats_dict = confusion_matrix.stats()
print( str( stats_dict ) )
print( str( stats_dict[ 'TPR' ] ) )

# get counts in variables
true_positive_count = confusion_matrix.TP
false_positive_count = confusion_matrix.FP
true_negative_count = confusion_matrix.TN
false_negative_count = confusion_matrix.FN

# and derive population and predicted counts
ground_truth_positive_count = true_positive_count + false_negative_count
predicted_positive_count = true_positive_count + false_positive_count
ground_truth_negative_count = true_negative_count + false_positive_count
predicted_negative_count = true_negative_count + false_negative_count


print( "==> Predicted positives: " + str( predicted_positive_count ) + " ( " + str( ( true_positive_count + false_positive_count ) ) + " )" )
print( "==> Ground truth positives: " + str( ground_truth_positive_count )  + " ( " + str( ( true_positive_count + false_negative_count ) ) + " )" )
print( "==> True positives: " + str( true_positive_count ) )
print( "==> False positives: " + str( false_positive_count ) )
print( "==> Predicted negatives: " + str( predicted_negative_count ) + " ( " + str( ( true_negative_count + false_negative_count ) ) + " )" )
print( "==> Ground truth negatives: " + str( ground_truth_negative_count ) + " ( " + str( ( true_negative_count + false_positive_count ) ) + " )" )
print( "==> True negatives: " + str( true_negative_count ) )
print( "==> False negatives: " + str( false_negative_count ) )
print( "==> Precision (true positive/predicted positive): " + str( ( true_positive_count / predicted_positive_count ) ) )
print( "==> Recall (true positive/ground truth positive): " + str( ( true_positive_count / ground_truth_positive_count ) ) )



    


















    






Confusion matrix:
Predicted   0     1  __all__
Actual                      
0           0    19       19
1          93  2309     2402
__all__    93  2328     2421
population: 2421
P: 2402
N: 19
PositiveTest: 2328
NegativeTest: 93
TP: 2309
TN: 0
FP: 19
FN: 93
TPR: 0.961282264779
TNR: 0.0
PPV: 0.991838487973
NPV: 0.0
FPR: 1.0
FDR: 0.00816151202749
FNR: 0.0387177352206
ACC: 0.953738124742
F1_score: 0.976321353066
MCC: -0.0177762555585
informedness: -0.0387177352206
markedness: -0.00816151202749
prevalence: 0.992152003304
LRP: 0.961282264779
LRN: inf
DOR: 0.0
FOR: 1.0
OrderedDict([('population', 2421), ('P', 2402), ('N', 19), ('PositiveTest', 2328), ('NegativeTest', 93), ('TP', 2309), ('TN', 0), ('FP', 19), ('FN', 93), ('TPR', 0.96128226477935053), ('TNR', 0.0), ('PPV', 0.99183848797250862), ('NPV', 0.0), ('FPR', 1.0), ('FDR', 0.0081615120274914094), ('FNR', 0.03871773522064946), ('ACC', 0.95373812474184216), ('F1_score', 0.97632135306553913), ('MCC', -0.017776255558484701), ('informedness', -0.038717735220649474), ('markedness', -0.0081615120274913799), ('prevalence', 0.99215200330441966), ('LRP', 0.96128226477935053), ('LRN', inf), ('DOR', 0.0), ('FOR', 1.0)])
0.961282264779
==> Predicted positives: 2328 ( 2328 )
==> Ground truth positives: 2402 ( 2402 )
==> True positives: 2309
==> False positives: 19
==> Predicted negatives: 93 ( 93 )
==> Ground truth negatives: 19 ( 19 )
==> True negatives: 0
==> False negatives: 93
==> Precision (true positive/predicted positive): 0.991838487973
==> Recall (true positive/ground truth positive): 0.961282264779










    






/home/jonathanmorgan/.virtualenvs/sourcenet/lib/python3.5/site-packages/pandas_ml/confusion_matrix/bcm.py:339: RuntimeWarning: divide by zero encountered in double_scalars
  return(np.float64(self.FNR) / self.TNR)

















In [10]:



    


confusion_helper = ConfusionMatrixHelper.populate_confusion_matrix( ground_truth_list, predicted_list )
print( str( confusion_helper ) )



    


















    






ConfusionMatrixHelper --> 
DictHelper --> 
----> ACC : 0.9537381247418422
----> BM : -0.038717735220649474
----> DOR : None
----> FDR : 0.00816151202749141
----> FNR : 0.03871773522064946
----> FOR : 1.0
----> FPR : 1.0
----> LR+ : 0.9612822647793505
----> LR- : None
----> MCC : -0.0177762555584847
----> MK : -0.00816151202749138
----> NPV : 0.0
----> PPV : 0.9918384879725086
----> SPC : 0.0
----> TNR : 0.0
----> TPR : 0.9612822647793505
----> accuracy : 0.9537381247418422
----> diagnostic_odds_ratio : None
----> f1_score : 0.976321353065539
----> false_discovery_rate : 0.00816151202749141
----> false_negative : 93
----> false_negative_rate : 0.03871773522064946
----> false_omission_rate : 1.0
----> false_positive : 19
----> false_positive_rate : 1.0
----> informedness : -0.038717735220649474
----> markedness : -0.00816151202749138
----> matthews_correlation_coefficient : -0.0177762555584847
----> negative_likelihood_ratio : None
----> negative_predictive_value : 0.0
----> population_negative : 19
----> population_positive : 2402
----> positive_likelihood_ratio : 0.9612822647793505
----> precision : 0.9918384879725086
----> predicted_negative : 93
----> predicted_positive : 2328
----> recall : 0.9612822647793505
----> specificity : 0.0
----> total_population : 2421
----> true_negative : 0
----> true_negative_rate : 0.0
----> true_positive : 2309

Person lookup

For each person detected across the set of articles, look at whether the automated coder correctly looked up the person (so compare person IDs).

Person lookup - build value lists

First, build lists of ground truth and predicted values per person.





In [11]:



    


# declare variables
reliability_names_label = None
label_in_list = []
reliability_names_qs = None
ground_truth_coder_index = 1
predicted_coder_index = 2

# processing
column_name = ""
predicted_value = -1
predicted_list = []
ground_truth_value = -1
ground_truth_list = []
reliability_names_instance = None

# set label
reliability_names_label = current_label  # "prelim_month_human"

# lookup Reliability_Names for selected label
label_in_list.append( reliability_names_label )
reliability_names_qs = Reliability_Names.objects.filter( label__in = label_in_list )

print( "Found " + str( reliability_names_qs.count() ) + " rows with label in " + str( label_in_list ) )

# loop over records
predicted_value = -1
predicted_list = []
ground_truth_value = -1
ground_truth_list = []
ground_truth_positive_count = 0
predicted_positive_count = 0
true_positive_count = 0
false_positive_count = 0
ground_truth_negative_count = 0
predicted_negative_count = 0
true_negative_count = 0
false_negative_count = 0
for reliability_names_instance in reliability_names_qs:
    
    # get person_id from ground truth and predicted columns and add them to list.
    
    # ==> ground truth
    column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
    column_name += str( ground_truth_coder_index )
    column_name += "_" + Reliability_Names.FIELD_NAME_SUFFIX_PERSON_ID
    ground_truth_value = getattr( reliability_names_instance, column_name )
    ground_truth_list.append( ground_truth_value )
    
    # ==> predicted
    column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
    column_name += str( predicted_coder_index )
    column_name += "_" + Reliability_Names.FIELD_NAME_SUFFIX_PERSON_ID
    predicted_value = getattr( reliability_names_instance, column_name )
    predicted_list.append( predicted_value )
    
#-- END loop over Reliability_Names instances. --#

print( "==> population values count: " + str( len( ground_truth_list ) ) )
print( "==> predicted values count: " + str( len( predicted_list ) ) )
print( "==> percentage agreement = " + str( StatsHelper.percentage_agreement( ground_truth_list, predicted_list ) ) )



    


















    






Found 2421 rows with label in ['prelim_month_human']
==> population values count: 2421
==> predicted values count: 2421
==> percentage agreement = 0.953325072284

















In [12]:



    


print( "==> population values: " + str( len( ground_truth_list ) ) )
list_name = "ACTUAL_VALUE_LIST"
string_list = map( str, ground_truth_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )

print( "==> predicted values count: " + str( len( predicted_list ) ) )
list_name = "PREDICTED_VALUE_LIST"
string_list = map( str, predicted_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )



    


















    






==> population values: 2421
ACTUAL_VALUE_LIST = [ 591, 1772, 1774, 1773, 73, 237, 620, 1321, 394, 66, 84, 591, 1704, 1133, 1698, 451, 1768, 102, 1769, 1700, 1705, 1708, 1707, 1702, 1765, 1767, 1766, 419, 1701, 1697, 1703, 416, 1706, 1699, 46, 1715, 12, 1716, 394, 29, 46, 377, 1695, 1696, 1693, 1546, 1691, 1692, 100, 1688, 1687, 1690, 386, 1689, 1694, 398, 161, 46, 1770, 1794, 1793, 617, 53, 102, 1442, 1771, 13, 1795, 15, 437, 377, 84, 84, 1813, 1814, 1811, 218, 664, 1817, 0, 1815, 1967, 1059, 1816, 1812, 2797, 1966, 437, 46, 1972, 1080, 1861, 1819, 1710, 1969, 218, 1709, 1817, 1059, 1059, 1970, 1971, 516, 591, 73, 1964, 1968, 1963, 1965, 1548, 36, 66, 73, 237, 1775, 1820, 1818, 1343, 620, 1819, 1779, 1776, 668, 0, 1344, 250, 1810, 1457, 1777, 1821, 1822, 1778, 618, 1780, 1781, 1319, 1453, 1823, 387, 2799, 1797, 1796, 443, 36, 13, 13, 1845, 1824, 668, 1847, 1832, 1718, 1846, 161, 73, 620, 1715, 1717, 716, 1778, 618, 1716, 66, 425, 1723, 1724, 1833, 1835, 1834, 377, 1109, 1693, 188, 1443, 1694, 30, 84, 84, 1719, 950, 1825, 1826, 1973, 628, 949, 951, 1974, 32, 437, 1980, 394, 387, 1829, 1828, 1851, 1850, 1849, 852, 1827, 1852, 1848, 1830, 1831, 46, 1688, 1687, 1976, 1975, 74, 223, 178, 1713, 1979, 1722, 1441, 102, 1712, 290, 1711, 1721, 1977, 250, 1714, 1978, 337, 1720, 443, 599, 1741, 1740, 1739, 1890, 1889, 1738, 1743, 1742, 591, 1725, 29, 29, 0, 66, 84, 1888, 950, 1988, 1726, 1986, 1727, 1987, 1692, 386, 574, 1984, 1985, 1841, 1729, 628, 1547, 949, 951, 1183, 1839, 1728, 302, 73, 1983, 901, 1981, 1982, 437, 66, 69, 84, 1731, 1860, 1732, 1734, 1737, 2805, 1735, 1733, 1858, 1730, 1203, 1859, 1857, 1736, 36, 13, 1900, 1902, 1901, 1908, 1907, 1899, 743, 2013, 2012, 69, 46, 377, 1760, 1749, 839, 102, 218, 1748, 1747, 1759, 131, 1287, 1674, 762, 1751, 1750, 178, 460, 84, 1744, 1892, 1895, 102, 1891, 1746, 1994, 1898, 1995, 1745, 1896, 1996, 1997, 1893, 1897, 1993, 1894, 1992, 332, 1056, 1758, 2004, 652, 437, 66, 1205, 1109, 1906, 1903, 1956, 617, 1955, 2011, 1757, 1321, 2007, 1642, 1909, 2008, 2009, 1905, 1756, 1904, 2010, 2005, 1910, 1755, 2006, 660, 637, 637, 1754, 1753, 1913, 1752, 417, 1911, 425, 2821, 1761, 1762, 1764, 1763, 377, 178, 2014, 2017, 100, 2822, 290, 1445, 2016, 250, 13, 357, 352, 353, 355, 356, 354, 69, 69, 69, 345, 0, 834, 339, 833, 832, 340, 346, 344, 178, 846, 845, 844, 347, 69, 344, 343, 848, 566, 332, 332, 336, 336, 1960, 334, 334, 1956, 333, 333, 852, 852, 1955, 1961, 1957, 850, 850, 338, 338, 1958, 1962, 853, 853, 1959, 337, 337, 851, 851, 335, 335, 761, 761, 302, 302, 302, 859, 857, 328, 331, 329, 330, 858, 324, 856, 323, 325, 322, 302, 599, 319, 327, 863, 321, 1914, 326, 865, 861, 866, 862, 864, 320, 860, 762, 1082, 1782, 1784, 1786, 1785, 1783, 2823, 1082, 1082, 1943, 1950, 1940, 1925, 1937, 1929, 1923, 1935, 1787, 1919, 1936, 1934, 1920, 1944, 1954, 1932, 1922, 1938, 1945, 1924, 1789, 1952, 1951, 1946, 1930, 1948, 1953, 1931, 1915, 1949, 1918, 1917, 1926, 1921, 1933, 1942, 1916, 1939, 1790, 1927, 1928, 1947, 1788, 1941, 74, 74, 867, 374, 375, 763, 373, 376, 868, 30, 178, 591, 372, 368, 371, 370, 363, 366, 369, 367, 365, 30, 364, 161, 161, 46, 46, 358, 165, 166, 164, 870, 163, 871, 362, 186, 1791, 361, 162, 360, 74, 36, 410, 406, 412, 405, 411, 408, 407, 84, 345, 0, 346, 29, 66, 384, 392, 382, 385, 383, 393, 381, 591, 404, 402, 403, 187, 394, 46, 178, 400, 181, 397, 396, 180, 882, 876, 880, 883, 401, 881, 179, 399, 395, 182, 398, 879, 29, 387, 30, 750, 173, 390, 386, 175, 176, 389, 174, 32, 391, 388, 172, 377, 336, 1448, 380, 379, 378, 886, 884, 29, 36, 13, 524, 764, 533, 529, 532, 523, 531, 525, 37, 530, 599, 591, 526, 892, 893, 895, 890, 894, 0, 891, 528, 633, 527, 217, 178, 336, 371, 896, 534, 543, 542, 897, 521, 441, 520, 522, 535, 582, 898, 516, 519, 84, 900, 539, 766, 899, 161, 73, 73, 237, 236, 902, 534, 541, 901, 518, 239, 886, 767, 302, 537, 538, 536, 217, 217, 66, 84, 904, 218, 512, 515, 753, 514, 769, 516, 517, 513, 505, 507, 506, 509, 508, 3, 302, 23, 425, 46, 591, 915, 413, 909, 132, 234, 913, 912, 918, 916, 917, 510, 911, 414, 415, 418, 419, 213, 910, 215, 914, 511, 908, 416, 74, 74, 36, 13, 274, 501, 498, 500, 502, 272, 275, 499, 503, 273, 3, 161, 46, 178, 485, 487, 770, 932, 24, 930, 489, 490, 441, 488, 491, 931, 481, 928, 486, 23, 84, 269, 936, 933, 935, 484, 268, 23, 66, 69, 937, 46, 478, 404, 476, 188, 188, 0, 477, 475, 480, 0, 479, 346, 387, 474, 46, 909, 132, 913, 912, 911, 213, 473, 914, 74, 599, 13, 13, 945, 562, 559, 556, 560, 557, 561, 558, 377, 947, 564, 2179, 217, 69, 46, 178, 958, 955, 295, 950, 954, 578, 772, 580, 574, 579, 773, 628, 949, 951, 957, 250, 956, 292, 753, 585, 771, 576, 584, 953, 952, 575, 948, 3, 959, 588, 589, 2180, 774, 217, 599, 583, 962, 565, 582, 581, 46, 776, 876, 563, 963, 223, 36, 69, 84, 572, 552, 551, 777, 570, 102, 102, 439, 553, 225, 290, 290, 555, 309, 568, 289, 569, 571, 554, 567, 217, 591, 545, 548, 971, 546, 137, 549, 547, 544, 443, 74, 36, 445, 444, 448, 977, 976, 446, 442, 447, 74, 979, 555, 978, 3, 69, 69, 460, 993, 25, 404, 451, 986, 450, 454, 188, 987, 102, 100, 988, 449, 452, 779, 985, 992, 981, 456, 996, 453, 383, 994, 990, 989, 982, 983, 984, 461, 66, 780, 0, 462, 463, 29, 46, 458, 781, 455, 2183, 997, 999, 457, 459, 2004, 46, 84, 307, 308, 1004, 464, 2181, 2008, 309, 1005, 466, 1001, 1003, 2182, 425, 591, 472, 311, 470, 310, 471, 1007, 2184, 312, 161, 469, 468, 467, 443, 2197, 2190, 2194, 2195, 2196, 2826, 2193, 2191, 2198, 2189, 2188, 2192, 217, 74, 2233, 2212, 2213, 2232, 2214, 440, 516, 74, 2216, 2215, 377, 2185, 2187, 188, 102, 1815, 2827, 1203, 2186, 394, 394, 302, 368, 2206, 954, 2211, 2829, 2204, 2201, 2205, 2210, 2202, 302, 599, 599, 1804, 1802, 1805, 2209, 2208, 1803, 2205, 2207, 1806, 1082, 1082, 703, 2836, 2838, 2200, 2199, 583, 2839, 2203, 1931, 1807, 2837, 1809, 1808, 598, 598, 598, 2841, 1800, 305, 1801, 1799, 1798, 1837, 1836, 599, 2218, 2220, 2219, 2217, 2221, 1655, 161, 29, 84, 73, 1343, 1838, 2223, 2319, 2320, 2224, 2325, 2222, 2326, 2327, 2225, 1778, 437, 73, 2322, 2329, 2330, 2324, 2321, 2323, 2331, 2328, 2332, 443, 417, 36, 2848, 2231, 2228, 2230, 2229, 2359, 2227, 2226, 1889, 29, 437, 599, 377, 950, 2356, 1843, 0, 1546, 1840, 386, 574, 1842, 1841, 628, 1547, 949, 951, 1839, 346, 1203, 2288, 952, 575, 948, 23, 599, 84, 84, 2358, 1014, 451, 438, 2339, 2260, 668, 2333, 669, 2336, 2335, 2337, 2334, 250, 2245, 2261, 2244, 2338, 1010, 2246, 276, 2357, 387, 425, 654, 2355, 2351, 102, 102, 2353, 2352, 2348, 2349, 289, 2350, 2354, 30, 377, 591, 73, 2249, 1853, 2250, 2248, 475, 2347, 1854, 1855, 2247, 2346, 652, 66, 69, 2240, 1206, 1844, 451, 2241, 534, 2237, 2268, 439, 2242, 2236, 2238, 1207, 2234, 2243, 2235, 516, 886, 2239, 2851, 443, 74, 1863, 137, 1862, 217, 437, 377, 1861, 2360, 2361, 2362, 904, 218, 2365, 2366, 1059, 2363, 516, 2364, 29, 1544, 1888, 1691, 1840, 386, 1842, 1839, 2317, 217, 534, 218, 2318, 1419, 1059, 516, 3, 161, 73, 1775, 487, 620, 1318, 2378, 1419, 2377, 2376, 901, 2379, 182, 516, 1778, 2381, 618, 2386, 1780, 2345, 2382, 2380, 1319, 1453, 767, 599, 30, 30, 2301, 2298, 2299, 1864, 2300, 2296, 2297, 1856, 1865, 66, 66, 1871, 1867, 1869, 1866, 1868, 2342, 1870, 2344, 2343, 2341, 2340, 505, 591, 188, 2383, 1601, 2387, 187, 1043, 2384, 2385, 1044, 2388, 443, 74, 425, 2389, 1877, 1876, 51, 1875, 633, 30, 2443, 750, 29, 591, 2398, 2399, 2392, 2393, 2400, 2401, 23, 66, 2374, 2403, 2373, 2136, 2375, 84, 1873, 1872, 1874, 29, 2371, 2368, 2367, 2369, 2027, 2372, 2860, 2370, 23, 437, 69, 377, 237, 2094, 0, 2396, 2390, 714, 250, 2397, 2391, 1886, 1733, 1730, 2394, 346, 2395, 2402, 443, 443, 74, 2414, 2411, 1991, 218, 2413, 0, 633, 2412, 599, 13, 2228, 2461, 2973, 2462, 2227, 2226, 1889, 2463, 387, 217, 2455, 2407, 753, 2456, 217, 66, 377, 84, 2406, 2405, 2404, 2464, 2465, 2469, 1989, 2459, 2096, 2457, 2467, 2466, 2468, 2458, 2460, 1990, 66, 1883, 1882, 1881, 1884, 23, 84, 1152, 2448, 0, 2454, 2450, 2965, 2452, 2447, 2451, 2446, 1886, 2449, 1885, 2453, 591, 1880, 138, 437, 69, 377, 2002, 2409, 1693, 1999, 2003, 218, 2001, 1289, 2408, 2000, 2410, 1878, 1879, 1998, 1286, 223, 2445, 204, 2444, 443, 74, 13, 2040, 2038, 1991, 2036, 2035, 2034, 2026, 2037, 2039, 591, 2041, 2042, 2044, 2043, 2045, 23, 652, 377, 219, 2432, 2032, 2033, 2095, 2433, 2474, 1674, 2473, 3, 2479, 2478, 437, 223, 13, 2535, 2470, 487, 1591, 2472, 2030, 102, 102, 2028, 290, 2027, 1977, 2031, 2471, 2029, 377, 2475, 1734, 100, 2477, 1203, 2476, 417, 2018, 2425, 2876, 2417, 2429, 2024, 2877, 2875, 2878, 2025, 2423, 2418, 2420, 2019, 2422, 2419, 2882, 2416, 2881, 2022, 2424, 2880, 2874, 2020, 2023, 2426, 2415, 2421, 2428, 2427, 2021, 1670, 161, 2431, 591, 2430, 1348, 591, 2493, 2495, 2494, 2492, 505, 223, 46, 2156, 2545, 2546, 184, 2475, 102, 2480, 2063, 290, 2064, 1977, 2066, 2067, 2549, 2547, 2548, 2155, 2544, 1730, 2065, 2543, 2550, 46, 2072, 2073, 2074, 2068, 2069, 2075, 2027, 2070, 2071, 217, 2439, 2438, 2441, 2442, 664, 2440, 2437, 753, 217, 2046, 2047, 2048, 302, 302, 599, 572, 2086, 2083, 2076, 2079, 1576, 2077, 2082, 2080, 2080, 2491, 2081, 2085, 2078, 2084, 2490, 599, 2483, 2484, 2481, 2482, 598, 598, 2090, 2087, 2088, 2488, 2091, 2489, 2089, 736, 598, 1082, 1082, 1782, 2051, 2052, 2053, 2059, 2486, 2050, 2485, 1608, 2487, 2892, 2434, 2054, 2435, 2436, 2055, 2056, 2057, 2058, 2060, 2049, 2061, 2062, 425, 2505, 2030, 2506, 2502, 2504, 2503, 23, 23, 30, 2498, 2094, 2497, 2500, 184, 102, 2096, 2098, 2095, 2097, 2499, 2099, 2501, 1755, 30, 364, 2496, 591, 1479, 2552, 2551, 217, 617, 439, 2093, 1287, 2092, 443, 591, 2569, 2558, 521, 2557, 502, 2568, 2905, 2904, 2556, 29, 46, 591, 2113, 548, 2111, 2112, 971, 2115, 1840, 137, 386, 1842, 1841, 2511, 1547, 1839, 138, 2114, 2106, 547, 23, 84, 1133, 2507, 2564, 2561, 2509, 2508, 2563, 2510, 2562, 3, 23, 505, 770, 2100, 24, 2122, 2124, 2123, 2101, 716, 2108, 2109, 2110, 2125, 394, 30, 591, 404, 954, 2102, 2103, 2265, 2512, 2104, 2105, 2107, 66, 178, 2907, 2127, 2129, 438, 0, 2118, 2121, 2117, 2116, 2126, 2120, 2128, 2119, 437, 438, 669, 13, 13, 2515, 2516, 2514, 2513, 2517, 23, 66, 66, 69, 2138, 950, 627, 2137, 2135, 2133, 574, 628, 949, 951, 1183, 2134, 2576, 569, 2136, 2577, 952, 575, 948, 437, 30, 2571, 2573, 2265, 2572, 2575, 2574, 223, 460, 84, 1015, 281, 451, 2132, 184, 1892, 1895, 102, 2280, 2281, 2521, 2282, 1898, 250, 2519, 2130, 1896, 2131, 1893, 1897, 2520, 1894, 29, 2570, 443, 443, 13, 2535, 2470, 2525, 2472, 2143, 2145, 2141, 2140, 2139, 2144, 2142, 2146, 460, 2579, 2581, 2919, 2582, 2584, 2583, 2585, 2578, 2918, 2580, 1472, 2586, 637, 66, 591, 2532, 2148, 2528, 487, 2533, 2529, 188, 2383, 2526, 1732, 2531, 1980, 2534, 2530, 2147, 2527, 3, 425, 178, 2589, 2153, 2233, 2522, 2152, 2151, 2524, 2588, 2149, 2587, 2523, 2150, 2154, 13, 591, 2540, 2158, 2925, 2157, 2542, 2541, 437, 437, 46, 2094, 102, 2592, 290, 2539, 1980, 2537, 1733, 2591, 2536, 2590, 2926, 2553, 2538, 29, 505, 2614, 2611, 2609, 2156, 2930, 2610, 2617, 2615, 2616, 2619, 2620, 2155, 2612, 2618, 2613, 443, 30, 2560, 2559, 2555, 2554, 443, 161, 387, 66, 66, 2656, 2660, 2655, 308, 742, 2663, 2666, 2662, 2935, 2658, 2629, 474, 2664, 2661, 2659, 2630, 2657, 2626, 2665, 0, 2934, 2627, 443, 36, 84, 2175, 2169, 2176, 2678, 2677, 2177, 2168, 2167, 2673, 2674, 0, 2937, 2675, 2676, 394, 417, 2159, 2163, 2165, 2162, 2161, 2671, 2166, 2160, 2164, 2672, 637, 2173, 2171, 2172, 2170, 29, 2939, 2700, 2701, 2703, 852, 2702, 2567, 482, 2608, 2607, 217, 217, 505, 505, 505, 74, 74, 74, 2255, 2255, 2255, 2599, 2599, 2599, 2565, 2565, 2566, 2566, 2254, 2254, 2254, 454, 454, 1476, 1476, 2265, 2265, 1300, 1298, 2263, 2263, 2256, 2266, 2266, 2600, 2600, 516, 516, 1203, 1203, 2262, 2262, 2264, 2264, 66, 66, 2633, 2633, 1819, 1819, 668, 668, 2632, 2632, 2631, 2631, 346, 346, 302, 599, 599, 2601, 2253, 2602, 2605, 2603, 2625, 2623, 2606, 2251, 2604, 2622, 2621, 2624, 2252, 599, 2686, 2685, 2684, 598, 1082, 1782, 2595, 2597, 2593, 1931, 2596, 2594, 2598, 2692, 1933, 598, 598, 703, 1782, 2699, 2697, 2690, 2687, 2691, 2257, 2696, 2688, 2944, 2693, 2259, 2698, 2258, 2689, 2695, 2694, 1808, 13, 2708, 2711, 2710, 2709, 84, 2726, 2950, 2724, 2725, 2727, 2721, 2723, 2722, 3, 29, 591, 2358, 2717, 2719, 2634, 2267, 2720, 2718, 2716, 3, 178, 2714, 2272, 2715, 2712, 2273, 2713, 2274, 2269, 2968, 2970, 2271, 2275, 984, 2270, 217, 425, 2636, 1299, 686, 1300, 1298, 1458, 2635, 217, 84, 2639, 534, 2637, 535, 2638, 516, 652, 2641, 2640, 2642, 2643, 3, 66, 332, 0, 1080, 1014, 2278, 451, 438, 0, 1292, 1892, 2260, 2286, 102, 668, 2280, 2287, 2284, 2281, 669, 2279, 2282, 250, 0, 1010, 276, 277, 1893, 1113, 2277, 2285, 2283, 161, 102, 558, 2728, 36, 425, 2291, 2290, 102, 2650, 2292, 2315, 2649, 2289, 2647, 2648, 161, 161, 23, 66, 73, 2295, 2295, 2305, 2956, 2738, 2294, 2294, 2743, 668, 553, 2306, 2742, 2293, 2293, 2309, 2741, 250, 2307, 2739, 2737, 276, 2740, 1113, 2308, 652, 66, 66, 219, 2591, 2590, 2926, 2644, 2304, 2302, 2303, 3, 29, 178, 84, 2014, 2654, 2651, 2729, 2731, 218, 2733, 2645, 386, 225, 2652, 2653, 2730, 2734, 1733, 2736, 2646, 1287, 2735, 2732, 591, 2744, 2745, 2310, 66, 2311, 513, 66, 377, 84, 2960, 2751, 2749, 2750, 2668, 2667, 2670, 102, 2752, 1734, 2669, 2476, 3, 23, 74, 2316, 2312, 2314, 2679, 236, 717, 2682, 2962, 2313, 2747, 2748, 2746, 2309, 2315, 716, 1968, 2680, 2681, 2683, 1548 ]
==> predicted values count: 2421
PREDICTED_VALUE_LIST = [ 591, 1772, 1774, 1773, 73, 237, 620, 1321, 394, 66, 84, 591, 1704, 1133, 1698, 451, 1768, 102, 1769, 1700, 1705, 1708, 1707, 1702, 1765, 1767, 1766, 419, 1701, 1697, 1703, 416, 1706, 1699, 46, 1715, 12, 1716, 394, 29, 46, 377, 1695, 1696, 1693, 1546, 1691, 1692, 100, 1688, 1687, 1690, 386, 1689, 1694, 398, 161, 46, 1770, 1794, 1793, 617, 53, 102, 1442, 1771, 13, 1795, 15, 437, 377, 84, 84, 1813, 1814, 1811, 218, 664, 1817, 1449, 1815, 1967, 1059, 1816, 1812, 0, 1966, 437, 46, 1972, 1080, 1861, 1819, 1710, 1969, 218, 1709, 1817, 1059, 1059, 1970, 1971, 516, 591, 73, 1964, 1968, 1963, 1965, 1548, 36, 0, 73, 237, 1775, 1820, 1818, 1343, 620, 1819, 1779, 1776, 668, 66, 1344, 250, 1810, 1457, 1777, 1821, 1822, 1778, 618, 1780, 1781, 1319, 1453, 1823, 387, 0, 1797, 1796, 443, 36, 13, 13, 1845, 1824, 668, 1847, 1832, 1718, 1846, 161, 73, 0, 1715, 1717, 716, 1778, 618, 1716, 66, 425, 1723, 1724, 1833, 1835, 1834, 377, 1109, 1693, 188, 1443, 1694, 30, 84, 84, 1719, 950, 1825, 1826, 1973, 628, 0, 0, 1974, 32, 437, 1980, 394, 387, 1829, 1828, 1851, 1850, 1849, 852, 1827, 1852, 1848, 1830, 1831, 46, 1688, 1687, 1976, 1975, 74, 223, 178, 1713, 1979, 1722, 1441, 102, 1712, 290, 1711, 1721, 1977, 250, 1714, 1978, 337, 1720, 443, 599, 1741, 1740, 1739, 1890, 1889, 1738, 1743, 1742, 591, 1725, 29, 29, 1692, 66, 84, 1888, 950, 1988, 1726, 1986, 1727, 1987, 0, 0, 574, 1984, 1985, 1841, 1729, 628, 1547, 0, 951, 1183, 1839, 1728, 302, 73, 1983, 901, 1981, 1982, 437, 66, 69, 84, 1731, 1860, 1732, 1734, 1737, 0, 1735, 1733, 1858, 1730, 1203, 1859, 1857, 1736, 36, 13, 1900, 1902, 1901, 1908, 1907, 1899, 743, 2013, 2012, 69, 46, 377, 1760, 1749, 839, 102, 218, 1748, 1747, 1759, 131, 1287, 1674, 762, 1751, 1750, 178, 460, 84, 1744, 1892, 1895, 102, 1891, 1746, 1994, 1898, 1995, 1745, 1896, 1996, 1997, 1893, 1897, 1993, 1894, 1992, 332, 1056, 1758, 2004, 652, 437, 66, 1205, 1109, 1906, 1903, 1956, 617, 1955, 2011, 1757, 1321, 2007, 1642, 1909, 2008, 2009, 1905, 1756, 1904, 2010, 2005, 1910, 1755, 2006, 660, 637, 637, 1754, 1753, 1913, 1752, 417, 1911, 425, 0, 1761, 1762, 1764, 1763, 377, 178, 2014, 2017, 100, 2015, 290, 1445, 2016, 250, 13, 357, 352, 353, 355, 356, 354, 69, 69, 69, 345, 839, 834, 339, 833, 0, 340, 346, 344, 178, 846, 0, 844, 347, 69, 344, 0, 848, 566, 332, 332, 336, 336, 1960, 334, 334, 1956, 333, 333, 852, 852, 1955, 1961, 1957, 850, 850, 338, 338, 1958, 1962, 853, 853, 1959, 337, 337, 851, 851, 335, 335, 761, 761, 302, 302, 302, 859, 0, 328, 331, 329, 330, 858, 324, 856, 323, 325, 322, 302, 599, 319, 327, 863, 321, 1914, 326, 0, 861, 866, 862, 0, 320, 860, 0, 1082, 1782, 1784, 1786, 1785, 1783, 0, 1082, 1082, 1943, 1950, 1940, 1925, 1937, 1929, 1923, 1935, 1787, 1919, 1936, 1934, 1920, 1944, 1954, 1932, 1922, 1938, 1945, 1924, 1789, 1952, 1951, 1946, 1930, 1948, 1953, 1931, 1915, 1949, 1918, 1917, 1926, 1921, 1933, 1942, 1916, 1939, 1790, 1927, 1928, 1947, 1788, 1941, 74, 74, 867, 374, 375, 763, 373, 376, 868, 30, 178, 591, 372, 368, 371, 370, 363, 366, 369, 367, 365, 30, 364, 161, 161, 46, 46, 358, 165, 166, 164, 870, 163, 871, 362, 186, 1791, 0, 162, 360, 74, 36, 410, 406, 412, 405, 411, 408, 407, 84, 345, 839, 346, 29, 66, 384, 392, 0, 0, 383, 0, 381, 591, 404, 402, 403, 187, 394, 46, 178, 400, 181, 397, 396, 180, 882, 876, 880, 883, 401, 881, 179, 399, 395, 182, 398, 879, 29, 387, 30, 750, 173, 390, 386, 175, 176, 389, 174, 32, 391, 388, 172, 377, 336, 1448, 380, 379, 378, 886, 884, 29, 36, 13, 524, 764, 533, 529, 532, 523, 531, 525, 37, 530, 599, 591, 526, 892, 893, 0, 890, 894, 1792, 891, 528, 633, 527, 0, 178, 336, 371, 896, 534, 543, 542, 897, 521, 441, 520, 522, 535, 582, 898, 516, 519, 84, 900, 539, 766, 899, 161, 73, 73, 237, 236, 902, 534, 541, 901, 518, 239, 886, 767, 302, 537, 538, 536, 217, 217, 66, 84, 904, 218, 512, 515, 753, 514, 769, 516, 517, 513, 505, 507, 506, 509, 508, 3, 302, 23, 425, 46, 591, 915, 413, 909, 132, 234, 913, 912, 918, 916, 917, 510, 911, 414, 415, 418, 419, 213, 910, 215, 914, 511, 908, 416, 74, 74, 36, 13, 274, 501, 498, 500, 502, 272, 275, 499, 503, 273, 3, 161, 46, 178, 485, 487, 770, 932, 24, 930, 489, 490, 441, 488, 491, 931, 481, 928, 486, 23, 84, 269, 936, 933, 935, 484, 268, 23, 66, 69, 937, 0, 478, 404, 476, 188, 188, 839, 477, 475, 480, 46, 479, 346, 387, 474, 46, 909, 132, 913, 912, 911, 213, 473, 914, 74, 599, 13, 13, 945, 562, 559, 556, 560, 557, 561, 558, 377, 947, 564, 2179, 217, 69, 46, 178, 958, 955, 295, 950, 954, 578, 772, 580, 574, 579, 773, 628, 0, 951, 957, 0, 956, 292, 753, 585, 771, 576, 584, 953, 952, 575, 948, 3, 959, 588, 589, 2180, 774, 217, 599, 583, 0, 565, 582, 581, 46, 776, 876, 563, 963, 223, 36, 69, 84, 572, 552, 551, 777, 570, 102, 102, 439, 553, 225, 290, 290, 555, 309, 0, 289, 569, 571, 554, 567, 217, 591, 545, 548, 971, 546, 137, 549, 547, 0, 443, 74, 36, 445, 444, 448, 0, 976, 446, 442, 447, 74, 979, 555, 978, 3, 69, 69, 460, 993, 25, 404, 451, 986, 450, 454, 188, 987, 0, 100, 988, 449, 452, 779, 985, 992, 981, 456, 996, 453, 383, 994, 990, 989, 0, 983, 984, 461, 66, 780, 839, 462, 463, 29, 46, 458, 781, 455, 2183, 997, 999, 457, 459, 2004, 46, 84, 307, 308, 1004, 464, 2181, 2008, 309, 1005, 466, 1001, 1003, 2182, 425, 591, 472, 311, 470, 310, 471, 1007, 2184, 312, 161, 469, 468, 467, 443, 2197, 2190, 2194, 2195, 2196, 0, 2193, 2191, 2198, 2189, 2188, 2192, 217, 74, 2233, 2212, 2213, 2232, 2214, 440, 516, 74, 2216, 2215, 377, 2185, 2187, 188, 102, 1815, 0, 1203, 2186, 394, 394, 302, 368, 2206, 954, 2211, 0, 2204, 2201, 2205, 2210, 2202, 302, 599, 599, 1804, 1802, 1805, 2209, 2208, 1803, 2205, 2207, 1806, 1082, 1082, 703, 0, 0, 2200, 2199, 583, 0, 2203, 1931, 1807, 0, 1809, 1808, 598, 598, 598, 0, 1800, 305, 1801, 1799, 1798, 1837, 1836, 599, 2218, 2220, 2219, 2217, 2221, 1655, 161, 29, 84, 73, 1343, 1838, 2223, 2319, 2320, 2224, 2325, 2222, 2326, 2327, 2225, 1778, 437, 73, 2322, 2329, 2330, 2324, 2321, 2323, 2331, 2328, 2332, 443, 417, 36, 0, 2231, 2228, 2230, 2229, 2359, 2227, 2226, 1889, 29, 437, 599, 377, 950, 2356, 1843, 839, 1546, 1840, 386, 574, 1842, 1841, 628, 1547, 0, 951, 1839, 346, 1203, 2288, 952, 0, 948, 23, 599, 84, 84, 2358, 1014, 451, 438, 2339, 2260, 668, 2333, 669, 2336, 2335, 2337, 2334, 250, 2245, 2261, 2244, 2338, 1010, 2246, 276, 2357, 387, 425, 654, 2355, 2351, 102, 102, 2353, 2352, 2348, 2349, 289, 2350, 2354, 30, 377, 591, 73, 2249, 1853, 2250, 2248, 475, 2347, 1854, 1855, 2247, 2346, 652, 66, 69, 2240, 1206, 1844, 451, 2241, 534, 2237, 2268, 439, 2242, 2236, 2238, 1207, 2234, 2243, 2235, 516, 886, 2239, 0, 443, 74, 1863, 137, 1862, 217, 437, 377, 1861, 2360, 2361, 2362, 904, 218, 2365, 2366, 1059, 2363, 516, 2364, 29, 1544, 1888, 1691, 1840, 386, 1842, 1839, 2317, 217, 534, 218, 2318, 1419, 1059, 516, 3, 161, 73, 1775, 487, 620, 1318, 2378, 1419, 2377, 2376, 901, 2379, 182, 516, 1778, 2381, 618, 2386, 1780, 2345, 2382, 2380, 1319, 0, 767, 599, 30, 30, 2301, 2298, 2299, 1864, 2300, 2296, 2297, 1856, 1865, 66, 66, 1871, 1867, 1869, 1866, 1868, 2342, 1870, 2344, 2343, 2341, 2340, 505, 591, 188, 2383, 1601, 2387, 187, 1043, 2384, 2385, 1044, 2388, 443, 74, 425, 2389, 1877, 1876, 51, 1875, 633, 30, 2443, 750, 29, 591, 2398, 2399, 2392, 2393, 2400, 2401, 23, 66, 2374, 2403, 2373, 2136, 2375, 84, 1873, 1872, 1874, 29, 2371, 2368, 2367, 2369, 2027, 2372, 0, 2370, 23, 437, 69, 377, 237, 2094, 839, 2396, 2390, 714, 0, 2397, 2391, 1886, 1733, 1730, 2394, 346, 2395, 2402, 443, 443, 74, 2414, 2411, 1991, 218, 2413, 1792, 633, 2412, 599, 13, 2228, 2461, 0, 2462, 2227, 2226, 1889, 2463, 387, 217, 2455, 2407, 753, 2456, 217, 66, 377, 84, 2406, 2405, 2404, 2464, 2465, 2469, 1989, 2459, 2096, 2457, 2467, 2466, 2468, 2458, 2460, 1990, 66, 1883, 1882, 1881, 1884, 23, 84, 1152, 2448, 1887, 2454, 2450, 0, 2452, 2447, 2451, 2446, 1886, 2449, 1885, 2453, 591, 1880, 138, 437, 69, 377, 2002, 2409, 1693, 1999, 2003, 218, 2001, 1289, 2408, 2000, 2410, 1878, 1879, 1998, 1286, 223, 2445, 204, 2444, 443, 74, 13, 2040, 2038, 1991, 2036, 2035, 2034, 2026, 2037, 2039, 591, 2041, 2042, 2044, 2043, 2045, 23, 652, 377, 219, 2432, 2032, 2033, 2095, 2433, 2474, 1674, 2473, 3, 2479, 2478, 437, 223, 13, 0, 2470, 487, 1591, 2472, 2030, 102, 102, 2028, 290, 2027, 1977, 2031, 2471, 2029, 377, 2475, 1734, 100, 2477, 1203, 2476, 417, 2018, 2425, 0, 2417, 2429, 2024, 0, 0, 0, 2025, 2423, 2418, 2420, 2019, 2422, 2419, 0, 2416, 0, 2022, 2424, 0, 0, 2020, 2023, 2426, 2415, 2421, 2428, 2427, 2021, 1670, 161, 2431, 591, 2430, 1348, 591, 2493, 2495, 2494, 2492, 505, 223, 46, 2156, 2545, 2546, 184, 2475, 102, 2480, 2063, 290, 2064, 1977, 2066, 2067, 2549, 2547, 2548, 2155, 2544, 1730, 2065, 2543, 2550, 46, 2072, 2073, 2074, 2068, 2069, 2075, 2027, 2070, 2071, 217, 2439, 2438, 2441, 2442, 664, 2440, 2437, 753, 217, 2046, 2047, 2048, 302, 302, 599, 572, 2086, 2083, 2076, 2079, 1576, 2077, 2082, 2080, 2080, 2491, 2081, 2085, 2078, 2084, 2490, 599, 2483, 2484, 2481, 2482, 598, 598, 2090, 2087, 2088, 2488, 2091, 2489, 2089, 736, 598, 1082, 1082, 1782, 2051, 2052, 2053, 2059, 2486, 2050, 2485, 1608, 2487, 0, 2434, 2054, 2435, 2436, 2055, 2056, 2057, 2058, 2060, 2049, 2061, 2062, 425, 2505, 2030, 2506, 2502, 2504, 2503, 23, 23, 30, 2498, 2094, 2497, 2500, 184, 102, 2096, 2098, 2095, 2097, 2499, 2099, 2501, 1755, 30, 364, 2496, 591, 1479, 2552, 2551, 217, 617, 439, 2093, 1287, 2092, 443, 591, 2569, 2558, 521, 2557, 502, 2568, 0, 0, 2556, 29, 46, 591, 2113, 548, 2111, 2112, 971, 2115, 1840, 137, 386, 1842, 1841, 2511, 1547, 1839, 138, 2114, 2106, 547, 23, 84, 1133, 2507, 2564, 2561, 2509, 2508, 2563, 2510, 2562, 3, 23, 505, 770, 2100, 24, 2122, 2124, 2123, 2101, 716, 2108, 2109, 2110, 2125, 394, 30, 591, 404, 954, 2102, 2103, 2265, 2512, 2104, 2105, 2107, 66, 178, 0, 2127, 2129, 438, 839, 2118, 2121, 2117, 2116, 2126, 2120, 2128, 2119, 437, 0, 669, 13, 13, 2515, 2516, 2514, 2513, 2517, 23, 66, 66, 69, 2138, 950, 627, 2137, 2135, 2133, 574, 628, 0, 951, 0, 2134, 2576, 569, 2136, 2577, 952, 0, 948, 437, 30, 2571, 2573, 2265, 2572, 2575, 2574, 223, 460, 84, 1015, 281, 451, 2132, 184, 1892, 1895, 102, 2280, 2281, 2521, 2282, 1898, 250, 2519, 2130, 1896, 2131, 1893, 1897, 2520, 1894, 29, 2570, 443, 443, 13, 2535, 2470, 2525, 2472, 2143, 2145, 2141, 2140, 2139, 2144, 2142, 2146, 460, 2579, 2581, 0, 2582, 2584, 2583, 2585, 2578, 0, 2580, 1472, 2586, 637, 66, 591, 2532, 2148, 2528, 487, 2533, 2529, 188, 2383, 2526, 1732, 2531, 1980, 2534, 2530, 2147, 2527, 3, 425, 178, 2589, 2153, 2233, 2522, 2152, 2151, 2524, 2588, 2149, 2587, 2523, 2150, 2154, 13, 591, 2540, 2158, 0, 2157, 2542, 2541, 437, 437, 46, 2094, 102, 2592, 290, 2539, 1980, 2537, 1733, 2591, 2536, 2590, 0, 2553, 2538, 29, 505, 2614, 2611, 2609, 2156, 0, 2610, 2617, 2615, 2616, 2619, 2620, 2155, 2612, 2618, 2613, 443, 30, 2560, 2559, 2555, 2554, 443, 161, 387, 66, 66, 2656, 2660, 2655, 308, 742, 2663, 2666, 2662, 0, 2658, 2629, 474, 2664, 2661, 2659, 2630, 2657, 2626, 2665, 2628, 0, 2627, 443, 36, 84, 2175, 2169, 2176, 2678, 2677, 2177, 2168, 2167, 2673, 2674, 2174, 0, 2675, 2676, 394, 417, 2159, 2163, 2165, 2162, 2161, 2671, 2166, 2160, 2164, 2672, 637, 2173, 2171, 2172, 2170, 29, 0, 2700, 2701, 2703, 852, 2702, 2567, 482, 2608, 2607, 217, 217, 505, 505, 505, 74, 74, 74, 2255, 2255, 2255, 0, 2599, 2599, 2565, 2565, 2566, 2566, 2254, 2254, 2254, 454, 454, 1476, 1476, 2265, 2265, 1300, 0, 2263, 2263, 2256, 2266, 2266, 2600, 2600, 516, 516, 1203, 1203, 2262, 2262, 2264, 2264, 66, 66, 2633, 2633, 1819, 1819, 668, 668, 2632, 2632, 2631, 2631, 346, 346, 302, 599, 599, 2601, 2253, 2602, 2605, 2603, 2625, 2623, 2606, 2251, 2604, 2622, 2621, 2624, 2252, 599, 2686, 2685, 2684, 598, 1082, 1782, 2595, 2597, 2593, 1931, 2596, 2594, 2598, 2692, 1933, 598, 598, 703, 1782, 2699, 2697, 2690, 2687, 2691, 2257, 2696, 2688, 0, 2693, 2259, 2698, 2258, 2689, 2695, 2694, 1808, 13, 2708, 2711, 2710, 2709, 84, 2726, 0, 2724, 2725, 2727, 2721, 2723, 2722, 3, 29, 591, 2358, 2717, 2719, 2634, 2267, 2720, 2718, 2716, 3, 178, 2714, 2272, 2715, 2712, 2273, 2713, 2274, 2269, 0, 0, 2271, 2275, 984, 2270, 217, 425, 2636, 1299, 686, 1300, 1298, 0, 2635, 217, 84, 2639, 534, 2637, 535, 2638, 516, 652, 2641, 2640, 2642, 2643, 0, 66, 0, 3, 1080, 1014, 2278, 451, 438, 2276, 0, 1892, 0, 2286, 102, 668, 2280, 2287, 2284, 2281, 669, 2279, 2282, 250, 332, 1010, 276, 277, 1893, 1113, 2277, 2285, 2283, 161, 102, 558, 2728, 36, 425, 2291, 2290, 102, 2650, 2292, 2315, 2649, 2289, 2647, 2648, 161, 161, 23, 66, 73, 2295, 2295, 2305, 0, 2738, 2294, 2294, 2743, 668, 553, 2306, 2742, 0, 2293, 2309, 2741, 250, 2307, 2739, 2737, 276, 2740, 1113, 2308, 652, 66, 66, 219, 2591, 2590, 0, 2644, 2304, 2302, 2303, 3, 29, 178, 84, 2014, 2654, 2651, 2729, 2731, 218, 2733, 2645, 386, 225, 2652, 2653, 2730, 2734, 1733, 2736, 2646, 1287, 2735, 2732, 591, 2744, 2745, 2310, 66, 2311, 513, 66, 377, 84, 0, 2751, 2749, 2750, 2668, 2667, 2670, 102, 2752, 1734, 2669, 2476, 3, 23, 74, 2316, 2312, 2314, 2679, 236, 717, 2682, 0, 2313, 2747, 2748, 2746, 2309, 2315, 716, 1968, 2680, 2681, 2683, 1548 ]

Person lookup - confusion matrix





In [13]:



    


confusion_helper = ConfusionMatrixHelper.populate_confusion_matrix( ground_truth_list, predicted_list )
print( str( confusion_helper ) )



    


















    






ConfusionMatrixHelper --> 
DictHelper --> 
----> ACC : 0.9533250722841801
----> BM : -0.039134054954204855
----> DOR : None
----> FDR : 0.00859106529209622
----> FNR : 0.03871773522064946
----> FOR : 1.0
----> FPR : 1.0526315789473684
----> LR+ : 0.9128226477935054
----> LR- : None
----> MCC : -0.01824185100070961
----> MK : -0.00859106529209619
----> NPV : 0.0
----> PPV : 0.9914089347079038
----> SPC : 0.0
----> TNR : 0.0
----> TPR : 0.9608659450457951
----> accuracy : 0.9533250722841801
----> diagnostic_odds_ratio : None
----> f1_score : 0.9758985200845665
----> false_discovery_rate : 0.00859106529209622
----> false_negative : 93
----> false_negative_rate : 0.03871773522064946
----> false_omission_rate : 1.0
----> false_positive : 20
----> false_positive_rate : 1.0526315789473684
----> informedness : -0.039134054954204855
----> markedness : -0.00859106529209619
----> matthews_correlation_coefficient : -0.01824185100070961
----> negative_likelihood_ratio : None
----> negative_predictive_value : 0.0
----> population_negative : 19
----> population_positive : 2402
----> positive_likelihood_ratio : 0.9128226477935054
----> precision : 0.9914089347079038
----> predicted_negative : 93
----> predicted_positive : 2328
----> recall : 0.9608659450457951
----> specificity : 0.0
----> total_population : 2421
----> true_negative : 0
----> true_negative_rate : 0.0
----> true_positive : 2308

Person Types

For each person type, will build binary lists of yes or no where each person type value will in turn be the value of interest, and positive or negative is whether the coder found the current person to be of that type (positive/1) or any other type (negative/0).

Function: build confusion lists for a given categorical value





In [14]:



    


def build_confusion_lists( column_name_suffix_IN,
                           desired_value_IN,
                           label_list_IN = [ "prelim_month_human", ],
                           ground_truth_coder_index_IN = 1,
                           predicted_coder_index_IN = 2,
                           debug_flag_IN = False ):

    '''
    Accepts suffix of column name of interest and desired value.  Also accepts optional labels
        list, indexes of ground_truth and predicted coder users, and a debug flag.  Uses these
        values to loop over records whose label matches the on in the list passed in.  For each,
        in the specified column, checks to see if the ground_truth and predicted values match
        the desired value.  If so, positive, so 1 is stored for the row.  If no, negative, so 0
        is stored for the row.
        
    Returns dictionary with value lists inside, ground truth values list mapped to key
        "ground_truth" and predicted values list mapped to key "predicted".
    '''
    
    # return reference
    lists_OUT = {}

    # declare variables
    reliability_names_label = None
    label_in_list = []
    reliability_names_qs = None
    ground_truth_coder_index = -1
    predicted_coder_index = -1

    # processing
    debug_flag = False
    desired_column_suffix = None
    desired_value = None
    ground_truth_column_name = None
    ground_truth_column_value = None
    ground_truth_value = -1
    ground_truth_list = []
    predicted_column_name = None
    predicted_column_value = None
    predicted_value = -1
    predicted_list = []
    reliability_names_instance = None

    # got required values?
    
    # column name suffix?
    if ( column_name_suffix_IN is not None ):
        
        # desired value?
        if ( desired_value_IN is not None ):
    
            # ==> initialize
            desired_column_suffix = column_name_suffix_IN
            desired_value = desired_value_IN
            label_in_list = label_list_IN
            ground_truth_coder_index = ground_truth_coder_index_IN
            predicted_coder_index = predicted_coder_index_IN
            debug_flag = debug_flag_IN
            
            # create ground truth column name
            ground_truth_column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
            ground_truth_column_name += str( ground_truth_coder_index )
            ground_truth_column_name += "_" + desired_column_suffix

            # create predicted column name.
            predicted_column_name = Reliability_Names.FIELD_NAME_PREFIX_CODER
            predicted_column_name += str( predicted_coder_index )
            predicted_column_name += "_" + desired_column_suffix

            # ==> processing
            
            # lookup Reliability_Names for selected label(s)
            reliability_names_qs = Reliability_Names.objects.filter( label__in = label_in_list )

            print( "Found " + str( reliability_names_qs.count() ) + " rows with label in " + str( label_in_list ) )

            # reset all lists and values.
            ground_truth_column_value = ""
            ground_truth_value = -1
            ground_truth_list = []
            predicted_column_value = ""
            predicted_value = -1
            predicted_list = []
            
            # loop over records to build ground_truth and predicted value lists
            #     where 1 = value matching desired value in multi-value categorical
            #     variable and 0 = any value other than the desired value.
            for reliability_names_instance in reliability_names_qs:

                # get detected flag from ground truth and predicted columns and add them to list.

                # ==> ground truth

                # get column value.
                ground_truth_column_value = getattr( reliability_names_instance, ground_truth_column_name )

                # does it match desired value?
                if ( ground_truth_column_value == desired_value ):

                    # it does - True (or positive or 1!)!
                    ground_truth_value = 1

                else:

                    # it does not - False (or negative or 0!)!
                    ground_truth_value = 0

                #-- END check to see if current value matches desired value. --#

                # add value to list.
                ground_truth_list.append( ground_truth_value )

                # ==> predicted

                # get column value.
                predicted_column_value = getattr( reliability_names_instance, predicted_column_name )

                # does it match desired value?
                if ( predicted_column_value == desired_value ):

                    # it does - True (or positive or 1!)!
                    predicted_value = 1

                else:

                    # it does not - False (or negative or 0!)!
                    predicted_value = 0

                #-- END check to see if current value matches desired value. --#

                # add to predicted list.
                predicted_list.append( predicted_value )

                if ( debug_flag == True ):        
                    print( "----> gt: " + str( ground_truth_column_value ) + " ( " + str( ground_truth_value ) + " ) - p: " + str( predicted_column_value ) + " ( " + str( predicted_value ) + " )" )
                #-- END DEBUG --#

            #-- END loop over Reliability_Names instances. --#
            
        else:
            
            print( "ERROR - you must specify a desired value." )
            
        #-- END check to see if desired value passed in. --#
        
    else:
        
        print( "ERROR - you must provide the suffix of the column you want to examine." )

    #-- END check to see if column name suffix passed in. --#
    
    # package up and return lists.
    lists_OUT[ "ground_truth" ] = ground_truth_list
    lists_OUT[ "predicted" ] = predicted_list
    
    return lists_OUT
    
#-- END function build_confusion_lists() --#

print( "Function build_confusion_lists() defined at " + str( datetime.datetime.now() ) )



    


















    






Function build_confusion_lists() defined at 2017-10-26 04:17:47.522442

Person type - Authors

For each person detected across the set of articles, look at whether the automated coder assigned the correct type.

Person type - Authors - build value lists

First, build lists of ground truth and predicted values per person.





In [15]:



    


confusion_lists = build_confusion_lists( Reliability_Names.FIELD_NAME_SUFFIX_PERSON_TYPE,
                                         Reliability_Names.PERSON_TYPE_AUTHOR )
ground_truth_list = confusion_lists.get( "ground_truth", None )
predicted_list = confusion_lists.get( "predicted", None )

print( "==> population values count: " + str( len( ground_truth_list ) ) )
print( "==> predicted values count: " + str( len( predicted_list ) ) )
print( "==> percentage agreement = " + str( StatsHelper.percentage_agreement( ground_truth_list, predicted_list ) ) )



    


















    






Found 2421 rows with label in ['prelim_month_human']
==> population values count: 2421
==> predicted values count: 2421
==> percentage agreement = 0.997521685254

















In [16]:



    


print( "==> population values: " + str( len( ground_truth_list ) ) )
list_name = "ACTUAL_VALUE_LIST"
string_list = map( str, ground_truth_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )

print( "==> predicted values count: " + str( len( predicted_list ) ) )
list_name = "PREDICTED_VALUE_LIST"
string_list = map( str, predicted_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )



    


















    






==> population values: 2421
ACTUAL_VALUE_LIST = [ 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
==> predicted values count: 2421
PREDICTED_VALUE_LIST = [ 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]

Person type - Authors - confusion matrix





In [17]:



    


confusion_helper = ConfusionMatrixHelper.populate_confusion_matrix( ground_truth_list, predicted_list )
print( str( confusion_helper ) )



    


















    






ConfusionMatrixHelper --> 
DictHelper --> 
----> ACC : 0.9975216852540273
----> BM : 0.9885735450818305
----> DOR : 177757.2
----> FDR : 0.0022026431718061676
----> FNR : 0.010917030567685589
----> FOR : 0.002541942043721403
----> FPR : 0.0005094243504839531
----> LR+ : 1941.5698689956332
----> LR- : 0.010922594803448935
----> MCC : 0.9919088535018595
----> MK : 0.9952554147844723
----> NPV : 0.9974580579562786
----> PPV : 0.9977973568281938
----> SPC : 0.999490575649516
----> TNR : 0.999490575649516
----> TPR : 0.9890829694323144
----> accuracy : 0.9975216852540273
----> diagnostic_odds_ratio : 177757.2
----> f1_score : 0.9934210526315789
----> false_discovery_rate : 0.0022026431718061676
----> false_negative : 5
----> false_negative_rate : 0.010917030567685589
----> false_omission_rate : 0.002541942043721403
----> false_positive : 1
----> false_positive_rate : 0.0005094243504839531
----> informedness : 0.9885735450818305
----> markedness : 0.9952554147844723
----> matthews_correlation_coefficient : 0.9919088535018595
----> negative_likelihood_ratio : 0.010922594803448935
----> negative_predictive_value : 0.9974580579562786
----> population_negative : 1963
----> population_positive : 458
----> positive_likelihood_ratio : 1941.5698689956332
----> precision : 0.9977973568281938
----> predicted_negative : 1967
----> predicted_positive : 454
----> recall : 0.9890829694323144
----> specificity : 0.999490575649516
----> total_population : 2421
----> true_negative : 1962
----> true_negative_rate : 0.999490575649516
----> true_positive : 453

Person type - Subjects

For each person detected across the set of articles classified by Ground truth as a subject, look at whether the automated coder assigned the correct person type.

Person type - Subjects - build value lists

First, build lists of ground truth and predicted values per person.





In [18]:



    


# subjects = "mentioned"
confusion_lists = build_confusion_lists( Reliability_Names.FIELD_NAME_SUFFIX_PERSON_TYPE,
                                         Reliability_Names.SUBJECT_TYPE_MENTIONED )
ground_truth_list = confusion_lists.get( "ground_truth", None )
predicted_list = confusion_lists.get( "predicted", None )

print( "==> population values count: " + str( len( ground_truth_list ) ) )
print( "==> predicted values count: " + str( len( predicted_list ) ) )
print( "==> percentage agreement = " + str( StatsHelper.percentage_agreement( ground_truth_list, predicted_list ) ) )



    


















    






Found 2421 rows with label in ['prelim_month_human']
==> population values count: 2421
==> predicted values count: 2421
==> percentage agreement = 0.956216439488

















In [19]:



    


print( "==> population values: " + str( len( ground_truth_list ) ) )
list_name = "ACTUAL_VALUE_LIST"
string_list = map( str, ground_truth_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )

print( "==> predicted values count: " + str( len( predicted_list ) ) )
list_name = "PREDICTED_VALUE_LIST"
string_list = map( str, predicted_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )



    


















    






==> population values: 2421
ACTUAL_VALUE_LIST = [ 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0 ]
==> predicted values count: 2421
PREDICTED_VALUE_LIST = [ 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0 ]

Person type - Subjects - confusion matrix





In [20]:



    


confusion_helper = ConfusionMatrixHelper.populate_confusion_matrix( ground_truth_list, predicted_list )
print( str( confusion_helper ) )



    


















    






ConfusionMatrixHelper --> 
DictHelper --> 
----> ACC : 0.956216439487815
----> BM : 0.8731866184612787
----> DOR : 547.8526422764227
----> FDR : 0.03582089552238806
----> FNR : 0.11263736263736264
----> FOR : 0.04683038263849229
----> FPR : 0.014176018901358535
----> LR+ : 62.59603937728937
----> LR- : 0.11425707306474234
----> MCC : 0.8949953231008959
----> MK : 0.9173487218391196
----> NPV : 0.9531696173615077
----> PPV : 0.9641791044776119
----> SPC : 0.9858239810986414
----> TNR : 0.9858239810986414
----> TPR : 0.8873626373626373
----> accuracy : 0.956216439487815
----> diagnostic_odds_ratio : 547.8526422764227
----> f1_score : 0.9241773962804005
----> false_discovery_rate : 0.03582089552238806
----> false_negative : 82
----> false_negative_rate : 0.11263736263736264
----> false_omission_rate : 0.04683038263849229
----> false_positive : 24
----> false_positive_rate : 0.014176018901358535
----> informedness : 0.8731866184612787
----> markedness : 0.9173487218391196
----> matthews_correlation_coefficient : 0.8949953231008959
----> negative_likelihood_ratio : 0.11425707306474234
----> negative_predictive_value : 0.9531696173615077
----> population_negative : 1693
----> population_positive : 728
----> positive_likelihood_ratio : 62.59603937728937
----> precision : 0.9641791044776119
----> predicted_negative : 1751
----> predicted_positive : 670
----> recall : 0.8873626373626373
----> specificity : 0.9858239810986414
----> total_population : 2421
----> true_negative : 1669
----> true_negative_rate : 0.9858239810986414
----> true_positive : 646

Person type - Sources

For each person detected across the set of articles classified by Ground truth as a source, look at whether the automated coder assigned the correct person type.

Person type - Sources - build value lists

First, build lists of ground truth and predicted values per person.





In [21]:



    


# subjects = "mentioned"
confusion_lists = build_confusion_lists( Reliability_Names.FIELD_NAME_SUFFIX_PERSON_TYPE,
                                         Reliability_Names.SUBJECT_TYPE_QUOTED )
ground_truth_list = confusion_lists.get( "ground_truth", None )
predicted_list = confusion_lists.get( "predicted", None )

print( "==> population values count: " + str( len( ground_truth_list ) ) )
print( "==> predicted values count: " + str( len( predicted_list ) ) )
print( "==> percentage agreement = " + str( StatsHelper.percentage_agreement( ground_truth_list, predicted_list ) ) )



    


















    






Found 2421 rows with label in ['prelim_month_human']
==> population values count: 2421
==> predicted values count: 2421
==> percentage agreement = 0.981825691863

















In [22]:



    


print( "==> population values: " + str( len( ground_truth_list ) ) )
list_name = "ACTUAL_VALUE_LIST"
string_list = map( str, ground_truth_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )

print( "==> predicted values count: " + str( len( predicted_list ) ) )
list_name = "PREDICTED_VALUE_LIST"
string_list = map( str, predicted_list )
list_values = ", ".join( string_list )
print( list_name + " = [ " + list_values + " ]" )



    


















    






==> population values: 2421
ACTUAL_VALUE_LIST = [ 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1 ]
==> predicted values count: 2421
PREDICTED_VALUE_LIST = [ 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1 ]

Person type - Sources - confusion matrix





In [23]:



    


confusion_helper = ConfusionMatrixHelper.populate_confusion_matrix( ground_truth_list,
                                                                    predicted_list,
                                                                    calc_type_IN = ConfusionMatrixHelper.CALC_TYPE_PANDAS_ML )
print( str( confusion_helper ) )



    


















    






ConfusionMatrixHelper --> 
DictHelper --> 
----> ACC : 0.981825691863
----> BM : 0.963695675912
----> DOR : 3152.97321429
----> FDR : 0.0132890365449
----> FNR : 0.0230263157895
----> FOR : 0.0230073952342
----> FPR : 0.0132780082988
----> LR+ : 73.5783305921
----> LR- : 0.0233361736975
----> MCC : 0.963699622058
----> MK : 0.963703568221
----> NPV : 0.976992604766
----> PPV : 0.986710963455
----> SPC : 0.986721991701
----> TNR : 0.986721991701
----> TPR : 0.976973684211
----> accuracy : 0.981825691863
----> diagnostic_odds_ratio : 3152.97321429
----> f1_score : 0.981818181818
----> false_discovery_rate : 0.0132890365449
----> false_negative : 28
----> false_negative_rate : 0.0230263157895
----> false_omission_rate : 0.0230073952342
----> false_positive : 16
----> false_positive_rate : 0.0132780082988
----> informedness : 0.963695675912
----> markedness : 0.963703568221
----> matthews_correlation_coefficient : 0.963699622058
----> negative_likelihood_ratio : 0.0233361736975
----> negative_predictive_value : 0.976992604766
----> population_negative : 1205
----> population_positive : 1216
----> positive_likelihood_ratio : 73.5783305921
----> precision : 0.986710963455
----> predicted_negative : 1217
----> predicted_positive : 1204
----> recall : 0.976973684211
----> specificity : 0.986721991701
----> total_population : 2421
----> true_negative : 1189
----> true_negative_rate : 0.986721991701
----> true_positive : 1188

TODO

todo

todo:

  • nothing to see here.

Content source: jonathanmorgan/msu_phd_work

Similar notebooks:

notebook.community | gallery | about