prelim_month - reliability

  • original title: 2017.10.25 - work log - prelim_month - Reliability_Names reliability
  • original file name: 2017.10.25-work_log-prelim_month-Reliability_Names_reliability.ipynb

Run the reliability calculations for prelim_month just to get lookup assessment (since it is not classification, precision and recall make no sense).

Setup

Setup - Imports


In [1]:
import datetime
import six

print( "packages imported at " + str( datetime.datetime.now() ) )


packages imported at 2017-10-26 00:26:52.608830

Setup - virtualenv jupyter kernel

If you are using a virtualenv, make sure that you:

  • have installed your virtualenv as a kernel.
  • choose the kernel for your virtualenv as the kernel for your notebook (Kernel --> Change kernel).

Since I use a virtualenv, need to get that activated somehow inside this notebook. One option is to run ../dev/wsgi.py in this notebook, to configure the python environment manually as if you had activated the sourcenet virtualenv. To do this, you'd make a code cell that contains:

%run ../dev/wsgi.py

This is sketchy, however, because of the changes it makes to your Python environment within the context of whatever your current kernel is. I'd worry about collisions with the actual Python 3 kernel. Better, one can install their virtualenv as a separate kernel. Steps:

  • activate your virtualenv:

      workon sourcenet
  • in your virtualenv, install the package ipykernel.

      pip install ipykernel
  • use the ipykernel python program to install the current environment as a kernel:

      python -m ipykernel install --user --name <env_name> --display-name "<display_name>"
    
    

    sourcenet example:

      python -m ipykernel install --user --name sourcenet --display-name "sourcenet (Python 3)"

More details: http://ipython.readthedocs.io/en/stable/install/kernel_install.html


In [2]:
%pwd


Out[2]:
'/home/jonathanmorgan/work/django/research/work/phd_work'

Setup - Initialize Django

First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.


In [3]:
%run ../django_init.py


django initialized at 2017-10-26 04:27:00.745968

Setup R

To allow Python to talk to R, at R prompt:

/* install packages */
install.packages( "Rserve" )
install.packages( "irr" )

/* load Rserve */
library( Rserve )

/* start server */
Rserve( args="--no-save" )

Setup database

Also need to either pass database connection information to names analyzer below, or store database configuration in Django_Config:

# database connection information - 2 options...  Enter it here:
#my_analysis_instance.db_username = ""
#my_analysis_instance.db_password = ""
#my_analysis_instance.db_host = "localhost"
#my_analysis_instance.db_name = "sourcenet"

# Or set up the following properties in Django_Config, inside the django admins.
#     All have application of: "sourcenet-db-admin":
#     - db_username
#     - db_password
#     - db_host
#     - db_port
#     - db_name

Reliability data assessment - prelim_month

Generate reliability analysis for label "prelim_month".


In [8]:
# start to support python 3:
from __future__ import unicode_literals
from __future__ import division

#==============================================================================#
# ! imports
#==============================================================================#

# grouped by functional area, then alphabetical order by package, then
#     alphabetical order by name of thing being imported.

# context_analysis imports
from context_analysis.reliability.reliability_names_analyzer import ReliabilityNamesAnalyzer

#==============================================================================#
# ! logic
#==============================================================================#

# declare variables
my_analysis_instance = None
label = ""
indices_to_process = -1
result_status = ""

# make reliability instance
my_analysis_instance = ReliabilityNamesAnalyzer()

# database connection information - 2 options...  Enter it here:
#my_analysis_instance.db_username = ""
#my_analysis_instance.db_password = ""
#my_analysis_instance.db_host = "localhost"
#my_analysis_instance.db_name = "sourcenet"

# Or set up the following properties in Django_Config, inside the django admins.
#     All have application of: "sourcenet-db-admin":
#     - db_username
#     - db_password
#     - db_host
#     - db_port
#     - db_name

# run the analyze method, see what happens.
#label = "prelim_reliability_test"
#indices_to_process = 3
#label = "prelim_reliability_combined_human"
#indices_to_process = 3
#label = "name_data_test_combined_human"
#indices_to_process = 3
#label = "prelim_reliability_combined_human_final"
#indices_to_process = 3
#label = "prelim_reliability_combined_all"
#indices_to_process = 4
#label = "prelim_reliability_combined_all_final"
#indices_to_process = 4
#label = "prelim_reliability_test_human"
#indices_to_process = 3
#label = "prelim_reliability_test_all"
#indices_to_process = 4
label = "prelim_month"
indices_to_process = 2
result_status = my_analysis_instance.analyze_reliability_names( label, indices_to_process )


==> current index = 1
====> comparison index = 2
====> current_column_name_suffix = _detected
====> current_column_result_name = detect
======> compare_column_name_1 = coder1_detected
======> compare_column_name_2 = coder2_detected
========> percentage_agreement = 1.0
========> R irr::kripp.alpha = 1.0
========> Potter's Pi = 1.0
====> current_column_name_suffix = _person_id
====> current_column_result_name = lookup
======> compare_column_name_1 = coder1_person_id
======> compare_column_name_2 = coder2_person_id
========> percentage_agreement = 1.0
========> R irr::kripp.alpha = 1.0
====> current_column_name_suffix = _person_type_int
====> current_column_result_name = type
======> compare_column_name_1 = coder1_person_type_int
======> compare_column_name_2 = coder2_person_type_int
========> percentage_agreement = 0.995614035088
========> R irr::kripp.alpha = -0.001098901098901095
========> Potter's Pi = 0.994152046784
====> current_column_name_suffix = _first_quote_graf
====> current_column_result_name = first_quote_graf
======> compare_column_name_1 = coder1_first_quote_graf
======> compare_column_name_2 = coder2_first_quote_graf
========> percentage_agreement = 0.0
========> R irr::kripp.alpha = 1.0
====> current_column_name_suffix = _first_quote_index
====> current_column_result_name = first_quote_index
======> compare_column_name_1 = coder1_first_quote_index
======> compare_column_name_2 = coder2_first_quote_index
========> percentage_agreement = 0.0
========> R irr::kripp.alpha = 1.0
====> current_column_name_suffix = _organization_hash
/home/jonathanmorgan/work/django/research/context_analysis/reliability/reliability_names_analyzer.py:442: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  compare_values_1[ compare_values_1.isnull() ] = "-1"
/home/jonathanmorgan/.virtualenvs/sourcenet/lib/python3.5/site-packages/pandas/core/generic.py:5233: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._update_inplace(new_data)
/home/jonathanmorgan/work/django/research/context_analysis/reliability/reliability_names_analyzer.py:698: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  column_name_prefix )
/home/jonathanmorgan/work/django/research/context_analysis/reliability/reliability_names_analyzer.py:444: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  compare_values_2[ compare_values_2.isnull() ] = "-1"
====> current_column_result_name = organization_hash
======> compare_column_name_1 = coder1_organization_hash
======> compare_column_name_2 = coder2_organization_hash
========> percentage_agreement = 0.991228070175
========> R irr::kripp.alpha = 0.7471025053785828
====> current_column_name_suffix = _person_id
====> current_column_result_name = lookup_non_zero
======> compare_column_name_1 = coder1_person_id
======> compare_column_name_2 = coder2_person_id
========> percentage_agreement = 1.0
========> R irr::kripp.alpha = 1.0
====> current_column_name_suffix = _person_type_int
====> current_column_result_name = type_non_zero
======> compare_column_name_1 = coder1_person_type_int
======> compare_column_name_2 = coder2_person_type_int
========> percentage_agreement = 0.995614035088
========> R irr::kripp.alpha = -0.001098901098901095
========> Potter's Pi = 0.993421052632
====> current_column_name_suffix = _detected
====> current_column_result_name = detect
======> compare_column_name_1 = coder1_detected
======> compare_column_name_2 = coder2_detected
========> percentage_agreement = 0.934170854271
========> R irr::kripp.alpha = -0.033775006495193605
========> Potter's Pi = 0.868341708543
====> current_column_name_suffix = _person_id
====> current_column_result_name = lookup
======> compare_column_name_1 = coder1_person_id
======> compare_column_name_2 = coder2_person_id
========> percentage_agreement = 0.927135678392
========> R irr::kripp.alpha = 0.9270088090918948
====> current_column_name_suffix = _person_type_int
====> current_column_result_name = type
======> compare_column_name_1 = coder1_person_type_int
======> compare_column_name_2 = coder2_person_type_int
========> percentage_agreement = 0.859798994975
========> R irr::kripp.alpha = 0.7240822488367566
========> Potter's Pi = 0.813065326633
====> current_column_name_suffix = _first_quote_graf
====> current_column_result_name = first_quote_graf
======> compare_column_name_1 = coder1_first_quote_graf
======> compare_column_name_2 = coder2_first_quote_graf
========> percentage_agreement = 0.343718592965
========> R irr::kripp.alpha = 0.6123922212309788
====> current_column_name_suffix = _first_quote_index
====> current_column_result_name = first_quote_index
======> compare_column_name_1 = coder1_first_quote_index
======> compare_column_name_2 = coder2_first_quote_index
========> percentage_agreement = 0.336683417085
========> R irr::kripp.alpha = 0.62067395381595
====> current_column_name_suffix = _organization_hash
====> current_column_result_name = organization_hash
======> compare_column_name_1 = coder1_organization_hash
======> compare_column_name_2 = coder2_organization_hash
========> percentage_agreement = 0.241206030151
========> R irr::kripp.alpha = -0.23496576773528122
====> current_column_name_suffix = _person_id
====> current_column_result_name = lookup_non_zero
======> compare_column_name_1 = coder1_person_id
======> compare_column_name_2 = coder2_person_id
========> percentage_agreement = 0.992469069392
========> R irr::kripp.alpha = 0.9924634556239207
====> current_column_name_suffix = _person_type_int
====> current_column_result_name = type_non_zero
======> compare_column_name_1 = coder1_person_type_int
======> compare_column_name_2 = coder2_person_type_int
========> percentage_agreement = 0.920387305003
========> R irr::kripp.alpha = 0.8309561562004499
========> Potter's Pi = 0.880580957504
==> current index = 2

In [9]:
print( "result status: {status_string}".format( status_string = result_status ) )


result status: 
  • results are in Dropbox/academia/MSU/program_stuff/prelim_paper/analysis/reliability/2016-data/prelim_month-reliability_results.pdf.