2016.12.11 - work log - prelim_month - remove single names

Table of Contents

Setup

• Back to Table of Contents

Setup - Imports

• Back to Table of Contents

Setup - Initialize Django

• Back to Table of Contents

First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.

You need to have installed your virtualenv with django as a kernel, then select that kernel for this notebook.

Data cleanup

• Back to Table of Contents

Remove single-name reliability data

• Back to Table of Contents

Next, remove all reliability data that refers to a single name using the "View reliability name information" screen:

• http://research.local/research/context/analysis/reliability/names/disagreement/view

To start, enter the following in fields there:

• Label: "prelim_month"
• Coders to compare (1 through ==>): 2
• Reliability names filter type: Select "Lookup"
• [Lookup] - Person has first name, no other name parts.: CHECK the checkbox

You should see lots of entries where coders detected people who were mentioned only by their first name.

Single-name data assessment

• Back to Table of Contents

Need to look at each instance where a person has a single name part.

Most are probably instances where the computer correctly detected the name part, but where you don't have enough name to match it to a person so the human coding protocol directed them to not capture the name fragment.

However, there might be some where a coder made a mistake and just captured a name part for a person whose full name was in the story. To check, click the "Article ID" in the column that has a link to article ID. It will take you to a view of the article where all the people who coded the article are included, with each detection of a mention or quotation displayed next to the paragraph where the person was originally first detected.

So for each instance of a single name part:

• click on the article ID link in the row to go to the article and check to see if there is person whose name the fragment is a part of ( http://research.local/research/context/text/article/article_data/view_with_text/ ).

  • If there is a person with a full name to which the name fragment is a reference, check to see if the human coder has data for the full person.

    • if human coder has data for the full person, merge:

      • go to the disagreement view page: http://research.local/research/context/analysis/reliability/names/disagreement/view

      • Configure:

        • Label: - "prelim_month"
        • Coders to compare (1 through ==>): - 2
        • Reliability names filter type: - Select "Lookup"
        • [Lookup] - Associated Article IDs (comma-delimited): - Enter the ID of the article the coding belonged to.

      • this will bring up all coding for the article whose ID you entered.

      • In the "select" column, click the checkbox in the row where there is a single name part that needs to be merged.
      • In the "merge INTO" column, click the checbox in the row with the full name for that person.
      • In "Reliability Names Action", choose "Merge Coding --> FROM 1 SELECTED / INTO 1"
      • Click "Do Action" button.

    • if human coder did not detect person:

      • create a copy of the person's Article_Data.
      • assign it to coder "ground_truth".
      • add person to the new record.
      • save

      • regenerate Reliability_Names for just that article.

        • (?) Remove old Reliability_Names for that article.
        • re run names creation for the article.
        • TK

      • merge the two Reliability_Names records for the person.

  • Remove the Reliability_Names row with the name fragment from reliability data.

Delete single-name data

• Back to Table of Contents

To get rid of all matching in this list, click the checkbox in the "select" column next to each one you want to delete (sorry, no "select all" just yet), choose "Delete selected" from the "Reliability names action:" field at the top of the list, then click the "Do action" button.

Reliability_Names records Removed:

Reliability_Names records merged:

Notes

• Back to Table of Contents

Notes:

• what to do about a misspelled name within an article? Single name - removing all. But making note:

  • In article 21080, Reliability_Names 9583, name = Culter, should have been Cutler - quoted, graf: 13, index: 1322

    • single name, remove it - but, this will cut both ways - when both name parts present, sometimes will work out, sometimes will be false positive.

• What to do about single last name that is the correct last name of a person where the other name parts were detected by a person? Leave it in and map it to the correct Article_Data?

• Obama? One name, but it is a well-known one, and preceded by "President". Still, single name, removed it.

Errors:

• Article 21116

  • Paragraph 12: More than 600 works of art were added to the museum's collection under her leadership, most notably Ellsworth Kelly's "Blue White," a 25-foot- tall wall sculpture that was commissioned in 2006 for the museum's entry pavilion.
  • User: 2 - automated (OpenCalais_REST_API_v2)
  • 11288 (AS) - More ( id = 2817; capture_method = OpenCalais_REST_API_v2 ) (mentioned; individual) ==> name: More

TODO

• Back to Table of Contents

TODO:

• TK

Coding to look into

• Back to Table of Contents

Coding decisions to look at more closely:

Debugging

• Back to Table of Contents

Debuggin: