corpkit: a Python-based toolkit for working with parsed linguistic corpora

Daniel McDonald**

SUMMARY: This IPython Notebook shows you how to use corpkit to investigate a corpus of paragraphs containing the word risk in the NYT between 1963 and 2014.

Orientation

First, let's import the functions we'll be using to investigate the corpus. These functions are designed for this interrogation, but also have more general use in mind, so you can likely use them on your own corpora.

Next, let's set the path to our corpus. If you were using this interface for your own corpora, you would change this to the path to your data.

The data

Our main corpus is comprised of paragraphs from New York Times articles that contain a risk word, which we have defined by regular expression as '(?i)'.?\brisk.?\b'. This includes low-risk, or risk/reward as single tokens, but excludes brisk or asterisk.

The data comes from a number of sources.

• 1963 editions were downloaded from ProQuest Newsstand as PDFs. Optical character recognition and manual processing was used to create a set of 1200 risk sentences.
• The 1987--2006 editions were taken from the NYT Annotated Corpus.
• 2007--2014 editions were downloaded from ProQuest Newsstand as HTML.

In total, 149,504 documents were processed. The corpus from which the risk corpus was made is over 150 million words in length!

The texts have been parsed for part of speech and grammatical structure by `Stanford CoreNLP*. In this Notebook, we are only working with the parsed versions of the texts. We rely on Tregex to interrogate the corpora. Tregex allows very complex searching of parsed trees, in combination with Java Regular Expressions. It's definitely worthwhile to learn the Tregex syntax, but in case you're time-poor, at the end of this notebook are a series of Tregex queries that you can copy and paste into *interrogator()andconc()` queries.

Interrogating the corpus

So, let's start by generating some general information about this corpus. First, let's define a query to find every word in the corpus. Run the cell below to define the allwords_query variable as the Tregex query to its right.

When writing Tregex queries or Regular Expressions, remember to always use r'...' quotes!

Next, we perform interrogations with interrogator(). Its most important arguments are:

1. path to corpus

2. Tregex options:

   • '-t': return only words
   • '-C': return a count of matches
   • '-u': return only the tag
   • '-o': return tag and word together

3. the Tregex query

We only need to count tokens, so we can use the -C option (it's often faster than getting lists of matching tokens). The cell below will run interrogator() over each annual subcorpus and count the number of matches for the query.

When the interrogation has finished, we can view our results:

If you want to see the query and options that created the results, you can use:

Plotting results

Lists of years and totals are pretty dry. Luckily, we can use the plotter() function to visualise our results. At minimum, plotter() needs two arguments:

1. a title (in quotation marks)
2. a list of results to plot

There is also an argument for projecting the 1963 and 2014 results, which can either be set to true or false. By default, it is true, and in this Notebook from now on we'll leave it turned on. We'll try both options here:

Great! So, we can see that the number of words per year varies quite a lot. That's worth keeping in mind.

Frequency of risk words in the NYT

Next, let's count the total number of risk words. Notice that we are using the '-o' flag, instead of the -C flag.

Even when do not use the -C flag, we can access the total number of matches as before:

At the moment, it's hard to tell whether or not these counts are simply because our annual NYT samples are different sizes. To account for this, we can calculate the percentage of parsed words that are risk words. This means combining the two interrogations we have already performed.

We can do this by passing a third argument to plotter().

That's more helpful. We can now see some interesting peaks and troughs in the proportion of risk words. We can also see that 1963 contains the highest proportion of risk words. This is because the manual corrector of 1963 OCR entries preserved only the sentence containing risk words, rather than the paragraph.

It's often helpful to not plot 1963 results for this reason. To do this, we can add an argument to the plotter() call:

Perhaps we're interested in not only the frequency of risk words, but the frequency of different kinds* of risk words. We actually already collected this data during our last *interrogator() query.

We can print just the first three entries of the results list, rather than the totals list:

We now have enough data to do some serious plotting.

Customising visualisations

By default, plotter() plots the seven most frequent results, including 1963 and projecting 1963 and 2014.

We can use other plotter() arguments to customise what our chart shows. plotter()'s possible arguments are:

You can easily use these to get different kinds of output. Try changing some parameters below:

If you just generated a csv file, you can quickly get the results with:

Another way to change plotter() visualisations is by not passing certain results to plotter().

Each entry in the list of results is indexed: the top result is item 0, the second result is item 1, and so on.

So, you can skip the first 2 results by using [2:] after the results list:

If you are after a specific set of indexed items, it's probably better to use surgeon() (described below). For completeness, though, here's another way:

Another neat thing you can do is save the results of an interrogation, so they don't have to be run the next time you load this notebook:

You can then load these results:

table()

If you want to quickly table the results of a csv file, you can use table(). Its only main argument is the path to the csv file as string. There are two optional arguments. First, you can set allresults to True to table all results, rather than just the plotted results. When this option is set to true, you may get way too many results. To cope with this, there is a maxresults argument, whose value by default is 50. You can overwrite this default to table more or fewer results.

quickview()

quickview() is a function that quickly shows the n most frequent items in a list. Its arguments are:

1. an interrogator() result
2. number of results to show (default = 50)

The number shown next to the item is its index. You can use this number to refer to an entry when editing results.

tally()

tally() simply displays the total occurrences of results. Its first argument is the list you want tallies from. For its second argument, you can use:

• a list of indices for results you want to tally
• a single integer, which will be interpreted as the index of the item you want
• a regular expression to search for
• a string, 'all', which will tally every result. This could be very many results, so it may be worth limiting the number of items you pass to it with [:n], as in the second example below:

The Regular Expression option is useful for merging results (see below).

surgeon()

Results lists can be edited quickly with surgeon(). surgeon()'s arguments are:

1. an interrogator() results list
2. criteria: either a Regular Expression or a list of indices.
3. remove = True/False

By default, surgeon() keeps anything matching the regex, but this can be inverted with a remove = True argument. Because you are duplicating the original list, you don't have to worry about deleting interrogator() results.

Note the warning you'll receive if you specify an interrogation, rather than a results list.

merger()

merger() is for merging items in a list. Like surgeon(), it duplicates the old list. Its arguments are:

1. the list you want to modify
2. the indices of results you want to merge, or a regex to match
3. newname = str/int/False:
   • if string, the string becomes the merged item name.
   • if integer, the merged entry takes the name of the item indexed with the integer.
   • if not specified/False, the most most frequent item in the list becomes the name.

Diversity of risk words

It's important to note that the kind of results we generate are hackable. Using some straight Python, combined with merger(), we can figure out how unique risk words appear in the NYT each year.

To do this, we can take riskwords.results, duplicate it, and change every count over 0 into 1.

We can then use merger() to merge every entry. This will tell use how many unique words there are each year.

So, we can see a generally upward trajectory, with more risk words constantly being used. Many of these results appear once, however, and many are nonwords. Can you figure out how to remove words that appear only once per year?

conc()

conc() produces concordances of a subcorpus based on a Tregex query. Its main arguments are:

1. A subcorpus to search (remember to put it in quotation marks!)
2. A Tregex query

You can set conc() to print n random concordances with the random = n parameter. You can also store the output to a variable for further searching.

conc() takes another argument, window, which alters the amount of co-text appearing either side of the match. The default is 50 characters

conc() also allows you to view parse trees. By default, it's false:

The final conc() argument is a csv = 'filename', which will produce a tab-separated spreadsheet with the results of your query. You can copy and paste this data into Excel.

Keywords, ngrams and collocates

There are also functions for keywording, ngramming and collocation. Each can take a number of kinds of input data:

1. a path to a subcorpus (of either parse trees or raw text)
2. a path to a csv file generated with conc()
3. a string of text
4. a list of strings (i.e. output from conc())

keywords() produces both keywords and ngrams. It relies on code from the Spindle project.

You can even use interrogator() to get keywords or ngrams from each subcorpus. To do this, use 'keywords' or 'ngrams' as the query argument.

With the collocates() function, you can specify the maximum distance at which two tokens will be considered collocates. The default is 5.

quicktree() and searchtree()

The two functions are useful for visualising and searching individual syntax trees. They might be useful if you want to practive your Tregex queries, or make sure you are getting the expected result.

The easiest place to get a parse tree is from a CSV file generated using conc() with trees set to True. Alternatively, you can open files in the data directory directly.

quicktree() generates a visual representation of a parse tree. Here's one from 1989:

searchtree() requires a tree and a Tregex query. It will return a list of query matches.

Process types

Systemic functional linguistics argues that the main verb (the process) can be one of a few types.

We can use these in our Tregex queries to look for the kinds of processes participant risks are involved in. First, let's get a count for all processes with risk participants:

Proper nouns and risk sentences

We searched to find the most common proper noun strings.

interrogator()'s titlefilter option removes common titles, first names and determiners to make for more accurate counts. It is useful when the results being returned are groups/phrases, rather than single words.

Notice that there are a few entries here that refer to the same group. (f.d.a and food and drug administration, for example). We can use merger() to fix these.

Now that we've merged some common results, we can use surgeon() to build some basic thematic categories.

These charts reveal some interesting patterns.

• We can clearly see presidencies and rival candidates come and go
• Similarly, the wars in Iraq and Afghanistan are easy to spot
• Naturally, the Soviet Union is a very frequent topic in 1963. It rises in frequency until its collapse. More recently, Russia can be seen as more frequently co-occurring with risk words.
• The Eurozone crisis is visible
• From the Organisations and Things, we can see the appearance of Merck and Vioxx in 2004, as well as Empire...

Vioxx was removed from shelves following the discovery that it increased the risk of heart attack. It's interesting how even though terrorism and war may come to mind when thinking of risk in the past 15 years, this health topic is easily more prominent in the data.

What now?

From here on out, it's up to you. Currently in development are resources for making parsed corpora, but really, that process only involves:

1. Getting some plain text
2. Parsing it with Stanford CoreNLP
3. Putting the files in sequential subfolders

Wrap up