SUMMARY: This IPython Notebook demonstrates the findings from our investigation of risk in the NYT, as well as the code used to generate these findings. If you have the necessary dependencies installed, you can also use this notebook to interrogate and visualise the corpus yourself.
If you haven't already done so, the first things we need to do are install corpkit, download data for NLTK's tokeniser, and unzip our corpus.
In [ ]:
# install corpkit with either pip or easy_install
! easy_install -u corpkit
In [ ]:
# download nltk tokeniser data
import nltk
nltk.download('punkt')
nltk.download('wordnet')
In [ ]:
# unzip and untar our data
! gzip -dc data/nyt.tar.gz | tar -xf - -C data
Great! Now we have everything we need to start.
Let's import the functions we'll be using to investigate the corpus.
| Function name | Purpose | |
|---|---|---|
interrogator() |
interrogate parsed corpora | |
editor() |
edit interrogator() results |
|
plotter() |
visualise interrogator() results |
|
quickview() |
view interrogator() results |
|
conc() |
complex concordancing of subcorpora | |
keywords() |
get keywords and ngrams from conc() output |
In [1]:
import pandas as pd
from IPython.display import clear_output
import mpld3
import corpkit
from corpkit import (interrogator, editor, plotter, quickview, conc,
save_result, load_result, load_all_results)
# show figures in browser
% matplotlib inline
Next, let's set the path to our corpus. If you were using this interface for your own corpora, you would change this to the path to your data.
In [2]:
# corpus of every article, with annual subcorpora
annual_trees = 'data/nyt/years'
Let's also quickly set some options for displaying raw data:
In [3]:
pd.set_option('display.max_rows', 10)
pd.set_option('display.max_columns', 8)
pd.set_option('max_colwidth',70)
pd.set_option('display.width', 1000)
pd.set_option('expand_frame_repr', False)
We can also quickly load all of our saved results as a dictionary. It's handy to make the name short:
In [4]:
r = load_all_results('data/saved_interrogations')
Let's start off with some quick examples. By the end of this Notebook, you should be more than capable of reproducing even the most complex examples!
In [19]:
# interrogate
modals = interrogator(annual_trees, 'words', 'MD < __')
# save
save_result(modals, 'modals')
# view
quickview(modals)
In [5]:
# if using loaded results:
modals = r['modals']
# simple stuff: make relative frequencies for individual or total results
rel_modals = editor(modals.results, '%', modals.totals)
# trickier: make an 'others' result from low-total entries
low_indices = [i for i, w in enumerate(list(modals.results.index)) if i > 6]
each_modal_total = editor(modals.results, '%', modals.totals, merge_entries = low_indices,
newname = 'other', just_totals = True)
# complex stuff: merge results
entries_to_merge = [r'(^w|\'ll|\'d)', r'^c', r'^m[^d]', r'^sh']
for regex in entries_to_merge:
modals = editor(modals.results, merge_entries = regex)
# complex stuff: merge subcorpora
subcorpora_to_merge = [('1960s', r'^196'), ('1980s', r'^198'), ('1990s', r'^199'),
('2000s', r'^200'), ('2010s', r'^201')]
for subcorp, search in subcorpora_to_merge:
modals = editor(modals.results, merge_subcorpora = search, new_subcorpus_name=subcorp)
# make relative, sort, remove what we don't want
modals = editor(modals.results, '%', modals.totals,
just_subcorpora = [n for n, s in subcorpora_to_merge], sort_by = 'total', keep_top = 4)
# clear output and show results
clear_output()
print rel_modals.results, '\n', each_modal_total.results, '\n', modals.results
In [8]:
# line chart
plotter('Common modals in the NYT', rel_modals.results.drop('1963'), y_label = 'Percentage of all modals',
style = 'fivethirtyeight', num_to_plot = 8, legend_pos = 'outside upper right',
show_totals = 'both')
# pie chart
plotter('Pie chart of common modals', each_modal_total.results, explode = ['other'], shadow = True,
num_to_plot = 'all', kind = 'pie', colours = 'Accent', figsize = (8, 8), show_totals = 'plot')
# stacked area chart
plotter('An ocean of modals: \emph{The New York Times}, 1987--2014', rel_modals.results.drop('1963'), kind = 'area',
stacked = True, colours = 'summer', figsize = (8, 10), num_to_plot = 'all',
legend_pos = 'lower right', y_label = 'Percentage of all modals')
# bar chart, transposing and reversing the data
plotter('Modal use by decade', modals.results.iloc[::-1].T.iloc[::-1], kind = 'barh',
x_label = 'Percentage of all modals', y_label = 'Modal group', style = 'bmh')
Some plots may also be viewed interactively: you can simply add interactive = True and get a version that can be scrolled through, zoomed, hovered over, or lit up:
In [6]:
import mpld3
In [10]:
plotter('Common modals in the NYT', rel_modals.results.drop('1963'), y_label = 'Percentage of all modals',
style = 'fivethirtyeight', num_to_plot = 8, legend_pos = 'outside upper right',
show_totals = 'both', interactive=True)
Out[10]:
This uses mpld3, which is still under a great deal of development. As such, its use here is currently poorly documented and largely experimental. Many features, such as legend placement, use of TeX, etc., will not work.
Anyway, as you can see, there's a lot to do. These results all came from one very simple query! So, let's begin.
The focus of this notebook is our methodology and findings during our 2014 research project. These parts of the project are contextualised and elaborated upon in our written report. Depending on your browser's capabilities/settings, the following will download or display our report:
In [ ]:
from corpkit import report_display
report_display()
Our main corpus is comprised of paragraphs from New York Times articles that contain a risk word, which we have defined by regular expression as '(?i)'.?\brisk.?\b'. This includes low-risk, or risk/reward as single tokens, but excludes brisk or asterisk.
The data comes from a number of sources.
In total, 149,504 documents were processed. The corpus from which the risk corpus was made is over 150 million words in length!
The texts have been parsed for part of speech and grammatical structure by `Stanford CoreNLP*. In this Notebook, we are only working with the parsed versions of the texts. We rely on Tregex to interrogate the corpora. Tregex allows very complex searching of parsed trees, in combination with Java Regular Expressions. It's definitely worthwhile to learn the Tregex syntax, but in case you're time-poor, at the end of this notebook are a series of Tregex queries that you can copy and paste into interrogator() and conc() queries.
So, let's start by finding out how many words we have in each subcorpus. To do this, we'll interrogate the corpus using interrogator(). Its most important arguments are:
There are many kinds of search options available:
| Option | Function |
|---|---|
b |
get tag and word of Tregex match |
c |
count Tregex match |
d |
get dependent of regular expression match and the r/ship |
f |
get dependency function of regular expression match |
g |
get governor of regular expression match and the r/ship |
i |
get dependency index of regular expression match |
k |
Find keywords |
n |
find n-grams |
p |
get part-of-speech tag with Tregex |
r |
regular expression, for plaintext corpora |
s |
simple search string or list of strings for plaintext corpora |
w |
get word(s)returned by Tregex/keywords/ngrams |
Right now, we only need to count tokens, so we can use the c option. The cell below will run interrogator() over each subcorpus and count the number of matches for the query.
In [4]:
allwords = interrogator(annual_trees, 'count', 'any')
When the interrogation has finished, we can view our results:
In [55]:
# from the allwords results, print the totals
allwords.totals
Out[55]:
If you want to see the query and options that created the results, you can use:
In [60]:
allwords.query
Out[60]:
You could even run the same query again by passing this dictionary back into interrogator():
In [ ]:
# again = interrogator(**allwords.query)
Lists of years and totals are pretty dry. Luckily, we can use the plotter() function to visualise our results. At minimum, plotter() needs two arguments:
In [6]:
plotter('Word counts in each subcorpus', allwords.totals)
Because we have smaller samples for 1963 and 2014, we might want to project them. To do that, we can pass subcorpus names and projection values to editor():
In [147]:
proj_vals = [(1963, 5), (2014, 1.37)]
projected = editor(allwords.totals, projection = proj_vals)
plotter('Word counts in each subcorpus (projected)', projected.totals, style = 'bmh')
Great! So, we can see that the number of words per year varies quite a lot, even after projection. That's worth keeping in mind.
Next, let's count the total number of risk words. Notice that we are using the 'both' flag, instead of the 'count' flag, because we want both the word and its tag.
In [10]:
# our query:
riskwords_query = r'__ < /(?i).?\brisk.?\b/' # any risk word and its word class/part of speech
# get all risk words and their tags :
riskwords = interrogator(annual_trees, 'both', riskwords_query)
Even when do not use the count flag, we can access the total number of matches as before:
In [11]:
riskwords.totals
Out[11]:
At the moment, it's hard to tell whether or not these counts are simply because our annual NYT samples are different sizes. To account for this, we can calculate the percentage of parsed words that are risk words. This means combining the two interrogations we have already performed.
We can do this by using editor():
In [149]:
rel_riskwords = editor(riskwords.totals, '%', allwords.totals)
print rel_riskwords.totals
In [9]:
plotter('Relative frequency of risk words', rel_riskwords.totals)
That's more helpful. We can now see some interesting peaks and troughs in the proportion of risk words. We can also see that 1963 contains the highest proportion of risk words. This is because the manual corrector of 1963 OCR entries preserved only the sentence containing risk words, rather than the paragraph.
Here are two methods for excluding 1963 from the chart:
In [152]:
# using Pandas syntax:
plotter('Relative frequency of risk words', rel_riskwords.totals.drop('1963'),
legend = True, style = 'fivethirtyeight')
# the other way: using editor()
#rel_riskwords = editor(rel_riskwords.totals, skip_subcorpora = [1963])
#plotter('Relative frequency of risk words', rel_riskwords.totals)
Perhaps we're interested in not only the frequency of risk words, but the frequency of different kinds of risk words. We actually already collected this data during our last interrogator() query.
We can print just the first few entries of the results list, rather than the totals list.
In [40]:
# using Pandas syntax:
riskwords.results.head(10)
Out[40]:
In [41]:
# using quickview
from corpkit import quickview
quickview(riskwords.results, n = 10)
So, let's use this data to do some more serious plotting:
In [12]:
frac1 = editor(riskwords.results, '%', riskwords.totals)
# alternative syntax:
# frac1 = editor(riskwords.results, '%', 'self')
In [13]:
# colormap is used for > 7 results
plotter('Risk word / all risk words', frac1.results, num_to_plot = 9)
If plotter() can't find a good spot for the legend, you can explicitly move it:
In [14]:
plotter('Risk word / all risk words', frac1.results, num_to_plot = 9, legend_pos = 'lower right', figsize = (8, 4))
plotter('Risk word / all risk words', frac1.results, num_to_plot = 9, legend_pos = 'outside right', figsize = (8, 4)) # 'o r' for short
In [154]:
frac2 = editor(riskwords.results, '%', allwords.totals, sort_by = 'total')
In [16]:
plotter('Risk word / all words', frac2.results, legend_pos = 'outside right')
Another neat feature is the .table attribute of interrogations, which shows the most common n results in each subcorpus:
In [55]:
riskwords.table
Out[55]:
By default, plotter() plots the seven most frequent results, including 1963.
We can use other plotter() arguments to customise what our chart shows. plotter()'s possible arguments are:
plotter() argument |
Mandatory/default? | Use | Type |
|---|---|---|---|
title |
mandatory | A title for your plot | string |
results |
mandatory | the results you want to plot | interrogator() or editor() output |
num_to_plot |
7 | Number of top entries to show | int |
x_label |
False | custom label for the x-axis | str |
y_label |
False | custom label for the y-axis | str |
figsize |
(13, 6) | set the size of the figure | tuple: (length, width) |
tex |
'try' |
use TeX to generate image text | boolean |
style |
'ggplot' |
use Matplotlib styles | str: 'dark_background', 'bmh', 'grayscale', 'ggplot', 'fivethirtyeight' |
legend_pos |
'default' |
legend position | str: 'outside right' to move legend outside chart |
show_totals |
False |
Print totals on legend or plot where possible | str: 'legend', 'plot', 'both', or 'False' |
save |
False |
Save to file | True: save as title.png. str: save as str |
colours |
'Paired' |
plot colours | str: any of Matpltlib's colormaps |
cumulative |
False |
plot entries cumulatively | bool |
**kwargs |
False | pass other options to Pandas plot/Matplotlib | rot = 45, subplots = True, fontsize = 16, etc. |
In [17]:
plotter('Risk words', frac2.results, num_to_plot = 5, y_label = 'Percentage of all words')
Keyword arguments for Pandas and matplotlib can also be used:
In [103]:
plotter('Risk words', frac2.results.drop('1963').T.head(5), subplots = True,
layout = (4,4), figsize = (10, 10), num_to_plot = 16, kind='pie', pie_legend = True, save = True)
In [22]:
plotter('Risk words', frac2.results.drop('1963'), kind = 'bar', stacked = True, legend_pos = 'o r')
In [156]:
plotter('Verbal risk words', editor(frac2.results, just_entries= r'^\(v', print_info = False).results,
kind = 'area', stacked = True, legend_pos = 'o r', colours = 'Oranges', num_to_plot = 'all',
fontsize = 16, tex = False, figsize = (8, 6), style = 'bmh')
In [38]:
letters = [('Adjective', 'j'),
('Noun', 'n'),
('Verb', 'v')]
for label, letter in letters:
riskwords = editor(riskwords.results, merge_entries = r'^\(' + letter, newname = label, print_info = False)
wordclasses = editor(riskwords.results, '%', riskwords.totals, just_entries = [n for n, l in letters], sort_by = 'total', print_info = False)
plotter('Nominalisation of risk: \emph{The New York Times}, 1987--2014', wordclasses.results.drop('1963'),
kind = 'area', stacked = True, figsize = (8, 10), num_to_plot = 'all',
legend_pos = 'lower right', y_label = 'Percentage of all risk words', save = True)
Those already proficient with Python can use Pandas' plot() function (docs) if they like.
Another neat thing you can do is save the results of an interrogation, so they don't have to be run the next time you load this notebook:
In [31]:
# specify what to save, and a name for the file.
from corpkit import save_result, load_result
save_result(allwords, 'allwords')
You can then load these results:
In [32]:
fromfile_allwords = load_result('allwords')
fromfile_allwords.totals
Out[32]:
... or erase them from memory:
In [35]:
fromfile_allwords = None
fromfile_allwords
quickview() is a function that quickly shows the n most frequent items in a list. Its arguments are:
interrogator() or editor() output (preferably, the whole interrogation, not just the .results branch.)
In [39]:
quickview(riskwords, n = 15)
Results lists can be edited quickly with editor(). It has a lot of different options:
editor() argument |
Mandatory/default? | Use | Type |
|---|---|---|---|
df |
mandatory | the results you want to edit | interrogator() or editor output |
operation |
'%' | if using second list, what operation to perform | '+', '-', '/', '*' or '%' |
df2 |
False | Results to comine in some way with df |
interrogator() or editor output (usually, a .totals branch) |
just_subcorpora |
False | Subcorpora to keep | list |
skip_subcorpora |
False | Subcorpora to skip | list |
merge_subcorpora |
False | Subcorpora to merge | list |
new_subcorpus_name |
False | name for merged subcorpora | index/str |
just_entries |
False | Entries to keep | list |
skip_entries |
False | Entries to skip | list |
merge_entries |
False | Entries to merge | list of words or indices/a regex to match |
sort_by |
False | sort results | str: 'total', 'infreq', 'name', 'increase', 'decrease' |
keep_top |
False | Keep only top n results after sorting | int |
just_totals |
False | Collapse all subcorpora, return Series | bool |
projection |
False | project smaller subcorpora | list of tuples: [(subcorpus_name, projection_value)] |
**kwargs |
False | pass options to Pandas' plot() function, Matplotlib |
various |
First, we can select specific subcorpora to keep, remove or span:
Let's try these out on a new interrogation. The query below will get adjectival risk words:
In [42]:
adj = r'/JJ.?/ < /(?i)\brisk/'
adj_riskwords = interrogator(annual_trees, 'words', adj, quicksave = 'adj_riskwords')
In [43]:
editor(adj_riskwords.results, skip_subcorpora = [1963, 1987, 1988]).results
Out[43]:
In [44]:
editor(adj_riskwords.results, just_subcorpora = [1963, 1987, 1988]).results
Out[44]:
In [45]:
editor(adj_riskwords.results, span_subcorpora = [2000, 2010]).results
Out[45]:
We can do similar kinds of things with each result:
In [46]:
quickview(adj_riskwords.results)
In [47]:
editor(adj_riskwords.results, skip_entries = [2, 5, 6]).results
Out[47]:
In [48]:
editor(adj_riskwords.results, just_entries = [2, 5, 6]).results
Out[48]:
We can also use the words themselves, rather than indices, for all of these operations:
In [49]:
editor(adj_riskwords.results, just_entries = ['risky', 'riskier', 'riskiest']).results
Out[49]:
Or, we can use Regular Expressions:
In [50]:
# skip any that start with 'r'
editor(adj_riskwords.results, skip_entries = r'^r').results
Out[50]:
We can also merge entries, and specify a new name for the merged items. In lieu of a name, we can pass an index.
In [51]:
editor(adj_riskwords.results, merge_entries = [2, 5, 6], newname = 'New name').results
Out[51]:
In [52]:
editor(adj_riskwords.results, merge_entries = ['risky', 'riskier', 'riskiest'], newname = 'risky').results
Out[52]:
Notice how the merged result appears as the final column. To reorder the columns by total frequency, we can use sort_by = 'total'.
In [53]:
# if we don't specify a new name, editor makes one for us
generated_name = editor(adj_riskwords.results, merge_entries = ['risky', 'riskier', 'riskiest'], sort_by = 'total')
quickview(generated_name.results)
editor() can sort also sort alphabetically, or by least frequent:
In [54]:
# alphabetically
editor(adj_riskwords.results, sort_by = 'name').results
Out[54]:
In [55]:
# least frequent
editor(adj_riskwords.results, sort_by = 'infreq').results
Out[55]:
Particularly cool is sorting by 'increase' or 'decrease': this calculates the trend lines of each result, and sort by the slope.
In [56]:
editor(adj_riskwords.results, sort_by = 'increase').results
Out[56]:
We can use just_totals to output just the sum of occurrences in each subcorpus:
In [11]:
adj_riskwords = load_result('adj_riskwords')
just_tot = editor(adj_riskwords.results, just_totals = True, keep_top = 20)
just_tot.results
Out[11]:
Any edited result also has a .query branch, which is a dictionary containing everything used to generate the result. This even includes the original data!
In [12]:
print just_tot.query['time_started']
just_tot.query
Out[12]:
You can save and load edited interrogations the same as you would an interrogator() result:
In [40]:
#save_result(just_tot, 'edited_adj_riskwords')
just_tot = None
just_tot = load_result('edited_adj_riskwords')
quickview(just_tot, 5)
A handy thing about working with Pandas DataFrames is that we can easily translate our results to other formats:
In [41]:
deceasing = editor(adj_riskwords.results, sort_by = 'decrease')
In [42]:
# tranpose with T, get just top 5 results, print as CSV
print deceasing.results.T.head().to_csv()
In [43]:
# or, print to latex markup:
print deceasing.results.T.head().to_latex()
Of course, you can perform many of these operations at the same time. Problems may arise, however, especially if your options contradict.
In [44]:
editor(adj_riskwords.results, '%', adj_riskwords.totals, span_subcorpora = [1990, 2000],
just_entries = r'^[^r]', merge_entries = r'er$', newname = 'risk-comparative?', sort_by = 'total').results
Out[44]:
It's important to note that the kind of results we generate are hackable. We could count the number of unique risk words in each subcorpus by changing any count over 1 to 1.
In [45]:
import numpy as np
# copy our list
uniques = riskwords.results.copy()
# divide every result by itself
for f in uniques:
uniques[f] = uniques[f] / uniques[f]
# get rid of inf scores (i.e. 0 / 0) using numpy
uniques = uniques.replace(np.inf, 0)
# sum the results
u = uniques.T.sum()
# give our data a name
u.name = 'Unique risk words'
In [53]:
plotter('Unique risk words', u.drop(['1963', '2014']), y_label = 'Number of unique risk words', legend = True,
num_to_plot = 'all')
So, we can see a generally upward trajectory, with more risk words constantly being used. Many of these results appear once, however, and many are nonwords. Can you figure out how to remove words that appear only once per year?
conc() produces concordances of a subcorpus based on a Tregex query. Its main arguments are:
In [132]:
# here, we use a subcorpus of politics articles,
# rather than the total annual editions.
adj_lines = conc('data/nyt/topics/politics/1999', r'/JJ.?/ << /(?i).?\brisk.?\b/') # adj containing a risk word
You can set conc() to print only the first ten examples with n = 10, or ten random these with the n = 15, random = True parameter.
In [99]:
lines = conc('data/nyt/years/2007', r'/VB.?/ < /(?i).?\brisk.?\b/', n = 15, random = True)
conc() takes another argument, window, which alters the amount of co-text appearing either side of the match. The default is 50 characters
In [100]:
lines = conc('data/nyt/topics/health/2013', r'/VB.?/ << /(?i).?\brisk.?\b/', n = 15, random = True, window = 20)
conc() also allows you to view parse trees. By default, it's false:
In [107]:
lines = conc('data/nyt/years/2013', r'/VB.?/ < /(?i)\btrad.?/', trees = True)
Just like our other data, conc lines can be edited with editor(), or outputted as CSV.
In [108]:
lines = editor(lines, skip_entries = [1, 2, 4, 5])
print lines
If the concordance lines aren't print well, you can use concprinter():
In [109]:
from corpkit import concprinter
concprinter(lines)
Or, you can just use Pandas syntax:
In [110]:
# Because there may be commas in the concordance lines,
# it's better to generate a tab-separated CSV:
print lines.to_csv(sep = '\t')
You can also print some TeX, if you're that way inclined:
In [111]:
print lines.to_latex()
corpkit has some functions for keywording, ngramming and collocation. Each can take a number of kinds of input data:
conc() outputkeywords() produces both keywords and ngrams. It relies on code from the Spindle project.
In [133]:
from corpkit import keywords
keys, ngrams = keywords(adj_lines, dictionary = 'bnc.p')
for key in keys[:10]:
print key
You can also use interrogator() to search for keywords or ngrams. To do this, instead of a Tregex query, pass 'keywords' or 'ngrams'. You should also specify a dictionary to use as the reference corpus. If you specify dictionary = 'self', a dictionary will be made of the entire corpus, saved, and used.
In [ ]:
kwds_bnc = interrogator(annual_trees, 'words', 'keywords', dictionary = 'bnc.p')
In [128]:
kwds = interrogator(annual_trees, 'words', 'keywords', dictionary = 'self', quicksave = 'kwds')
Now, rather than a frequency count, you will be given the keyness of each word.
In [86]:
quickview(kwds.results)
In [65]:
kwds.table
Out[65]:
Let's sort these, based on those increasing/decreasing frequency:
In [66]:
inc = editor(kwds.results, sort_by = 'increase')
dec = editor(kwds.results, sort_by = 'decrease')
... and have a look:
In [67]:
quickview(inc, 15)
In [68]:
quickview(dec, 15)
As expected. Defunct states and former politicans are on the way out, while newer politicans are on the way in. We can do the same with n-grams, of course:
In [7]:
ngms = interrogator(annual_trees, 'words', 'ngrams', quicksave = 'ngms')
Neat. Now, let's make some thematic categories. This time, we'll make a list of tuples, containing regexes to match, and the result names:
In [69]:
regexes = [(r'\b(legislature|medicaid|republican|democrat|federal|council)\b', 'Government organisations'),
(r'\b(empire|merck|commerical)\b', 'Companies'),
(r'\b(athlete|policyholder|patient|yorkers|worker|infant|woman|man|child|children|individual|person)\b', 'People, everyday'),
(r'\b(marrow|blood|lung|ovarian|breast|heart|hormone|testosterone|estrogen|pregnancy|prostate|cardiovascular)\b', 'The body'),
(r'\b(reagan|clinton|obama|koch|slaney|starzl)\b', 'Specific people'),
(r'\b(implant|ect|procedure|abortion|radiation|hormone|vaccine|medication)\b', 'Treatments'),
(r'\b(addiction|medication|drug|statin|vioxx)\b', 'Drugs'),
(r'\b(addiction|coronary|aneurysm|mutation|injury|fracture|cholesterol|obesity|cardiovascular|seizure|suicide)\b', 'Symptoms'),
(r'\b(worker|physician|doctor|midwife|dentist)\b', 'Healthcare professional'),
(r'\b(transmission|infected|hepatitis|virus|hiv|lung|aids|asbestos|malaria|rabies)\b', 'Infectious disease'),
(r'\b(huntington|lung|prostate|breast|heart|obesity)\b', 'Non-infectious disease'),
(r'\b(policyholder|reinsurance|applicant|capitation|insured|insurer|insurance|uninsured)\b', 'Finance'),
(r'\b(experiment|council|journal|research|university|researcher|clinical)\b', 'Research')]
Now, let's loop through out list and merge keyword and n-gram entries:
In [71]:
# NOTE: you can use `print_info = False` if you don't want all this stuff printed.
for regex, name in regexes:
kwds = editor(kwds.results, merge_entries = regex, newname = name, print_info = False)
ngms = editor(ngms.results, merge_entries = regex, newname = name, print_info = False)
# now, remove all other entries
kwds = editor(kwds.results, just_entries = [name for regex, name in regexes])
ngms = editor(ngms.results, '%', ngms.totals, just_entries = [name for regex, name in regexes])
Pretty nifty, eh? Welp, let's plot them:
In [72]:
plotter('Key themes: keywords', kwds.results.drop('1963'), y_label = 'L/L Keyness', legend_pos = 'upper left')
plotter('Key themes: n-grams', ngms.results, y_label = 'Percentage of all n-grams')
So, we can now do some pretty cool stuff in just a few lines of code. Let's concordance the top five keywords, looking at the year in which they are most key:
In [782]:
import os
# iterate through results
for index, w in enumerate(list(kwds.results)[:5]):
# get the year with most occurrences
top_year = kwds.results[w].idxmax()
# print some info
print '\n%d: %s, %s' % (index + 1, w, str(top_year))
# get path to that subcorpus
top_dir = os.path.join(annual_trees, str(top_year))
# make a tregex query with token start and end defined
query = r'/(?i)^' + w + r'/'
# do concordancing
lines = conc(top_dir, query, random = True, n = 10)
You can easily generate collocates for corpora, subcorpora or concordance lines:
In [135]:
from corpkit import collocates
conc_colls = collocates(adj_lines)
for coll in conc_colls:
print coll
subc_colls = collocates('data/nyt/years/2003')
for coll in subc_colls:
if 'risk' not in coll:
print coll
With the collocates() function, you can specify the maximum distance at which two tokens will be considered collocates.
In [136]:
colls = collocates(adj_lines, window = 3)
for coll in colls:
print coll
The two functions are useful for visualising and searching individual syntax trees. They have proven useful as a way to practice your Tregex queries.
You could get trees by using conc() with a very large window and trees set to True. Alternatively, you can open files in the data directory directly, and paste them in.
quicktree() generates a visual representation of a parse tree. Here's one from 1989:
In [138]:
tree = '(ROOT (S (NP (NN Pre-conviction) (NN attachment)) (VP (VBZ carries) (PP (IN with) (NP (PRP it))) (NP (NP (DT the) (JJ obvious) (NN risk)) (PP (IN of) (S (VP (VBG imposing) (NP (JJ drastic) (NN punishment)) (PP (IN before) (NP (NN conviction)))))))) (. .)))'
# currently broken!
# quicktree(tree)
searchtree() requires a tree and a Tregex query. It will return a list of query matches.
In [139]:
from corpkit import searchtree
print searchtree(tree, r'/VB.?/ >># (VP $ NP)')
print searchtree(tree, r'NP')
Now you're familiar with the corpus and functions. In the sections below, we'll perform a formal, followed by a functional, analysis of risk. Let's start with the formal side of things:
In formal grammar, as we saw earlier, risk words can be nouns, verbs, adjectives and adverbs. Though we've seen that there are a lot of nouns, and that nouns are becoming more frequent, we don't yet know whether or not nouns are becoming more frequent in the NYT generally. To test this, we can do as follows:
In [87]:
# 'any' is a special query, which finds any tag if 'pos'
# and any word if 'words'.
baseline = interrogator(annual_trees, 'pos', 'any', lemmatise = True, quicksave = 'baseline')
riskpos = interrogator(annual_trees, 'pos', r'__ < /(?i).?\brisk.?/', lemmatise = True, quicksave = 'riskpos')
In the cell above, the lemmatise = True option will convert tags like 'NNS' to 'Noun'.
In [88]:
quickview(baseline.results, n = 10)
In [89]:
quickview(riskpos.results)
Now, we can calculate the percentage of the time that a noun is a risk noun (and so on).
In [15]:
open_words = ['Noun', 'Verb', 'Adjective', 'Adverb']
maths_done = editor(riskpos.results, '%', baseline.results, sort_by = 'total', just_entries = open_words, skip_subcorpora = [1963])
In [21]:
plotter('Percentage of open word classes that are risk words', maths_done.results,
y_label = 'Percentage', legend_pos = 'lower left')
plotter('Percentage of open word classes that are risk words', maths_done.results,
y_label = 'Percentage', legend_pos = 'lower left', kind = 'area', stacked = True)
Neat, huh? We can see that nominalisation of risk is a very real thing.
Our problem, however, is that formal categories like noun and verb only take us so far: in the phrase "risk metrics", risk is a noun, but performs a modifier function, for example. In the next section, we interrogate the corpus for functional, rather than formal categorisations of risk words.
Before we start our corpus interrogation, we'll also present a very brief explanation of Systemic Functional Linguistics—the theory of language that underlies our analytical approach.
Functional linguistics is a research area concerned with how realised language (lexis and grammar) work to achieve meaningful social functions. One functional linguistic theory is Systemic Functional Linguistics, developed by Michael Halliday.
In [145]:
from IPython.display import HTML
HTML('<iframe src=http://en.mobile.wikipedia.org/wiki/Michael_Halliday?useformat=mobile width=700 height=350></iframe>')
Out[145]:
Central to the theory is a division between experiential meanings and interpersonal meanings.
Halliday argues that these two kinds of meaning are realised simultaneously through different parts of English grammar.
Here's one visualisation of it. We're concerned with the two left-hand columns. Each level is an abstraction of the one below it.
Transitivity choices include fitting together configurations of:
Mood features of a language include:
Lexical density is usually a good indicator of the general tone of texts. The language of academia, for example, often has a huge number of nouns to verbs. We can approximate an academic tone simply by making nominally dense clauses:
The consideration of interest is the potential for a participant of a certain demographic to be in Group A or Group B.
Notice how not only are there many nouns (consideration, interest, potential, etc.), but that the verbs are very simple (is, to be).
In comparison, informal speech is characterised by smaller clauses, and thus more verbs.
A: Did you feel like dropping by?
B: I thought I did, but now I don't think I want to
Here, we have only a few, simple nouns (you, I), with more expressive verbs (feel, dropping by, think, want)
Note: SFL argues that through grammatical metaphor, one linguistic feature can stand in for another. Would you please shut the door? is an interrogative, but it functions as a command. invitation is a nominalisation of a process, invite. We don't have time to deal with these kinds of realisations, unfortunately.
A discourse analysis that is not based on grammar is not an analysis at all, but simply a running commentary on a text. - M.A.K. Halliday, 1994
Our analysis proceeded according to the description of the transitivity system in systemic functional grammar (SFG: see Halliday & Matthiessen, 2004).
The main elements of the transitivity system are participants (the arguments of main verbs) and processes (the verbal group). Broadly speaking, processes can be modified by circumstances (adverbs and prepositional phrases, and participants can be modified through epithets, classifiers (determiners, adjectives, etc).
This is an oversimplification, of course. Grab a copy of the Introduction to Functional Grammar to find out more.
Risk words can potentially be participants, processes or modifiers.
Risk-as-participant: any nominal argument of a process that is headed by a risk word. Examples:
Risk-as-process: risk word as the rightmost component of a VP. Examples:
Risk-as-modifier: any risk word that modifies a participant or process. This includes many adjectival risk words and many risk words appearing within prepositional or adverbial phrases. Examples:
To find the distributions of these, we define three (very long and complicated) Tregex queries as sublists of titles and patterns under query. We then use multiquery() to search for each query in turn.
In [4]:
from corpkit import multiquery
query = (['Participant', r'/(?i).?\brisk.?/ > (/NN.?/ >># (NP !> PP !> (VP <<# (/VB.?/ < '
'/(?i)\b(take|takes|taking|took|taken|run|runs|running|ran|pose|poses|posed|posing)\b/)))) | >># (ADJP > VP)'],
['Process', r'VP !> VP << (/VB.?/ < /(?i).?\brisk.?/) | > VP <+(VP) (/VB.?/ < '
'/(?i)(take|taking|takes|taken|took|run|running|runs|ran|put|putting|puts|pose|poses|posed|posing)/'
'>># (VP < (NP <<# (/NN.?/ < /(?i).?\brisk.?/))))'],
['Modifier', r'/(?i).?\brisk.?/ !> (/NN.?/ >># (NP !> PP !> (VP <<# (/VB.?/ < '
'/(?i)\b(take|takes|taking|took|taken|run|runs|running|ran|pose|poses|posed|posing)\b/)))) & !>># '
'(ADJP > VP) & !> (/VB.?/ >># VP) & !> (/NN.?/ >># (NP > (VP <<# (/VB.?/ < /(?i)\b('
'take|takes|taking|took|taken|run|runs|running|ran|pose|poses|posed|posing)\b/))))'])
functional_role = multiquery(annual_trees, query, quicksave = 'functional_role')
In [5]:
ppm = editor(functional_role.results, '%', allwords.totals)
In [6]:
plotter('Risk as participant, process and modifier', ppm.results)
Here we can see that modifier forms are become more frequent over time, and have overtaken risk processes. Later, we determine which modifier forms in particular are becoming more common.
In [7]:
# Perhaps you want to see the result without 1963?
plotter('Risk as participant, process and modifier', ppm.results.drop('1963'))
You shall know a word by the company it keeps. - J.R. Firth, 1957
Functionally, risk is most commonly a participant in the NYT. This gives us a lot of potential areas of interest. We'll go through a few here, but there are plenty of other things that we have to leave out for reasons of space.
Here, we need to import verbose regular expressions that match any relational, verbal or mental process.
In [8]:
from dictionaries.process_types import processes
print processes.relational
print processes.verbal
We can use these in our Tregex queries to look for the kinds of processes participant risks are involved in. First, let's get a count for all processes with risk participants:
In [9]:
# get total number of processes with risk participant
query = r'/VB.?/ ># (VP ( < (NP <<# /(?i).?\brisk.?/) | >+(/.P$/) (VP $ (NP <<# /(?i).?\brisk.?/))))'
proc_w_risk_part = interrogator(annual_trees, 'count', query, quicksave = 'proc_w_risk_part')
In [10]:
# subj_query = r'/VB.?/ < %s ># (VP >+(/.P$/) (VP $ (NP <<# /(?i).?\brisk.?/)))' % processes.relational
# obj_query = r'/VB.?/ < %s ># (VP < (NP <<# /(?i).?\brisk.?/))' % processes.relational
query = r'/VB.?/ < /%s/ ># (VP ( < (NP <<# /(?i).?\brisk.?/) | >+(/.P$/) (VP $ (NP <<# /(?i).?\brisk.?/))))' % processes.relational
relationals = interrogator(annual_trees, 'words', query, lemmatise = True, quicksave = 'relationals')
In [11]:
rels = editor(relationals.results, '%', proc_w_risk_part.totals)
In [12]:
plotter('Relational processes', rels.results)
First, we can look at adjectives that modify a participant risk.
In [14]:
query = r'/JJ.?/ > (NP <<# /(?i).?\brisk.?/)'
adj_modifiers = interrogator(annual_trees, 'words', query, lemmatise = True, quicksave = 'adj_modifiers')
In [13]:
adj_modifiers = load_result('adj_modifiers')
adj = editor(adj_modifiers.results, '%', adj_modifiers.totals)
plotter('Adjectives modifying nominal risk (lemmatised)', adj.results, num_to_plot = 7)
Yuck! That doesn't tell us much. Let's try visualising the data in a few different ways. First, let's see what the top results look like...
In [14]:
quickview(adj_modifiers.results)
OK, here are some ideas:
In [15]:
# remove words with five or more letters
small_adjs = editor(adj_modifiers.results, '%', adj_modifiers.totals, skip_entries = r'.{5,}')
plotter('Adjectives modifying nominal risk (lemmatised)', small_adjs.results, num_to_plot = 6)
#get results with seven or more letters
big_adjs = editor(adj_modifiers.results, '%', adj_modifiers.totals, just_entries = '.{10,}')
plotter('Adjectives modifying nominal risk (lemmatised)', big_adjs.results, num_to_plot = 4)
#get a few interesting points
lst = ['more', 'high', 'calculated', 'potential']
select_adjs = editor(adj_modifiers.results, '%', adj_modifiers.totals, just_entries = lst)
plotter('Adjectives modifying nominal risk (lemmatised)', select_adjs.results,
num_to_plot = 4)
Wow! What's happening with calculated risk in 1963? Let's modify the original Tregex query a little and use conc() to find out.
In [171]:
### old query: r'/JJ.?/ > (NP <<# /(?i).?\brisk.?/ ( > VP | $ VP))'
calculated_risk = r'/JJ.?/ < /(?i)calculated/> (NP <<# /(?i).?\brisk.?/ ( > VP | $ VP))'
# remove '( > VP | $ VP)' from the line above to get more instances
lines = conc('data/nyt/years/1963', calculated_risk)
Next, we'll look at risk of (noun) constructions, as in:
In [17]:
riskofsomething = r'/NN.?/ >># (NP > (PP <<# /(?i)of/ > (NP <<# (/NN.?/ < /(?i).?\brisk.?/))))'
lines = conc('data/nyt/years/1988', riskofsomething, n = 25, random = True)
Notice that singular and plural forms may be in the results: both substance and substances are returned, and would be counted as unique items.
If we want to ignore the difference between singular and plural (or different inflections of a verb), we need to use a lemmatiser. Luckily, interrogator() has one built in.
When lemmatisation is necessary, we can pass a lemmatise = True parameter to interrogator().
Lemmatisation requires knowing the part of speech of the input. interrogator() determines this by looking at the first part of the Tregex query: if it's /JJ.?/, the lemmatiser will be told that the word is an adjective. If the part of speech cannot be located, noun is used as a default. You can also manually pass a tag to the lemmatiser with a lemmatag = 'n/v/r/a' option.
In [18]:
# Risk of (noun)
risk_of = interrogator(annual_trees, 'words', riskofsomething, lemmatise = True, quicksave = 'risk_of')
In [20]:
rel_riskof = editor(risk_of.results, '%', risk_of.totals, print_info = False)
plotter('Risk of (noun)', rel_riskof.results, y_label = 'Percentage of all results',
legend_pos = 'upper right', kind = 'bar')
plotter('Risk of (noun), 1999-2013', editor(rel_riskof.results, span_subcorpora = [1999,2013], print_info = False).results)
At one point in our investigation, we looked specifically for military risks. From these results, we saw that risk of attack and risk of war were common. So, we plotted them:
In [21]:
quickview(risk_of, n = 20)
In [22]:
military = editor(risk_of.results, '%', risk_of.totals, just_entries = ['attack', 'war'])
plotter('Risk of (noun)', military.results)
# barh, just for fun
plotter('Risk of (noun)', military.results, kind = 'barh')
We thought it was interesting how risk of attack rose in frequency shortly after 9/11. So, we decided to look more closely at risk of attack:
In [177]:
attackrisk = r'/NN.?/ < /(?i)attack.?/ >># (NP > (PP <<# /(?i)of/ > (NP <<# (/NN.?/ < /(?i).?\brisk.?/))))'
lines = conc('data/nyt/years/2004', attackrisk, n = 15, random = True)
Whoops. We were wrong. Almost all occurrences actually referred to heart attack!
In [23]:
query = r'/NN.?/ < /(?i)\b(heart|terror).?/ $ (/NN.?/ < /(?i)\battack.?/ >># (NP > (PP <<# /(?i)of/ > (NP <<# (/NN.?/ < /().?\brisk.?/)))))'
terror_heart = interrogator(annual_trees, 'words', query, lemmatise = True, quicksave = 'terror_heart')
In [24]:
plotter('Risk of heart and terror* attack', terror_heart.results, num_to_plot = 2, legend_pos = 'upper left')
plotter('Risk of heart and terror* attack', terror_heart.results, num_to_plot = 2, kind = 'area', stacked = True)
So, we were a long way off-base. This is an ever-present danger in corpus linguistics. The decontextualisation needed to investigate the lexicogrammar of texts makes it easy to misunderstand (or worse, misrepresent) the data. Though concordancing is one of the oldest tasks in the corpus linguistic playbook, it remains a fundamental one, especially in discourse-analytic investigations.
... why did heart attacks become a big deal in 2004, you ask? Stay tuned ...
Here, we look at the kinds of predicators that occur when risk subject or object. Note that we remove run/take/pose risk, as these are actually verbal risks (see below).
By navigating parse trees in more complex ways, we can learn the kinds of processes risk as a participant is involved in.
In [25]:
query = (r'/VB.?/ !< /(?i)(take|taking|takes|taken|took|run|running|runs|ran|put|putting|puts|pose|poses|posing|posed)/' \
r' > (VP ( < (NP <<# (/NN.?/ < /(?i).?\brisk.?/))) | >+(VP) (VP $ (NP <<# (/NN.?/ < /(?i).?\brisk.?/))))')
predicators = interrogator(annual_trees, 'words', query, lemmatise = True, quicksave = 'predicators')
In [26]:
# Processes in which risk is subject/object
plotter('Processes in which risk is subject or object', editor(predicators.results, '%', predicators.totals).results, num_to_plot = 7)
# skip be:
plotter('Processes in which risk is subject or object', editor(predicators.results,
'%', predicators.totals, skip_entries = ['be']).results, num_to_plot = 5)
Interesting!
When risk is the main verb in a clause (e.g. don't risk it), it is the process. There are other kinds of risk processes, however: when risk occurs as the first object argument of certain nouns, it may be classified as a process-range configuration (an SFL term). Searching the data reveals four other main kinds of risk process:
In these cases, the expression is more or less idiomatic, and the main verb carries little semantic weight (Eggins, 2004).
We tracked the relative frequency of each construction over time.
In [28]:
query = ([u'risk', r'VP <<# (/VB.?/ < /(?i).?\brisk.?\b/)'],
[u'take risk', r'VP <<# (/VB.?/ < /(?i)\b(take|takes|taking|took|taken)+\b/) < (NP <<# /(?i).?\brisk.?\b/)'],
[u'run risk', r'VP <<# (/VB.?/ < /(?i)\b(run|runs|running|ran)+\b/) < (NP <<# /(?i).?\brisk.?\b/)'],
[u'put at risk', r'VP <<# /(?i)(put|puts|putting)\b/ << (PP <<# /(?i)at/ < (NP <<# /(?i).?\brisk.?/))'],
[u'pose risk', r'VP <<# (/VB.?/ < /(?i)\b(pose|poses|posed|posing)+\b/) < (NP <<# /(?i).?\brisk.?\b/)'])
processes = multiquery(annual_trees, query, quicksave = 'processes')
In [29]:
proc_rel = editor(processes.results, '%', processes.totals)
In [30]:
plotter('Risk processes', proc_rel.results)
Subordinate processes are often embedded within clauses containing a risk predicator, as in Obama risks alienating voters.
In [31]:
# to risk losing/being/having etc
query = r'VBG >># (VP > (S > (VP <<# (/VB.?/ < /(?i).?\brisk.?/))))'
risk_verbing = interrogator(annual_trees, 'words', query, quicksave = 'risk_verbing')
In [32]:
r_verbing = editor(risk_verbing.results, '%', risk_verbing.totals)
plotter('Process as risked thing', r_verbing.results, y_label = 'Percentage of all occurrences')
In this kind of risk process, the risker is typically a powerful member of society. While this is rather explicit in some cases (it's hard to image that a mechanic would risk alienating his/her apprentice), we can observe that this is the case for less obvious examples, like to risk becoming:
In [33]:
lines = conc('data/nyt/years/2013', r'VBG < /(?i)becom/ >># (VP > (S > (VP <<# (/VB.?/ < /(?i).?\brisk.?/))))', n = 15, random = True)
In [34]:
query = r'/NN.?/ !< /(?i).?\brisk.?/ >># (@NP $ (VP <+(VP) (VP ( <<# (/VB.?/ < /(?i).?\brisk.?/) | <<# (/VB.?/ < /(?i)(take|taking|takes|taken|took|run|running|runs|ran|put|putting|puts|pose|poses|posed|posing)/) < (NP <<# (/NN.?/ < /(?i).?\brisk.?/))))))'
subj_of_risk_process = interrogator(annual_trees, 'words', query,
lemmatise = True, quicksave = 'subj_of_risk_process')
In [61]:
quickview(subj_of_risk_process)
In [65]:
for s in ['total', 'increase', 'decrease']:
with_sort = editor(subj_of_risk_process.results, '%', subj_of_risk_process.totals,
sort_by = s, skip_subcorpora = [1963], print_info = False)
plotter('Subjects of risk processes', with_sort.results, num_to_plot = 10)
In [66]:
inst = ['government', 'bank', 'senator', 'republican',
'democrat', 'senate', 'congress', 'agency', 'firm']
peop = ['man', 'person', 'woman', 'child', 'baby', 'worker', 'consumer']
cats = editor(subj_of_risk_process.results, merge_entries = inst, newname = 'Institutions')
cats = editor(cats.results, merge_entries = peop, newname = 'People',
just_entries = ['Institutions', 'People'], sort_by = 'total')
In [67]:
for k in ['bar', 'line']:
plotter('People vs. institutions', cats.results.drop('1963'), kind = k)
A novel thing we can do with our data is determine the amount of time a word occurs in a given role. We know that Bush, Clinton, woman, bank, and child are common nouns in the corpus, but we do not yet know what percentage of the time they are playing a specific role in the risk frame.
To determine what percentage of the time these words take the role of risker, we start by counting their occurrences as risker, and in the corpus as a whole:
In [64]:
n_query = r'/NN.?/ !< /(?i).?\brisk.?/ >># NP'
noun_lemmata = interrogator(annual_trees, 'words', n_query, lemmatise = True, quicksave = 'noun_lemmata')
Then, we pass editor() a second list of results, rather than just totals, and use the just_totals = True' argument:
In [35]:
subj_of_risk_process = load_result('subj_of_risk_process')
noun_lemmata = load_result('noun_lemmata')
rel_risker = editor(subj_of_risk_process.results, '%', noun_lemmata.results, just_totals = True, sort_by = 'total')
Note that a threshold was printed. This number represents the minimum number of times an entry must occur in noun_lemmata.totals in order for the result to count.
We can pass in a threshold of our own. Note that if we set it to zero, unusual words are at the top of the results list:
In [71]:
rel_risker = editor(subj_of_risk_process.results, '%', noun_lemmata.results,
just_totals = True, threshold = 1, sort_by = 'total')
print rel_risker.results
Aside from giving it an integer value, you can pass it 'low', 'medium' or 'high'. editor() then creates thresholds based on the total total of noun_lemmata.totals. Passing no threshold results in 'medium being used as the default (total words in second list / 5000):
In [72]:
#total / 10,000
tmp = editor(subj_of_risk_process.results, '%', noun_lemmata.results,
just_totals = True, threshold = 'low', sort_by = 'total')
#total / 5,000
tmp = editor(subj_of_risk_process.results, '%', noun_lemmata.results,
just_totals = True, threshold = 'medium', sort_by = 'total')
#total / 2,500
rel_risker = editor(subj_of_risk_process.results, '%', noun_lemmata.results,
just_totals = True, threshold = 'high', sort_by = 'total')
OK, let's see what we have:
In [45]:
riskobject_regex = (r'(?i)^\b(life|everything|money|career|health|reputation|capital|future|'
r'job|safety|possibility|anything|return|neck|nothing|lot)$\b')
riskedthings = editor(risk_objects.results, skip_entries = riskobject_regex)
potentialharm = editor(risk_objects.results, just_entries = riskobject_regex)
plotter('Risked things', potentialharm.results, num_to_plot = 7)
# a method for quickly removing entries from the plot:
plotter('Risked things (minus life)', potentialharm.results.drop('life', axis = 1), num_to_plot = 3)
plotter('Potential harm', riskedthings.results, num_to_plot = 7)
It's interesting how powerful people risk losing and alienating electorates, fanbases or contracts, while less powerful people risk their jobs and safety, or their life or neck.
Risk words can serve as modifiers in a number of ways. We divided risk as modifier into five main types.
| Modifier type | Example |
|---|---|
| Adjectival modifiers of participants | the riskiest decision |
| Pre-head nominal modifiers of participants | risk management |
| Post-head nominal modifiers of participants | the money to risk |
| Adverbial modifiers of processes | it riskily moved |
| As head of NP that is head of a cirumstance | she was at risk |
In [64]:
query = ([u'Adjectival modifier', r'/NN.?/ >># (NP < (/JJ.?/ < /(?i).?\brisk.?/))'],
[u'Pre-head nominal modifier', r'/NN.?/ < /(?i).?\brisk.?/ $ (/NN.?/ >># NP !> CC)'],
[u'Post-head modifier', r'/NN.?/ >># (NP < (PP < (NP <<# /(?i).?\brisk.?/)))'],
[u'Adverbial modifier', r'RB < /(?i).?\brisk.?/'],
[u'Circumstance head', r'/NN.?/ < /(?i).?\brisk.?/ >># (NP > (PP > (VP > /\b(S|SBAR|ROOT)\b/)))'])
modifiers = multiquery(annual_trees, query, quicksave = 'modifiers')
Here are a few examples of each type:
In [28]:
for name, q in query:
print '\n%s:' % name
l = conc('data/nyt/years/2012', q, n = 5, random = True)
In [484]:
plotter('Types of risk modifiers', editor(modifiers.results, '%', modifiers.totals, skip_subcorpora = [1963]).results)
This is very interesting: the most common form in 1987 has become the least common in 2014!
We can also pull out words modified by adjectival risk:
In [421]:
# Participants modified by risk word
query = r'/NN.?/ >># (NP < (/JJ.?/ < /(?i).?\brisk.?/) ( > VP | $ VP))'
mod_by_adj_risk = interrogator(annual_trees, 'words', query,
lemmatise = True, titlefilter = False, quicksave = 'mod_by_adj_risk')
In [68]:
mod_by_adj_risk = load_result('mod_by_adj_risk')
plotter('Participants modified by risk', mod_by_adj_risk.results,
num_to_plot = 7)
We looked at the most common adjectival risks already. We can load that now, and look at the query to make sure:
In [47]:
#query = r'/JJ.?/ < /(?i).?\brisk.?/'
#adjrisks = interrogator(annual_trees, 'words', query,
#lemmatise = False, quicksave = 'adjrisks')
adjrisks = load_result('adj_riskwords')
In [48]:
arisk = editor(adjrisks.results, '%', allwords.totals)
In [49]:
# remember that we can still plot using all words/all risk words
plotter('Most common adjectival risks', arisk.results, y_label = 'Percentage of all words', num_to_plot = 5)
Given the increasing frequency of at-risk constructions, we then looked at what it is that this modifier typically modifies.
In [786]:
# At-risk thing
query = r'/NN.?/ >># (NP < (/JJ.?/ < /(?i).?\bat-risk/) ( > VP | $ VP))'
at_risk_things = interrogator(annual_trees, 'words', query, lemmatise = True, quicksave = 'at_risk_things')
In [134]:
at_risk_things = load_result('at_risk_things')
a_r_t = editor(at_risk_things.results, '%', at_risk_things.totals, skip_subcorpora = [1963], just_totals = True)
In [143]:
plotter('At-risk things', a_r_t.results, kind = 'pie', num_to_plot = 8, partial_pie = True,
show_totals = 'legend', shadow = True)
plotter('At-risk things', a_r_t.results, kind = 'pie', num_to_plot = 'all',
colours = 'summer', show_totals = False, pie_legend = False, fontsize = 15, figsize = (7, 8))
plotter('At-risk things', a_r_t.results, kind = 'bar', num_to_plot = 20, cumulative = True, colours = 'autumn')
The query below finds both thing at risk and at-risk thing.
In [50]:
# at-risk person / person at risk combined
query = r'/NN.?/ ( >># (NP < (PP <<# /(?i)at/ << (NP <<# /(?i)\brisk.?/))) | ( >># (NP < (/JJ.?/ < /(?i)at-risk.?/))))'
n_atrisk_n = interrogator(annual_trees, 'words', query,
lemmatise = False, titlefilter = False, quicksave = 'n_atrisk_n')
In [ ]:
n_atrisk_n = load_result('n_atrisk_n')
plotter('At-risk thing or thing at risk', n_atrisk_n.results, legend_pos = 'upper left')
Vulnerable human populations are the main theme of this category: indeed, it's difficult to imagine at-risk corporations or at-risk leaders.
We searched to find the most common proper noun strings.
interrogator()'s titlefilter option removes common titles, first names and determiners to make for more accurate counts. It is useful when the results being returned are groups/phrases, rather than single words.
In [65]:
# Most common proper noun phrases
query = r'NP <# NNP >> (ROOT << /(?i).?\brisk.?\b/)'
propernouns = interrogator(annual_trees, 'words', query,
titlefilter = True, quicksave = 'propernouns')
In [ ]:
propernouns = load_result('propernouns')
for s in ['total', 'increase', 'decrease']:
r = editor(propernouns.results, '%', propernouns.totals, skip_subcorpora=[1963], sort_by = s, print_info = False)
plotter('Most common proper noun phrases', r.results, num_to_plot = 9)
In [491]:
quickview(propernouns, n = 200)
Notice that there are a few entries here that refer to the same group. (f.d.a. and food and drug administration, for example). We can use editor() to fix these.
In [1]:
to_merge = [("f.d.a .", "food and drug administration"),
("fed", "federal reserve"),
("s.e.c .", "securities and exchange commission"),
("disease control", "disease control and prevention"),
("goldman sachs", "goldman"),
("e.p.a .", "envi,ronmental protection agency"),
("calif .", "california"),
("i.m.f .", "international monetary fund")]
propernouns = load_result('propernouns') # just in case
for short, lon in to_merge:
propernouns = editor(propernouns.results, '%', propernouns.totals, merge_entries = [short, lon],
newname = short, print_info = False)
propernouns = editor(propernouns.results, sort_by = 'total', print_info = False)
propernouns.results
Out[1]:
Now that we've merged some common results, we can build some basic thematic categories. Let's make a list of lists:
In [2]:
theme_list = [['People', r'(?i)^\b(bush|clinton|obama|greenspan|gore|johnson|mccain|romney|kennedy|giuliani|reagan)$\b'],
['Nations', r'(?i)^\b(iraq|china|america|israel|russia|japan|frace|germany|iran|britain|u\.s\.|afghanistan|australia|canada|spain|mexico|pakistan|soviet union|india)$\b'],
['Geopolitical entities', r'(?i)^\b(middle east|asia|europe|america|soviet union|european union)$\b'],
['US places', r'(?i)^\b(new york|washington|wall street|california|manhattan|new york city|new jersey|north korea|italy|greece|bosniaboston|los angeles|broadway|texas)$\b'],
['Companies', r'(?i)^\b(merck|avandia|citigroup|pfizer|bayer|enron|apple|microsoft|empire)$\b'],
['Organisations', r'(?i)^\b(white house|congress|federal reserve|nasa|pentagon|f\.d\.a \.|c\.i\.a \.|f\.b\.i \.|e\.p\.a \.)$'],
['Medical', r'(?i)^\b(vioxx|aids|aid|celebrex|f\.d\.a \.|pfizer|bayer|merck|avandia)$']]
We can add each result to its entry in theme_list, and give the totals a name:
In [3]:
# add data to our sublists
for entry in theme_list:
entry.append(editor(propernouns.results, '%', propernouns.totals,
just_entries = entry[1], print_info = False))
# rename the newly created data
entry[2].totals.name = entry[0]
In [4]:
# plot some results
ystring = 'Percentage of all proper noun groups'
for name, query, data in theme_list:
plotter(name, data.results, y_label = ystring, legend_pos = 'upper left')
Let's compare these topics in the same chart, using Pandas to join everything together:
In [5]:
import pandas
# get the totals from each theme and put them together
them_comp = pandas.concat([data.totals for name, query, data in theme_list], axis=1)
them_comp = editor(them_comp, sort_by = 'total')
quickview(them_comp)
In [8]:
plotter('Themes', them_comp.results, y_label = 'Percentage of all words')
plotter('Themes', them_comp.results, kind = 'area', stacked = True, y_label = 'Percentage of all words')
plotter('Themes', them_comp.results, subplots = True, y_label = 'Percentage of all words', figsize = (8, 16))
These charts reveal some interesting patterns.
In [158]:
vioxx = editor(propernouns.results, '%', propernouns.totals, just_entries = r'(?i)^\b(vioxx|merck)\b$', skip_subcorpora=1963)
plotter('Merck and Vioxx', vioxx.results)
Vioxx was removed from shelves following the discovery that it increased the risk of heart attack. It's interesting how even though terrorism and war may come to mind when thinking of risk in the past 15 years, this health topic is easily more prominent in the data.
OK, that's enough interrogating and plotting for now. Future documents will demonstrate how we can use Stanford Dependencies, instead of parse trees, to understand change in the risk semantic.
A key challenge accounting for the diverse ways in which a semantic meaning can be made in lexis and grammar. If we are interested in how often money is the risked thing, we have to design searches that find:
She risked her money
She risked losing her money
Money was risked
It was risked money
The risk of money loss was there
She took her money from her purse and risked it.
Though we can design queries to match any of these, it is very difficult to automate this process for every possible 'risked thing'. It's also very hard to know when we have finally developed a query that matches everything we want.
An added issue is how to treat things like:
She didn't risk her money
She risked no money
She could risk money
Here, the semantic meanings are very different (the risking of money did not occur), but each would match the queries we designed for the above.
Should these results be counted or excluded? Why?
Eggins, S. (2004). Introduction to systemic functional linguistics. Continuum International Publishing Group.
Firth, J. (1957). A Synopsis of Linguistic Theory 1930-1955. In: Studies in Linguistic Analysis, Philological Society, Oxford; reprinted in Palmer, F. (ed.) 1968 Selected Papers of J. R. Firth, Longman, Harlow.
Halliday, M., & Matthiessen, C. (2004). An Introduction to Functional Grammar. Routledge.
In [ ]:
In [ ]:
In [ ]:
In [4]:
modals = load_result('modals')
allwords = load_result('allwords')
edited_adj_riskwords = load_result('edited_adj_riskwords')
propernouns = load_result('propernouns')
n_atrisk_n = load_result('n_atrisk_n')
risk_objects = load_result('risk_objects')
subj_of_risk_process = load_result('subj_of_risk_process')
risk_verbing = load_result('risk_verbing')
processes = load_result('processes')
predicators = load_result('predicators')
terror_heart = load_result('terror_heart')
risk_of = load_result('risk_of')
relationals = load_result('relationals')
proc_w_risk_part = load_result('proc_w_risk_part')
functional_role = load_result('functional_role')
riskpos = load_result('riskpos')
baseline = load_result('baseline')
adj_riskwords = load_result('adj_riskwords')
at_risk_things = load_result('at_risk_things')
sayers = load_result('sayers')
noun_lemmata = load_result('noun_lemmata')
x_subj_of_risk_process = load_result('x_subj_of_risk_process')
adj_modifiers = load_result('adj_modifiers')
ngms = load_result('ngms')
modals_lemmatised = load_result('modals_lemmatised')
riskwords = load_result('riskwords')
kwds = load_result('kwds')
In [5]:
from corpkit import load_all_results
r = load_all_results()
In [ ]: