We've spent a lot of time in python dealing with text data, and that's because text data is everywhere. It is the primary form of communication between persons and persons, persons and computers, and computers and computers. The kind of inferential methods that we apply to text data, however, are different from those applied to tabular data.
This is partly because documents are typically specified in a way that expresses both structure and content using text (i.e. the document object model).
Largely, however, it's because text is difficult to turn into numbers in a way that preserves the information in the document. Today, we'll talk about dominant language model in NLP and the basics of how to implement it in Python.
This is also sometimes referred to as "bag-of-words" by those who don't think very highly of it. The term document model looks at language as individual communicative efforts that contain one or more tokens. The kind and number of the tokens in a document tells you something about what is attempting to be communicated, and the order of those tokens is ignored.
To start with, let's load a document.
In [1]:
import nltk
#nltk.download('webtext')
document = nltk.corpus.webtext.open('grail.txt').read()
Let's see what's in this document
In [2]:
len(document.split('\n'))
Out[2]:
In [3]:
document.split('\n')[0:10]
Out[3]:
It looks like we've gotten ourselves a bit of the script from Monty Python and the Holy Grail. Note that when we are looking at the text, part of the structure of the document is written in tokens. For example, stage directions have been placed in brackets, and the names of the person speaking are in all caps.
If we wanted to read out all of the stage directions for analysis, or just King Arthur's lines, doing so in base python string processing will be very difficult. Instead, we are going to use regular expressions. Regular expressions are a method for string manipulation that match patterns instead of bytes.
In [4]:
import re
snippet = "I fart in your general direction! Your mother was a hamster, and your father smelt of elderberries!"
re.search(r'mother', snippet)
Out[4]:
Just like with str.find, we can search for plain text. But re also gives us the option for searching for patterns of bytes - like only alphabetic characters.
In [5]:
re.search(r'[a-z]', snippet)
Out[5]:
In this case, we've told re to search for the first sequence of bytes that is only composed of lowercase letters between a and z. We could get the letters at the end of each sentence by including a bang at the end of the pattern.
In [6]:
re.search(r'[a-z]!', snippet)
Out[6]:
If we wanted to pull out just the stage directions from the screenplay, we might try a pattern like this:
In [7]:
re.findall(r'[a-zA-Z]', document)[0:10]
Out[7]:
So that's obviously no good. There are two things happening here:
[ and ] do not mean 'bracket'; they are special characters which mean 'any thing of this class'A better regular expression, then, would wrap this in escaped brackets, and include a command saying more than one letter.
Re is flexible about how you specify numbers - you can match none, some, a range, or all repetitions of a sequence or character class.
| character | meaning |
|---|---|
{x} |
exactly x repetitions |
{x,y} |
between x and y repetitions |
? |
0 or 1 repetition |
* |
0 or many repetitions |
+ |
1 or many repetitions |
In [8]:
re.findall(r'\[[a-zA-Z]+\]', document)[0:10]
Out[8]:
This is better, but it's missing that [clop clop clop] we saw above. This is because we told the regex engine to match any alphabetic character, but we did not specify whitespaces, commas, etc. to match these, we'll use the dot operator, which will match anything expect a newline.
Part of the power of regular expressions are their special characters. Common ones that you'll see are:
| character | meaning |
|---|---|
. |
match anything except a newline |
^ |
match the start of a line |
$ |
match the end of a line |
\s |
matches any whitespace or newline |
Finally, we need to fix this + character. It is a 'greedy' operator, which means it will match as much of the string as possible. To see why this is a problem, try:
In [9]:
snippet = 'This is [cough cough] and example of a [really] greedy operator'
re.findall(r'\[.+\]', snippet)
Out[9]:
Since the operator is greedy, it is matching everything inbetween the first open and the last close bracket. To make + consume the least possible amount of string, we'll add a ?.
In [10]:
p = re.compile(r'\[.+?\]')
re.findall(p, document)[0:10]
Out[10]:
What if we wanted to grab all of Arthur's speech? This one is a little trickier, since:
If we wanted to do this using base string manipulation, we would need to do something like:
split the document into lines
create a new list of just lines that start with ARTHUR
create a newer list with ARTHUR removed from the front of each elementRegex gives us a way of doing this in one line, by using something called groups. Groups are pieces of a pattern that can be ignored, negated, or given names for later retrieval.
| character | meaning |
|---|---|
(x) |
match x |
(?:x) |
match x but don't capture it |
(?P<x>) |
match something and give it name x |
(?=x) |
match only if string is followed by x |
(?!x) |
match only if string is not followed by x |
In [11]:
p = re.compile(r'(?:ARTHUR: )(.+)')
re.findall(p, document)[0:10]
Out[11]:
Because we are using findall, the regex engine is capturing and returning the normal groups, but not the non-capturing group. For complicated, multi-piece regular expressions, you may need to pull groups out separately. You can do this with names.
In [12]:
p = re.compile(r'(?P<name>[A-Z ]+)(?::)(?P<line>.+)')
match = re.search(p, document)
match
Out[12]:
In [13]:
match.group('name'), match.group('line')
Out[13]:
To check that you've understood something about regular expressions, we're going to have you do a small test challenge. Partner up with the person next to you - we're going to do this as a pair coding exercise - and choose which computer you are going to use.
Then, navigate to challenges/03_analysis/ and read through challenge A. When you think you've completed it successfully, run py.test test_A.py .
In [14]:
p = re.compile(r'(?:ARTHUR: )(.+)')
arthur = ' '.join(re.findall(p, document))
arthur[0:100]
Out[14]:
In our model for natural language, we're interested in words. The document is currently a continuous string of bytes, which isn't ideal. You might be tempted to separate this into words using your newfound regex knowledge:
In [15]:
p = re.compile(r'\w+', flags=re.I)
re.findall(p, arthur)[0:10]
Out[15]:
But this is problematic for languages that make extensive use of punctuation. For example, see what happens with:
In [16]:
re.findall(p, "It isn't Dav's cheesecake that I'm worried about")
Out[16]:
The practice of pulling apart a continuous string into units is called "tokenizing", and it creates "tokens". NLTK, the canonical library for NLP in Python, has a couple of implementations for tokenizing a string into words.
In [17]:
from nltk import word_tokenize
word_tokenize("It isn't Dav's cheesecake that I'm worried about")
Out[17]:
The distinction here is subtle, but look at what happened to "isn't". It's been separated into "IS" and "N'T", which is more in keeping with the way contractions work in English.
In [18]:
tokens = word_tokenize(arthur)
tokens[0:10]
Out[18]:
At this point, we can start asking questions like what are the most common words, and what words tend to occur together.
In [19]:
len(tokens), len(set(tokens))
Out[19]:
So we can see right away that Arthur is using the same words a whole bunch - on average, each unique word is used four times. This is typical of natural language.
Not necessarily the value, but that the number of unique words in any corpus increases much more slowly than the total number of words.
A corpus with 100M tokens, for example, probably only has 100,000 unique tokens in it.
For more complicated metrics, it's easier to use NLTK's classes and methods.
In [20]:
from nltk import collocations
fd = collocations.FreqDist(tokens)
fd.most_common()[:10]
Out[20]:
In [21]:
measures = collocations.BigramAssocMeasures()
c = collocations.BigramCollocationFinder.from_words(tokens)
c.nbest(measures.pmi, 10)
Out[21]:
In [22]:
c.nbest(measures.likelihood_ratio, 10)
Out[22]:
We see here that the collocation finder is pulling out some things that have face validity. When Arthur is talking about peasants, he calls them "bloody" more often than not. However, collocations like "Brother Maynard" and "BLACK KNIGHT" are less informative to us, because we know that they are proper names.
If you were interested in collocations in particular, what step do you think you would have to take during the tokenizing process?
This has gotten us as far identical tokens, but in language processing, it is often the case that the specific form of the word is not as important as the idea to which it refers. For example, if you are trying to identify the topic of a document, counting 'running', 'runs', 'ran', and 'run' as four separate words is not useful. Reducing words to their stems is a process called stemming.
A popular stemming implementation is the Snowball Stemmer, which is based on the Porter Stemmer. It's algorithm looks at word forms and does things like drop final 's's, 'ed's, and 'ing's.
Just like the tokenizers, we first have to create a stemmer object with the language we are using.
In [23]:
snowball = nltk.SnowballStemmer('english')
Now, we can try stemming some words
In [24]:
snowball.stem('running')
Out[24]:
In [25]:
snowball.stem('eats')
Out[25]:
In [26]:
snowball.stem('embarassed')
Out[26]:
Snowball is a very fast algorithm, but it has a lot of edge cases. In some cases, words with the same stem are reduced to two different stems.
In [27]:
snowball.stem('cylinder'), snowball.stem('cylindrical')
Out[27]:
In other cases, two different words are reduced to the same stem.
This is sometimes referred to as a 'collision'
In [28]:
snowball.stem('vacation'), snowball.stem('vacate')
Out[28]:
In [29]:
snowball.stem('organization'), snowball.stem('organ')
Out[29]:
In [30]:
snowball.stem('iron'), snowball.stem('ironic')
Out[30]:
In [31]:
snowball.stem('vertical'), snowball.stem('vertices')
Out[31]:
A more accurate approach is to use an English word bank like WordNet to call dictionary lookups on word forms, in a process called lemmatization.
In [32]:
# nltk.download('wordnet')
wordnet = nltk.WordNetLemmatizer()
In [33]:
wordnet.lemmatize('iron'), wordnet.lemmatize('ironic')
Out[33]:
In [34]:
wordnet.lemmatize('vacation'), wordnet.lemmatize('vacate')
Out[34]:
Nothing comes for free, and you've probably noticed already that the lemmatizer is slower. We can see how much slower with one of IPYthon's magic functions.
In [38]:
%timeit wordnet.lemmatize('table')
In [39]:
4.45 * 5.12
Out[39]:
In [37]:
%timeit snowball.stem('table')
Frequently, we are interested in text to learn something about the person who is speaking. One of these things we've talked about already - linguistic diversity. A similar metric was used a couple of years ago to settle the question of who has the largest vocabulary in Hip Hop.
Unsurprisingly, top spots go to Canibus, Aesop Rock, and the Wu Tang Clan. E-40 is also in the top 20, but mostly because he makes up a lot of words; as are OutKast, who print their lyrics with words slurred in the actual typography
Another thing we can learn is about how the speaker is feeling, with a process called sentiment analysis. Before we start, be forewarned that this is not a robust method by any stretch of the imagination. Sentiment classifiers are often trained on product reviews, which limits their ecological validity.
We're going to use TextBlob's built-in sentiment classifier, because it is super easy.
In [40]:
from textblob import TextBlob
In [43]:
blob = TextBlob(arthur)
In [46]:
for sentence in blob.sentences[10:25]:
print(sentence.sentiment.polarity, sentence)
Another common NLP task is to look for semantic distance between documents. This is used by search engines like Google (along with other things like PageRank) to decide which websites to show you when you search for things like 'bike' versus 'motorcycle'.
It is also used to cluster documents into topics, in a process called topic modeling. The math behind this is beyond the scope of this course, but the basic strategy is to represent each document as a one-dimensional array, where the indices correspond to integer ids of tokens in the document. Then, some measure of semantic similarity, like the cosine of the angle between unitized versions of the document vectors, is calculated.
Luckily for us there is another python library that takes care of the heavy lifting for us.
In [62]:
from gensim import corpora, models, similarities
We already have a document for Arthur, but let's grab the text from someone else to compare it with.
In [60]:
p = re.compile(r'(?:GALAHAD: )(.+)')
galahad = ' '.join(re.findall(p, document))
arthur_tokens = tokens
galahad_tokens = word_tokenize(galahad)
Now, we use gensim to create vectors from these tokenized documents:
In [76]:
dictionary = corpora.Dictionary([arthur_tokens, galahad_tokens])
corpus = [dictionary.doc2bow(doc) for doc in [arthur_tokens, galahad_tokens]]
tfidf = models.TfidfModel(corpus, id2word=dictionary)
Then, we create matrix models of our corpus and query
In [77]:
query = tfidf[dictionary.doc2bow(['peasant'])]
index = similarities.MatrixSimilarity(tfidf[corpus])
And finally, we can test our query, "peasant" on the two documents in our corpus
In [78]:
list(enumerate(index[query]))
Out[78]:
So we see here that "peasant" does not match Galahad very well (a really bad match would have a negative value), and is more similar to the kind of speach output that we see from King Arthur.
In data storage, data visualization, inferential statistics, and machine learning, the most common way to pass data between applications is in the form of tables (these are called tabular, structured, or rectangular data). These are convenient in that, when used correctly, they store data in a DRY and easily queryable way, and are also easily turned into matrices for numeric processing.
note - it is sometimes tempting to refer to N-dimensional matrices as arrays, following the numpy naming convention, but these are not the same as arrays in C++ or Java, which may cause confusion
It is common in enterprise applications to store tabular data in a SQL database. In the sciences, data is typically passed around as comma separated value files (.csv), which you have already been dealing with over the course of the last two days.
For this brief introduction to analyzing tabular data, we'll be using the scipy stack, which includes numpy, pandas, scipy, and "scikits" like sk-learn and sk-image.
In [6]:
import pandas as pd
You might not have seen this as convention yet. It is just telling python that when we import pandas, we don't want to access it in the namespace as pandas but as pd instead.
We'll start by making a small table to practice on. Tables in pandas are called data frames, so we'll start by making an instance of class DataFrame, and initialize it with some data.
note - pandas and R use the same name for their tables, but their behavior is often very different
In [2]:
table = pd.DataFrame({'id': [1,2,3], 'name':['dillon','juan','andrew'], 'age':[47,27,23]})
print(table)
Variables in pandas are represented by a pandas-specific data structure, called a Series. You can grab a Series out of a DataFrame by using the slicing operator with the name of the variable that you want to pull.
In [3]:
table['name'], type(table['name'])
Out[3]:
We could have made each variable a Series, and then put it into the DataFrame object, but it's easier in this instance to pass in a dictionary where the keys are variable names and the values are lists. You can also modify a data frame in place using similar syntax:
In [4]:
table['fingers'] = [9, 10, None]
If you try to run that code without the None there, pandas will return an error. In a table (in any language) each column must have the same number of rows.
We've entered None, base python's missingness indicator, but pandas is going to swap this out with something else:
In [5]:
table['fingers']
Out[5]:
You might be tempted to write your own control structures around these missing values (which are variably called NaN, nan, and NA), but this is always a bad idea:
In [6]:
table['fingers'][2] == None
Out[6]:
In [7]:
table['fingers'][2] == 'NaN'
Out[7]:
In [8]:
type(table['fingers'][2]) == str
Out[8]:
None of this works because the pandas NaN is a subclass of numpy's double precision floating point number. However, for ambiguous reasons, even numpy.nan does not evaluate as being equal to itself.
To handle missing data, you'll need to use the pandas method isnull.
In [9]:
pd.isnull(table['fingers'])
Out[9]:
In the same way that we've been pulling out columns by name, you can pull out rows by index. If I want to grab the first row, I can use:
In [10]:
table[:1]
Out[10]:
Recall that indices in python start at zero, and that selecting by a range does not include the final value (i.e. [ , )).
Unlike other software languages (R, I'm looking at you here), row indices in pandas are immutable. So, if I rearrange my data, the index also get shuffled.
In [11]:
table.sort_values('age')
Out[11]:
Because of this, it's common to set the index to be something like a timestamp or UUID.
We can select parts of a DataFrame with conditional statements:
In [12]:
table[table['age'] < 40]
Out[12]:
In [13]:
other_table = pd.DataFrame({
'name':['dav', 'juan', 'dillon'],
'languages':['python','python','python']})
In [14]:
table.merge(other_table, on='name')
Out[14]:
Note that we have done an "inner join" here, which means we are only getting the intersection of the two tables. If we want the union, we can specify that we want an outer join:
In [15]:
table.merge(other_table, on='name', how='outer')
Out[15]:
Or maybe we want all of the data from table, but not other_table
In [16]:
table.merge(other_table, on='name', how='left')
Out[16]:
To make analysis easier, you may have to reshape your data. It's easiest to deal with data when each table meets the follwing criteria:
This kind of format is easy to work with, because:
To make this more concrete, let's take an example table.
| name | city1 | city2 | population |
|---|---|---|---|
| dillon | williamsburg | berkeley | 110 |
| juan | berkeley | berkeley | 110 |
| dav | cambridge | berkeley | 110 |
This table violates all three of the rules above. Specifically, it:
In this particular example, our data is too wide. If we create that dataframe in pandas
In [17]:
wide_table = pd.DataFrame({'name' : ['dillon', 'juan', 'dav'],
'city1' : ['williamsburg', 'berkeley', 'cambridge'],
'city2' : ['berkeley', 'berkeley', 'berkeley'],
'population' : [110, 110, 110]
})
wide_table
Out[17]:
We can make this longer in pandas using the melt function
In [18]:
long_table = pd.melt(wide_table, id_vars = ['name'])
long_table
Out[18]:
We can make the table wider using the pivot method
side note - this kind of inconsistency between
meltandpivotis un-pythonic and should not be emulated
In [19]:
long_table.pivot(columns='variable')
Out[19]:
WHOA
One of the really cool things about pandas is that it allows you to have multiple indexes for rows and columns. Since pandas couldn't figure out what do with two kinds of value variables, it doubled up our column index. We can fix this by specifying that we only want the 'values' values
In [20]:
long_table.pivot(columns='variable', values='value')
Out[20]:
In [21]:
table['fingers'].mean()
Out[21]:
In [22]:
table['fingers'].std()
Out[22]:
In [23]:
table['fingers'].quantile(.25)
Out[23]:
In [24]:
table['fingers'].kurtosis()
Out[24]:
You can call several of these at once with the describe method
In [25]:
table.describe()
Out[25]:
In [7]:
from scipy import stats
data = pd.read_csv('../data/03_feedback.csv')
Using what you've learned so far about manipulating pandas objects, how would you find out the names of the variables in this dataset? Their datatypes? The distribution of their values?
A common statistical procedure is to look for differences between groups of values. Typically, the values are grouped by a variable of interest, like sex or age. Here, we are going to compare the barriers of access to technology that people experience in the D-Lab compared to the world outside.
If you only have two groups in your sample, you can use a t-test:
In [45]:
i = data['inside.barriers'].dropna()
o = data['outside.barriers'].dropna()
stats.ttest_ind(i, o)
Out[45]:
Notice that here, we are passing in two whole columns, but we could also be subsetting by some other factor.
If you have more than two groups (or levels) that you would like to compare, you'll have to use something like an ANOVA:
In [44]:
m = data[data.gender == "Male/Man"]['outside.barriers'].dropna()
f = data[data.gender == "Female/Woman"]['outside.barriers'].dropna()
q = data[data.gender == "Genderqueer/Gender non-conforming"]['outside.barriers'].dropna()
stats.f_oneway(m, f, q)
Out[44]:
Another common task is to establish if/how two variables are related across linear space. This could be something, for example, like relating shoe size to height. Here, we are going to ask whether barriers to access to technology inside and outside of the D-Lab are related.
One implementation of linear relationships is correlation testing:
In [53]:
intermediate = data.dropna(subset=['inside.barriers', 'outside.barriers'])
stats.pearsonr(intermediate['outside.barriers'], intermediate['inside.barriers'])
Out[53]:
At this point, we're going to pivot to using statsmodels
In [3]:
import statsmodels.formula.api as smf
The formulas module in statsmodels lets us work with pandas dataframes, and linear model specifications that are similar to R and other variants of statistical software, e.g.:
outcome ~ var1 + var2
In [8]:
model_1 = smf.ols("inside_barriers ~ outside_barriers", data=data).fit()
model_1
Out[8]:
To get a summary of the test results, call the model's summary method
In [9]:
model_1.summary()
Out[9]:
Since Python does not have private data or hidden attributes, you can pull out just about any intermediate information you want, including coefficients, residuals, and eigenvalues
Raymond Hettinger would say that Python is a "consenting adult language"
In [10]:
model_1.params['outside_barriers']
Out[10]:
statsmodels also exposes methods for validity checking your regressions, like looking for outliers by influence statistics
In [13]:
model_1.get_influence().summary_frame()
Out[13]:
If, at this stage, you suspect that one or more outliers is unduly influencing your model fit, you can transform your results into robust OLS with a method call:
In [15]:
model_1.get_robustcov_results().summary()
Out[15]:
This isn't very different, so we're probably okay.
If you want to add more predictors to your model, you can do so inside the function string:
In [16]:
smf.ols("inside_barriers ~ outside_barriers + gender", data=data).fit().summary()
Out[16]:
Note that our categorical/factor variable has been automatically one-hot encoded as treatment conditions. There's not way to change this within statsmodels, but you can specify your contrasts indirectly using a library called (Patsy)[http://statsmodels.sourceforge.net/stable/contrasts.html].
To add interactions to your model, you can use :, or * [for full factorial]
In [17]:
smf.ols("inside_barriers ~ outside_barriers * gender", data=data).fit().summary()
Out[17]:
In the time remaining, pull up a dataset that you have, and that you'd like to work with in Python. The instructors will be around to help you apply what you've learned today to problems in your data that you are dealing with.
If you don't have data of your own, you should practice with the test data we've given you here. For example, you could try to figure out: