Vocabulary and Matching

So far we've seen how a body of text is divided into tokens, and how individual tokens are parsed and tagged with parts of speech, dependencies and lemmas.

In this section we will identify and label specific phrases that match patterns we can define ourselves.

Rule-based Matching

spaCy offers a rule-matching tool called Matcher that allows you to build a library of token patterns, then match those patterns against a Doc object to return a list of found matches. You can match on any part of the token including text and annotations, and you can add multiple patterns to the same matcher.



In [1]:

    
# Perform standard imports
import spacy
nlp = spacy.load('en_core_web_sm')



In [2]:

    
# Import the Matcher library
from spacy.matcher import Matcher
matcher = Matcher(nlp.vocab)

Here `matcher` is an object that pairs to the current `Vocab` object. We can add and remove specific named matchers to `matcher` as needed.

Creating patterns

In literature, the phrase 'solar power' might appear as one word or two, with or without a hyphen. In this section we'll develop a matcher named 'SolarPower' that finds all three:



In [3]:

    
pattern1 = [{'LOWER': 'solarpower'}]
pattern2 = [{'LOWER': 'solar'}, {'LOWER': 'power'}]
pattern3 = [{'LOWER': 'solar'}, {'IS_PUNCT': True}, {'LOWER': 'power'}]

matcher.add('SolarPower', None, pattern1, pattern2, pattern3)

Let's break this down:

pattern1 looks for a single token whose lowercase text reads 'solarpower'
pattern2 looks for two adjacent tokens that read 'solar' and 'power' in that order
pattern3 looks for three adjacent tokens, with a middle token that can be any punctuation.*

\* Remember that single spaces are not tokenized, so they don't count as punctuation.
Once we define our patterns, we pass them into matcher with the name 'SolarPower', and set callbacks to None (more on callbacks later).

Applying the matcher to a Doc object



In [4]:

    
doc = nlp(u'The Solar Power industry continues to grow as demand \
for solarpower increases. Solar-power cars are gaining popularity.')



In [5]:

    
found_matches = matcher(doc)
print(found_matches)









    



[(8656102463236116519, 1, 3), (8656102463236116519, 10, 11), (8656102463236116519, 13, 16)]

matcher returns a list of tuples. Each tuple contains an ID for the match, with start & end tokens that map to the span doc[start:end]



In [6]:

    
for match_id, start, end in found_matches:
    string_id = nlp.vocab.strings[match_id]  # get string representation
    span = doc[start:end]                    # get the matched span
    print(match_id, string_id, start, end, span.text)









    



8656102463236116519 SolarPower 1 3 Solar Power
8656102463236116519 SolarPower 10 11 solarpower
8656102463236116519 SolarPower 13 16 Solar-power

The match_id is simply the hash value of the string_ID 'SolarPower'

Setting pattern options and quantifiers

You can make token rules optional by passing an 'OP':'*' argument. This lets us streamline our patterns list:



In [7]:

    
# Redefine the patterns:
pattern1 = [{'LOWER': 'solarpower'}]
pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'power'}]

# Remove the old patterns to avoid duplication:
matcher.remove('SolarPower')

# Add the new set of patterns to the 'SolarPower' matcher:
matcher.add('SolarPower', None, pattern1, pattern2)



In [8]:

    
found_matches = matcher(doc)
print(found_matches)









    



[(8656102463236116519, 1, 3), (8656102463236116519, 10, 11), (8656102463236116519, 13, 16)]

This found both two-word patterns, with and without the hyphen!

The following quantifiers can be passed to the 'OP' key:

OP	Description
\!	Negate the pattern, by requiring it to match exactly 0 times
?	Make the pattern optional, by allowing it to match 0 or 1 times
\+	Require the pattern to match 1 or more times
\*	Allow the pattern to match zero or more times

Be careful with lemmas!

If we wanted to match on both 'solar power' and 'solar powered', it might be tempting to look for the lemma of 'powered' and expect it to be 'power'. This is not always the case! The lemma of the adjective 'powered' is still 'powered':



In [9]:

    
pattern1 = [{'LOWER': 'solarpower'}]
pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LEMMA': 'power'}] # CHANGE THIS PATTERN

# Remove the old patterns to avoid duplication:
matcher.remove('SolarPower')

# Add the new set of patterns to the 'SolarPower' matcher:
matcher.add('SolarPower', None, pattern1, pattern2)



In [10]:

    
doc2 = nlp(u'Solar-powered energy runs solar-powered cars.')



In [11]:

    
found_matches = matcher(doc2)
print(found_matches)









    



[(8656102463236116519, 0, 3)]

The matcher found the first occurrence because the lemmatizer treated 'Solar-powered' as a verb, but not the second as it considered it an adjective.
For this case it may be better to set explicit token patterns.



In [12]:

    
pattern1 = [{'LOWER': 'solarpower'}]
pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'power'}]
pattern3 = [{'LOWER': 'solarpowered'}]
pattern4 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'powered'}]

# Remove the old patterns to avoid duplication:
matcher.remove('SolarPower')

# Add the new set of patterns to the 'SolarPower' matcher:
matcher.add('SolarPower', None, pattern1, pattern2, pattern3, pattern4)



In [13]:

    
found_matches = matcher(doc2)
print(found_matches)









    



[(8656102463236116519, 0, 3), (8656102463236116519, 5, 8)]

Other token attributes

Besides lemmas, there are a variety of token attributes we can use to determine matching rules:

Attribute	Description
`ORTH`	The exact verbatim text of a token
`LOWER`	The lowercase form of the token text
`LENGTH`	The length of the token text
`IS_ALPHA`, `IS_ASCII`, `IS_DIGIT`	Token text consists of alphanumeric characters, ASCII characters, digits
`IS_LOWER`, `IS_UPPER`, `IS_TITLE`	Token text is in lowercase, uppercase, titlecase
`IS_PUNCT`, `IS_SPACE`, `IS_STOP`	Token is punctuation, whitespace, stop word
`LIKE_NUM`, `LIKE_URL`, `LIKE_EMAIL`	Token text resembles a number, URL, email
`POS`, `TAG`, `DEP`, `LEMMA`, `SHAPE`	The token's simple and extended part-of-speech tag, dependency label, lemma, shape
`ENT_TYPE`	The token's entity label

Token wildcard

You can pass an empty dictionary {} as a wildcard to represent any token. For example, you might want to retrieve hashtags without knowing what might follow the # character:

[{'ORTH': '#'}, {}]

PhraseMatcher

In the above section we used token patterns to perform rule-based matching. An alternative - and often more efficient - method is to match on terminology lists. In this case we use PhraseMatcher to create a Doc object from a list of phrases, and pass that into matcher instead.



In [14]:

    
# Perform standard imports, reset nlp
import spacy
nlp = spacy.load('en_core_web_sm')



In [15]:

    
# Import the PhraseMatcher library
from spacy.matcher import PhraseMatcher
matcher = PhraseMatcher(nlp.vocab)

For this exercise we're going to import a Wikipedia article on Reaganomics
Source: https://en.wikipedia.org/wiki/Reaganomics



In [16]:

    
with open('../TextFiles/reaganomics.txt', encoding='utf8') as f:
    doc3 = nlp(f.read())



In [17]:

    
# First, create a list of match phrases:
phrase_list = ['voodoo economics', 'supply-side economics', 'trickle-down economics', 'free-market economics']

# Next, convert each phrase to a Doc object:
phrase_patterns = [nlp(text) for text in phrase_list]

# Pass each Doc object into matcher (note the use of the asterisk!):
matcher.add('VoodooEconomics', None, *phrase_patterns)

# Build a list of matches:
matches = matcher(doc3)



In [18]:

    
# (match_id, start, end)
matches









    Out[18]:





[(3473369816841043438, 41, 45),
 (3473369816841043438, 49, 53),
 (3473369816841043438, 54, 56),
 (3473369816841043438, 61, 65),
 (3473369816841043438, 673, 677),
 (3473369816841043438, 2985, 2989)]

The first four matches are where these terms are used in the definition of Reaganomics:



In [19]:

    
doc3[:70]









    Out[19]:





REAGANOMICS
https://en.wikipedia.org/wiki/Reaganomics

Reaganomics (a portmanteau of [Ronald] Reagan and economics attributed to Paul Harvey)[1] refers to the economic policies promoted by U.S. President Ronald Reagan during the 1980s. These policies are commonly associated with supply-side economics, referred to as trickle-down economics or voodoo economics by political opponents, and free-market economics by political advocates.

Viewing Matches

There are a few ways to fetch the text surrounding a match. The simplest is to grab a slice of tokens from the doc that is wider than the match:



In [20]:

    
doc3[665:685]  # Note that the fifth match starts at doc3[673]









    Out[20]:





same time he attracted a following from the supply-side economics movement, which formed in opposition to Keynesian



In [21]:

    
doc3[2975:2995]  # The sixth match starts at doc3[2985]









    Out[21]:





against institutions.[66] His policies became widely known as "trickle-down economics", due to the significant

Another way is to first apply the sentencizer to the Doc, then iterate through the sentences to the match point:



In [22]:

    
# Build a list of sentences
sents = [sent for sent in doc3.sents]

# In the next section we'll see that sentences contain start and end token values:
print(sents[0].start, sents[0].end)



In [23]:

    
# Iterate over the sentence list until the sentence end value exceeds a match start value:
for sent in sents:
    if matches[4][1] < sent.end:  # this is the fifth match, that starts at doc3[673]
        print(sent)
        break









    



At the same time he attracted a following from the supply-side economics movement, which formed in opposition to Keynesian demand-stimulus economics.

For additional information visit https://spacy.io/usage/linguistic-features#section-rule-based-matching