Regexs

Up until now, to search in text we have used string methods find, startswith, endswith, etc. But sometimes you need more power.

Regular expressions are their own little language that allows you to search through text and find matches with incredibly complex patterns.

A regular expression, also referred to as "regex" or "regexp", provides a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters.

To use regular you need to import python's regex library re https://docs.python.org/2/library/re.html

Searching

The simplest thing you can do with regexs in python is search through text to see if there is a match. To do this you use the methods search or match. match only checks if it matches at the beginning of the string and search check the whole string.

re.match(pattern, string)  
re.search(pattern, string) 

TRY IT

Search for the word May in the django logs

Special Characters

So far we can't do anything that you couldn't do with find, but don't worry. Regexs have many special characters to allow you to look for thing like the beginning of a word, whitespace or classes of characters.

You include the character in the pattern.

• ^ Matches the beginning of a line
• $ Matches the end of the line
• . Matches any character
• \s Matches whitespace
• \S Matches any non-whitespace character
• * Repeats a character zero or more times
• *? Repeats a character zero or more times (non-greedy)
• + Repeats a character one or more times
• +? Repeats a character one or more times (non-greedy)
• ? Repeats a character 0 or one time
• [aeiou] Matches a single character in the listed set
• [^XYZ] Matches a single character not in the listed set
• [a-z0-9] The set of characters can include a range
• {10} Specifics a match the preceding character(s) {num} number or times
• \d Matches any digit
• \b Matches a word boundary

Hint if you want to match the literal character (like $) as opposed to its special meaning, you would escape it with a \

TRY IT

Match anything between angled brackets < >

Ignoring case

match and search both take an optional third argument that allows you to include flags. The most common flag is ignore case.

re.search(pattern, string, re.IGNORECASE)
re.match(pattern, string, re.IGNORECASE)

TRY IT

search for 'django' in 'Both Django and Flask are very useful python frameworks' ignoring case

Extracting Matches

Finding is only half the battle. You can also extract what you match.

To get the string that your regex matched you can store the match object in a variable and run the group method on that

m = re.search(pattern, string)
print m.group(0)

If you want to find all the matches, not just the first, you can use the findall method. It returns a list of all the matches

re.findall(pattern, string)

If you want to have only part of the match returned to you in findall, you can use parenthesis to set a capture point

pattern = 'sads (part to capture) asdjklajsd'
print re.findall(pattern, string) # prints part to capture

TRY IT

Capture the host of the email address (alphanumerics between @ and .com) Hint remember to escape the . in .com

Practice

There is a lot more that you can do, but it can feel overwhelming. The best way to learn is with practice. A great way to experiment is this website http://www.regexr.com/ You can put a section of text and see what regexs match patterns in your text. The site also has a cheatsheet for special characters.

Project: Doc Clerk

Let's imagine you are working in a law office. You have millions of e-mails and other documents to go through to see what is relevant to the case. You are going to write a program to go though a file, check for key words (client's name, phone number, defendant's name) and print out the whole paragraph. It should not print any paragraphs with no relevant info. Paragraphs will be separated by an empty line.

Your program should match the following items: Gold E. Locks (case insensitive, E. or E) Three bears or 3 bears 571 209-4000 (with parens, dashes, or no spaces)

1. Import re
2. Initialize a variable called paragraph to be an empty list and a variable called found_match to false.
3. Create a list of patterns to match and store in variable called patterns
4. Read in test file 'evidence.txt'.
5. For line in evidence:

    a. check if it matches any of the patterns, if so set found_match to true
    b. append line to paragraph
    c. if line is empty (just a newline character) 
        - print paragraph if found_match is true **Hint** use the join method to print a string instead of a list
        - reset paragraph to empty list and found_match to false