In [1]:
# import required modules
import requests
from bs4 import BeautifulSoup
from datetime import datetime
import time
import re
import sys
In [8]:
# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp')
# read the content of the server’s response
src = req.text
In [9]:
# parse the response into an HTML tree
soup = BeautifulSoup(src, 'lxml')
# take a look
print(soup.prettify()[:1000])
BeautifulSoup has a number of functions to find things on a page. Like other webscraping tools, Beautiful Soup lets you find elements by their:
Let's search first for HTML tags.
The function find_all searches the soup tree to find all the elements with an a particular HTML tag, and returns all of those elements.
What does the example below do?
In [10]:
# find all elements in a certain tag
# these two lines of code are equivilant
# soup.find_all("a")
NB: Because find_all() is the most popular method in the Beautiful Soup search API, you can use a shortcut for it. If you treat the BeautifulSoup object as though it were a function, then it’s the same as calling find_all() on that object.
These two lines of code are equivalent:
In [11]:
# soup.find_all("a")
# soup("a")
That's a lot! Many elements on a page will have the same html tag. For instance, if you search for everything with the a tag, you're likely to get a lot of stuff, much of which you don't want. What if we wanted to search for HTML tags ONLY with certain attributes, like particular CSS classes?
We can do this by adding an additional argument to the find_all
In the example below, we are finding all the a tags, and then filtering those with class = "sidemenu".
In [12]:
# Get only the 'a' tags in 'sidemenu' class
soup("a", class_="sidemenu")
Out[12]:
Oftentimes a more efficient way to search and find things on a website is by CSS selector. For this we have to use a different method, select(). Just pass a string into the .select() to get all elements with that string as a valid CSS selector.
In the example above, we can use "a.sidemenu" as a CSS selector, which returns all a tags with class sidemenu.
In [13]:
# get elements with "a.sidemenu" CSS Selector.
soup.select("a.sidemenu")
Out[13]:
In [16]:
# SOLUTION
soup.select("a.mainmenu")
Out[16]:
In [8]:
# this is a list
soup.select("a.sidemenu")
# we first want to get an individual tag object
first_link = soup.select("a.sidemenu")[0]
# check out its class
type(first_link)
Out[8]:
It's a tag! Which means it has a text member:
In [9]:
print(first_link.text)
Sometimes we want the value of certain attributes. This is particularly relevant for a tags, or links, where the href attribute tells us where the link goes.
You can access a tag’s attributes by treating the tag like a dictionary:
In [10]:
print(first_link['href'])
In [21]:
# SOLUTION
[link['href'] for link in soup.select("a.mainmenu")]
Out[21]:
Believe it or not, that's all you need to scrape a website. Let's apply these skills to scrape http://www.ilga.gov/senate/default.asp?GA=98
NB: we're just going to scrape the 98th general assembly.
Our goal is to scrape information on each senator, including their:
- name
- district
- party
In [12]:
# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')
# read the content of the server’s response
src = req.text
# soup it
soup = BeautifulSoup(src, "lxml")
In [13]:
# get all tr elements
rows = soup.find_all("tr")
len(rows)
Out[13]:
But remember, find_all gets all the elements with the tr tag. We can use smart CSS selectors to get only the rows we want.
In [14]:
# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')
print(rows[2].prettify())
We can use the select method on anything. Let's say we want to find everything with the CSS selector td.detail in an item of the list we created above.
In [15]:
# select only those 'td' tags with class 'detail'
row = rows[2]
detailCells = row.select('td.detail')
detailCells
Out[15]:
Most of the time, we're interested in the actual text of a website, not its tags. Remember, to get the text of an HTML element, use the text member.
In [16]:
# Keep only the text in each of those cells
rowData = [cell.text for cell in detailCells]
Now we can combine the beautifulsoup tools with our basic python skills to scrape an entire web page.
In [17]:
# check em out
print(rowData[0]) # Name
print(rowData[3]) # district
print(rowData[4]) # party
In [18]:
# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')
# read the content of the server’s response
src = req.text
# soup it
soup = BeautifulSoup(src, "lxml")
# Create empty list to store our data
members = []
# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')
# loop through all rows
for row in rows:
# select only those 'td' tags with class 'detail'
detailCells = row.select('td.detail')
# get rid of junk rows
if len(detailCells) is not 5:
continue
# Keep only the text in each of those cells
rowData = [cell.text for cell in detailCells]
# Collect information
name = rowData[0]
district = int(rowData[3])
party = rowData[4]
# Store in a tuple
tup = (name,district,party)
# Append to list
members.append(tup)
In [19]:
len(members)
Out[19]:
The code above retrieves information on:
- the senator's name
- their district number
- and their party
We now want to retrieve the URL for each senator's list of bills. The format for the list of bills for a given senator is:
http://www.ilga.gov/senate/SenatorBills.asp + ? + GA=98 + &MemberID=memberID + &Primary=True
to get something like:
http://www.ilga.gov/senate/SenatorBills.asp?MemberID=1911&GA=98&Primary=True
You should be able to see that, unfortunately, memberID is not currently something pulled out in our scraping code.
Your initial task is to modify the code above so that we also retrieve the full URL which points to the corresponding page of primary-sponsored bills, for each member, and return it along with their name, district, and party.
Tips:
<a>) in each legislator's row of the table. You can again use the .select() method on the row object in the loop to do this — similar to the command that finds all of the td.detail cells in the row. Remember that we only want the link to the legislator's bills, not the committees or the legislator's profile page.<a href="/senate/Senator.asp/...">Bills</a>. The string in the href attribute contains the relative link we are after. You can access an attribute of a BeatifulSoup Tag object the same way you access a Python dictionary: anchor['attributeName']. (See the documentation for more details).
In [20]:
# SOLUTION
# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')
# read the content of the server’s response
src = req.text
# soup it
soup = BeautifulSoup(src, "lxml")
# Create empty list to store our data
members = []
# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')
# loop through all rows
for row in rows:
# select only those 'td' tags with class 'detail'
detailCells = row.select('td.detail')
# get rid of junk rows
if len(detailCells) is not 5:
continue
# Keep only the text in each of those cells
rowData = [cell.text for cell in detailCells]
# Collect information
name = rowData[0]
district = int(rowData[3])
party = rowData[4]
# add href
href = row.select('a')[1]['href']
# add full path
full_path = "http://www.ilga.gov/senate/" + href + "&Primary=True"
# Store in a tuple
tup = (name,district,party, full_path)
# Append to list
members.append(tup)
In [21]:
members[:5]
Out[21]:
In [22]:
# SOLUTION
def get_members(url):
src = requests.get(url).text
soup = BeautifulSoup(src, "lxml")
rows = soup.select('tr')
members = []
for row in rows:
detailCells = row.select('td.detail')
if len(detailCells) is not 5:
continue
rowData = [cell.text for cell in detailCells]
name = rowData[0]
district = int(rowData[3])
party = rowData[4]
href = row.select('a')[1]['href']
full_path = "http://www.ilga.gov/senate/" + href + "&Primary=True"
tup = (name,district,party,full_path)
members.append(tup)
return(members)
In [23]:
# Test you code!
senateMembers = get_members('http://www.ilga.gov/senate/default.asp?GA=98')
len(senateMembers)
Out[23]:
Now we want to scrape the webpages corresponding to bills sponsored by each bills.
Write a function called get_bills(url) to parse a given Bills URL. This will involve:
BeautifulSoup library to find all of the <td> elements with the class billlistI've started the function for you. Fill in the rest.
In [25]:
# SOLUTION
def get_bills(url):
src = requests.get(url).text
soup = BeautifulSoup(src, "lxml")
rows = soup.select('tr tr tr')
bills = []
rowData = []
for row in rows:
detailCells = row.select('td.billlist')
if len(detailCells) is not 5:
continue
rowData = [cell.text for cell in row]
bill_id = rowData[0]
description = rowData[2]
champber = rowData[3]
last_action = rowData[4]
last_action_date = rowData[5]
tup = (bill_id,description,champber,last_action,last_action_date)
bills.append(tup)
return(bills)
In [30]:
# uncomment to test your code:
test_url = senateMembers[0][3]
get_bills(test_url)[0:5]
Out[30]:
Finally, create a dictionary bills_dict which maps a district number (the key) onto a list_of_bills (the value) eminating from that district. You can do this by looping over all of the senate members in members_dict and calling get_bills() for each of their associated bill URLs.
NOTE: please call the function time.sleep(0.5) for each iteration of the loop, so that we don't destroy the state's web site.
In [ ]:
# SOLUTION
bills_dict = {}
for member in senateMembers[:5]:
bills_dict[member[1]] = get_bills(member[3])
time.sleep(0.5)
In [ ]:
bills_dict[52]
In [ ]: