These problem sets focus on using the Beautiful Soup library to scrape web pages.
I've made a web page for you to scrape. It's available here. The page concerns the catalog of a famous widget company. You'll be answering several questions about this web page. In the cell below, I've written some code so that you end up with a variable called html_str that contains the HTML source code of the page, and a variable document that stores a Beautiful Soup object.
In [1]:
from bs4 import BeautifulSoup
from urllib.request import urlopen
html_str = urlopen("http://static.decontextualize.com/widgets2016.html").read()
document = BeautifulSoup(html_str, "html.parser")#python读取html里所有代码，用beautifulsoup
Now, in the cell below, use Beautiful Soup to write an expression that evaluates to the number of <h3> tags contained in widgets2016.html.
In [2]:
h3_tag=document.find_all('h3')
len(h3_tag)
Out[2]:
Now, in the cell below, write an expression or series of statements that displays the telephone number beneath the "Widget Catalog" header.
In [3]:
a_tag=document.find('a', attrs={'class':'tel'})
print(a_tag.string)
In the cell below, use Beautiful Soup to write some code that prints the names of all the widgets on the page. After your code has executed, widget_names should evaluate to a list that looks like this (though not necessarily in this order):
Skinner Widget
Widget For Furtiveness
Widget For Strawman
Jittery Widget
Silver Widget
Divided Widget
Manicurist Widget
Infinite Widget
Yellow-Tipped Widget
Unshakable Widget
Self-Knowledge Widget
Widget For Cinema
In [4]:
td_tags = document.find_all('td', attrs={'class':'wname'})
for widget_name in td_tags:
print(widget_name.string)
For this problem set, we'll continue to use the HTML page from the previous problem set. In the cell below, I've made an empty list and assigned it to a variable called widgets. Write code that populates this list with dictionaries, one dictionary per widget in the source file. The keys of each dictionary should be partno, wname, price, and quantity, and the value for each of the keys should be the value for the corresponding column for each row. After executing the cell, your list should look something like this:
[{'partno': 'C1-9476',
'price': '$2.70',
'quantity': u'512',
'wname': 'Skinner Widget'},
{'partno': 'JDJ-32/V',
'price': '$9.36',
'quantity': '967',
'wname': u'Widget For Furtiveness'},
...several items omitted...
{'partno': '5B-941/F',
'price': '$13.26',
'quantity': '919',
'wname': 'Widget For Cinema'}]And this expression:
widgets[5]['partno']
... should evaluate to:
LH-74/O
In [5]:
widgets = []
winfo_info = document.find_all('tr', attrs={'class':'winfo'})
for winfo in winfo_info:#从winfo_info的list里提取winfo这个dictionary, dictionary才有value
partno_tag=winfo.find('td', attrs={'class':'partno'})
price_tag=winfo.find('td',attrs={'class':'price'})
quantity_tag=winfo.find('td',attrs={'class':'quantity'})
wname_tag=winfo.find('td', attrs={'class':'wname'})
widgets_dictionary = {}
widgets_dictionary['partno'] = partno_tag.string
widgets_dictionary['price'] = price_tag.string
widgets_dictionary['quantity'] = quantity_tag.string
widgets_dictionary['wname'] = wname_tag.string
widgets.append(widgets_dictionary)
widgets
Out[5]:
In [6]:
widgets = []
winfo_info = document.find_all('tr', attrs={'class':'winfo'})
for winfo in winfo_info:
partno_tag=winfo.find('td', attrs={'class':'partno'})
price_tag=winfo.find('td',attrs={'class':'price'})
quantity_tag=winfo.find('td',attrs={'class':'quantity'})
wname_tag=winfo.find('td', attrs={'class':'wname'})
for price in price_tag:
price=float(price[1:])
for quantity in quantity_tag:
quantity=int(quantity)
widgets_dictionary = {}
widgets_dictionary['partno'] = partno_tag.string
widgets_dictionary['price'] = price
widgets_dictionary['quantity'] = quantity
widgets_dictionary['wname'] = wname_tag.string
widgets.append(widgets_dictionary)
widgets
Out[6]:
Great! I hope you're having fun. In the cell below, write an expression or series of statements that uses the widgets list created in the cell above to calculate the total number of widgets that the factory has in its warehouse.
Expected output: 7928
In [7]:
#sum() 需要是一个list []
total_widgets = []
for widget in widgets:
total_widgets.append(widget['quantity'])#.append 把widget['quality']里的每一个值加到total_widgets这个list里
sum(total_widgets)
Out[7]:
In the cell below, write some Python code that prints the names of widgets whose price is above $9.30.
Expected output:
Widget For Furtiveness
Jittery Widget
Silver Widget
Infinite Widget
Widget For Cinema
In [8]:
for widget in widgets:
if widget['price'] > 9.30:
print(widget['wname'])
In the following problem set, you will yet again be working with the data in widgets2016.html. In order to accomplish the tasks in this problem set, you'll need to learn about Beautiful Soup's .find_next_sibling() method. Here's some information about that method, cribbed from the notes:
Often, the tags we're looking for don't have a distinguishing characteristic, like a class attribute, that allows us to find them using .find() and .find_all(), and the tags also aren't in a parent-child relationship. This can be tricky! For example, take the following HTML snippet, (which I've assigned to a string called example_html):
In [9]:
example_html = """
<h2>Camembert</h2>
<p>A soft cheese made in the Camembert region of France.</p>
<h2>Cheddar</h2>
<p>A yellow cheese made in the Cheddar region of... France, probably, idk whatevs.</p>
"""
If our task was to create a dictionary that maps the name of the cheese to the description that follows in the <p> tag directly afterward, we'd be out of luck. Fortunately, Beautiful Soup has a .find_next_sibling() method, which allows us to search for the next tag that is a sibling of the tag you're calling it on (i.e., the two tags share a parent), that also matches particular criteria. So, for example, to accomplish the task outlined above:
In [10]:
example_doc = BeautifulSoup(example_html, "html.parser")
cheese_dict = {}
for h2_tag in example_doc.find_all('h2'):
cheese_name = h2_tag.string
cheese_desc_tag = h2_tag.find_next_sibling('p')
cheese_dict[cheese_name] = cheese_desc_tag.string
cheese_dict
Out[10]:
With that knowledge in mind, let's go back to our widgets. In the cell below, write code that uses Beautiful Soup, and in particular the .find_next_sibling() method, to print the part numbers of the widgets that are in the table just beneath the header "Hallowed Widgets."
Expected output:
MZ-556/B
QV-730
T1-9731
5B-941/F
In [14]:
# TA-COMMENT: This code doesn't do what we want it to! I'm going to comment out what you had.
# h3_tag = document.find_all('h3')
# for item in h3_tag:
# if item.string == "Hallowed Widgets":
# partno_tag = item.find_next_sibling('table', {'class':'widgetlist'})
# td_tag = hallowed_widgets_table.find_all('td', {'class':'partno'})
# for partno in td_tag:
# print(partno.string)
h3_tag = document.find_all('h3')
for item in h3_tag:
if item.string == "Hallowed widgets": # TA-COMMENT: You had capitalized "widgets" -- you were not getting
# anything that fit your condition when it was == "Hallowed widgets"! Always print to learn more
partno_tag = item.find_next_sibling('table', {'class':'widgetlist'})
# TA-COMMENT: When I ran the code you had, it said that "hallowed_widgets_table" was not defined
# And if you look, you don't define it anywhere but use it in "td_tag"
# You can't do that... you should have used:
td_tag = partno_tag.find_all('td', {'class':'partno'})
for partno in td_tag:
print(partno.string)
# TA-COMMENT: Let me know if you have any questions!
Okay, now, the final task. If you can accomplish this, you are truly an expert web scraper. I'll have little web scraper certificates made up and I'll give you one, if you manage to do this thing. And I know you can do it!
In the cell below, I've created a variable category_counts and assigned to it an empty dictionary. Write code to populate this dictionary so that its keys are "categories" of widgets (e.g., the contents of the <h3> tags on the page: "Forensic Widgets", "Mood widgets", "Hallowed Widgets") and the value for each key is the number of widgets that occur in that category. I.e., after your code has been executed, the dictionary category_counts should look like this:
{'Forensic Widgets': 3,
'Hallowed widgets': 4,
'Mood widgets': 2,
'Wondrous widgets': 3}
In [12]:
category_counts = {}
# your code here
# end your code
category_counts
Out[12]:
Congratulations! You're done.
In [ ]:
In [ ]: