Lessons learned in web scraping



In [1]:

    
import requests

Making requests

A head request is a lighter and faster way of checking if a url is serviceable, a given file exists, etc.. LastModified and ContentLength are useful.



In [2]:

    
req = requests.head('http://www.google.com')
print(req.headers['Content-Length'])



In [3]:

    
req = requests.get('http://www.google.com')
print(req.headers['Content-Length'])

Show benchmarks



In [ ]: