Info from the web

This notebook goes with a blog post at Agile*.

We're going to get some info from Wikipedia, and some financial prices from Yahoo Finance. We'll make good use of the requests library, a really nicely designed Python library for making web requests in Python.

Geological ages from Wikipedia

We'll start with the Jurassic, then generalize.



In [1]:

    
url = "http://en.wikipedia.org/wiki/Jurassic"  # Line 1

I used View Source in my browser to figure out where the age range is on the page, and what it looks like. The most predictable spot, that will work on every period's page, is in the infobox. It's given as a range, in italic text, with "million years ago" right after it.

Try to find the same string here.



In [2]:

    
import requests  # I don't count these lines.
r = requests.get(url)  # Line 2

Now we have the entire text of the webpage, along with some metadata. The text is stored in r.text, and I happen to know roughly where the relevant bit of text is: around the 7500th character, give or take:



In [3]:

    
r.text[7400:7600]  # I don't count these lines either.









    Out[3]:





'"firstHeading" lang="en">Jurassic</h1>\n\t\t\t\t\t\t\t\t\t<div id="bodyContent" class="mw-body-content">\n\t\t\t\t\t\t\t\t\t<div id="siteSub">From Wikipedia, the free encyclopedia</div>\n\t\t\t\t\t\t\t\t<div id="contentSub"></div'

We can get at that bit of text using a regular expression:



In [4]:

    
import re

s = re.search(r'<i>(.+?million years ago)</i>', r.text)
text = s.group(1)
text









    Out[4]:





'201.3–145&#160;million years ago'

And if we're really cunning, we can get the start and end ages:



In [5]:

    
start, end = re.search(r'<i>([\.0-9]+)–([\.0-9]+)&#160;million years ago</i>', r.text).groups()  # Line 3
duration = float(start) - float(end)  # Line 4

print("According to Wikipedia, the Jurassic lasted {:.2f} Ma.".format(duration))  # Line 5









    



According to Wikipedia, the Jurassic lasted 56.30 Ma.

An exercise for you, dear reader: Make a function to get the start and end ages of any geologic period, taking the name of the period as an argument. I have left some hints.



In [6]:

    
def get_age(period):
    url =  "http://en.wikipedia.org/wiki/" + period
    r = requests.get(url)
    start, end =   re.search(r'<i>([\.0-9]+)–([\.0-9]+)&#160;million years ago</i>', r.text).groups()
    return float(start), float(end)

You should be able to call your function like this:



In [7]:

    
period = "Jurassic"
get_age(period)









    Out[7]:





(201.3, 145.0)

Now we can make a function that makes the sentence we made before, calling the function you just wrote:



In [8]:

    
def duration(period):
    t0, t1 = get_age(period)
    duration = t0 - t1
    response = "According to Wikipedia, the {0} lasted {1:.2f} Ma.".format(period, duration)
    return response



In [9]:

    
duration('Cretaceous')









    Out[9]:





'According to Wikipedia, the Cretaceous lasted 79.00 Ma.'

Natural gas prices from Yahoo Finance

Here is an explanation of how to form Yahoo Finance queries.



In [10]:

    
import requests

url = "http://download.finance.yahoo.com/d/quotes.csv"
params = {'s': 'HHG17.NYM', 'f': 'l1'}

r = requests.get(url, params=params)



In [11]:

    
price = float(r.text)  # Line 9

print("Henry Hub price for Feb 2017: ${:.2f}".format(price))









    



Henry Hub price for Feb 2017: $3.38

The symbol s we're passing is HHF17.NYM. This means:

The ticker symbols we're passing look like XXMYY.NYM, with components as follows:

XX — commodity 'benchmark' symbol, as explained below.
M — a month code, symbolizing January to December: [F,G,H,J,K,M,N,Q,U,V,X,Z]
YY — a two-digit year, like 17.
.NYM — the Nymex symbol.

Benchmarks that seem to work with this service:

CL — West Texas Intermediate or WTI, light sweet crude
BB — Brent crude penultimate financial futures
BZ — Brent look-alike crude oil futures
MB — Gulf Coast Sour Crude
RE — Russian Export Blend Crude Oil (REBCO) futures

Gas spot prices that work:

NG — Henry Hub physical futures
HH — Henry Hub last day financial futures

Symbols that don't work:

DC — Dubai crude calendar futures
WCC — Canadian Heavy (differential, cf CL)
WCE — Western Canadian select crude oil futures (differential, cf CL) — but this seems to be the same price as WCC, which can't be right
LN — European options

As an exercise, write a function to get the futures price for a given benchmark, based on the contract price 90 days from 'now', whenever now is.



In [12]:

    
import time

def get_symbol(benchmark):
    
    future = time.gmtime(time.time() + 90*24*60*60)
    mo = future.tm_mon
    yr = future.tm_year
    month_codes = 'FGHJKMNQUVXZ'

    month = month_codes[mo]
    year = str(yr)[-2:]
    
    return benchmark + month + year + ".NYM"

This should work:



In [13]:

    
get_symbol('CL')









    Out[13]:





'CLU17.NYM'