Getting Data from the the web

The world bank provides climate data via it's web

http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/var/year/iso3.ext

var is either tas or pr ext is usually CSV iso3 is the ISO standard 3 letter code for the country of interset (in capitals)


In [18]:
# Python requets Library lets us get data straight from a URL
import requests

In [19]:
url = "http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/GBR.csv"

response = requests.get(url)

if response.status_code != 200:
    print ('Failed to get data:', response.status_code)
else:
    print ('First 100 characters of the data are:')
    print ( response.text[:100])


First 100 characters of the data are:
year,data
1901,8.16360092163086
1902,7.798388481140137
1903,8.02857780456543
1904,8.073899269104004

Gettting Status codes for Guatemala (Country Code is GTM)

Fetch rainfall for Afghanistan between 1980 and 1999


In [20]:
url = "http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/GTM.csv"

response = requests.get(url)

if response.status_code != 200:
    print ('Failed to get data:', response.status_code)
else:
    print ('First 100 characters of the data are:')
    print ( response.text[:100])


First 100 characters of the data are:
year,data
1901,23.428869247436523
1902,23.555654525756836
1903,23.56220245361328
1904,23.44627952575

In [21]:
url = "http://climatedataapi.worldbank.org/climateweb/rest/v1/country/annualavg/pr/1980/1999/AFG.csv"

response = requests.get(url)

if response.status_code != 200:
    print ('Failed to get data:', response.status_code)
else:
    print ('First 100 characters of the data are:')
    print ( response.text[:100])


First 100 characters of the data are:
GCM,var,from_year,to_year,annual
bccr_bcm2_0,pr,1980,1999,429.92613067639326
cccma_cgcm3_1,pr,1980,1

In [22]:
# Create a csv file: test01.csv

1901,12.3
1902,45.6
1903,78.9


Out[22]:
(1903, 78.9)

In [23]:
with open ('test01.csv', 'r') as reader:
    for line in reader:
        print (len (line))


10
10
9

In [24]:
with open ('test01.csv', 'r') as reader:
    for line in reader:
        fields = line.split(',')
        print (fields)


['1901', '12.3\n']
['1902', '45.6\n']
['1903', '78.9']

In [25]:
# We need to get rid of the hidden newline \n 
with open ('test01.csv', 'r') as reader:
    for line in reader:
        fields = line.strip().split(',')
        print (fields)


['1901', '12.3']
['1902', '45.6']
['1903', '78.9']

Using CSV library instead


In [34]:
import csv

In [41]:
with open ('test01.csv', 'r') as rawdata:
    csvdata = csv.reader(rawdata)
    for record in csvdata:
        print (record)


['1901', '12.3']
['1902', '45.6']
['1903', '78.9']

In [38]:
url = "http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/pr/year/GBR.csv"

response = requests.get(url)

if response.status_code != 200:
    print ('Failed to get data:', response.status_code)
else:
    wrapper = csv.reader(response.text.strip().split('\n'))
    for record in wrapper:
        if record[0] != 'year':
            year = int(record [0])
            value = float(record [1])
            print (year, value)


1901 89.73978424072266
1902 84.22315216064453
1903 119.31846618652344
1904 91.66348266601562
1905 89.65657043457031
1906 98.95526885986328
1907 98.48038482666016
1908 95.18717956542969
1909 98.14665985107422
1910 103.52928161621094
1911 95.45286560058594
1912 108.9598617553711
1913 93.2777099609375
1914 102.29579162597656
1915 97.06973266601562
1916 107.87296295166016
1917 98.88864135742188
1918 105.67728424072266
1919 98.01356506347656
1920 105.2422103881836
1921 86.1358871459961
1922 96.65888214111328
1923 113.46175384521484
1924 109.6202392578125
1925 98.27181243896484
1926 102.01596069335938
1927 110.8193359375
1928 115.2984619140625
1929 100.12940216064453
1930 108.5672607421875
1931 102.66221618652344
1932 101.97649383544922
1933 79.8314437866211
1934 100.34272766113281
1935 107.33514404296875
1936 98.53144836425781
1937 94.9148178100586
1938 111.8836441040039
1939 98.04186248779297
1940 97.34461975097656
1941 86.44178771972656
1942 99.07189178466797
1943 100.81599426269531
1944 100.7071533203125
1945 93.96513366699219
1946 104.6628189086914
1947 91.6974868774414
1948 110.64335632324219
1949 99.34590911865234
1950 111.71575927734375
1951 109.48321533203125
1952 95.30955505371094
1953 88.76084899902344
1954 119.55448150634766
1955 83.76612091064453
1956 94.41340637207031
1957 102.37519836425781
1958 102.82244873046875
1959 94.30015563964844
1960 109.70860290527344
1961 102.67496490478516
1962 95.75684356689453
1963 91.66914367675781
1964 88.49595642089844
1965 103.84146881103516
1966 108.05892181396484
1967 111.24398040771484
1968 96.91545104980469
1969 93.35447692871094
1970 103.3067626953125
1971 87.93647766113281
1972 90.83573150634766
1973 89.29792022705078
1974 107.51863098144531
1975 86.90357971191406
1976 92.11167907714844
1977 100.82731628417969
1978 98.16151428222656
1979 109.2149429321289
1980 107.2969741821289
1981 108.48765563964844
1982 112.34332275390625
1983 100.25491333007812
1984 102.18620300292969
1985 104.05267333984375
1986 111.35047912597656
1987 99.15782165527344
1988 108.25444030761719
1989 100.03694915771484
1990 114.27284240722656
1991 95.51949310302734
1992 114.05003356933594
1993 105.5682373046875
1994 114.24977111816406
1995 100.28663635253906
1996 90.22846221923828
1997 96.02803039550781
1998 118.20884704589844
1999 118.4039306640625
2000 119.49745178222656
2001 96.44131469726562
2002 115.35612487792969
2003 84.73816680908203
2004 106.19056701660156
2005 100.51760864257812
2006 104.275634765625
2007 101.73286437988281
2008 116.0248794555664
2009 114.0855712890625
2010 90.66874694824219
2011 106.08039855957031
2012 118.43199157714844

In [ ]: