Homework 6: Web Applications

For this homework, you're going to write a web API for the lake data in the MONDIAL database. (Make sure you've imported the data as originally outlined in our week 1 tutorial.)

The API should perform the following tasks:

  • A request to /lakes should return a JSON list of dictionaries, with the information from the name, elevation, area and type fields from the lake table in MONDIAL.
  • The API should recognize the query string parameter sort. When left blank or set to name, the results should be sorted by the name of the lake (in alphabetical order). When set to area or elevation, the results should be sorted by the requested field, in descending order.
  • The API should recognize the query string parameter type. When specified, the results should only include rows that have the specified value in the type field.
  • You should be able to use both the sort and type parameters in any request.

This notebook contains only test requests to your API. Write the API as a standalone Python program, start the program and then run the code in the cells below to ensure that your API produces the expected output. When you're done, paste the source code in the final cell (so we can check your work, if needed).

Hints when writing your API code:

  • You'll need to construct the SQL query as a string, piece by piece. This will likely involve a somewhat messy tangle of if statements. Lean into the messy tangle.
  • Make sure to use parameter placeholders (%s) in the query.
  • If you're getting SQL errors, print out your SQL statement in the request handler function so you can debug it. (When you use print() in Flask, the results will display in your terminal window.)
  • When in doubt, return to the test code. Examine it carefully and make sure you know exactly what it's trying to do.

Problem set #1: A list of lakes

Your API should return a JSON list of dictionaries (objects). Use the code below to determine what the keys of the dictionaries should be. (For brevity, this example only prints out the first ten records, but of course your API should return all of them.)

Expected output:

143 lakes
Ammersee - elevation: 533 m / area: 46 km^2 / type: None
Arresoe - elevation: None m / area: 40 km^2 / type: None
Atlin Lake - elevation: 668 m / area: 798 km^2 / type: None
Balaton - elevation: 104 m / area: 594 km^2 / type: None
Barrage de Mbakaou - elevation: None m / area: None km^2 / type: dam
Bodensee - elevation: 395 m / area: 538 km^2 / type: None
Brienzersee - elevation: 564 m / area: 29 km^2 / type: None
Caspian Sea - elevation: -28 m / area: 386400 km^2 / type: salt
Chad Lake - elevation: 250 m / area: 23000 km^2 / type: salt
Chew Bahir - elevation: 520 m / area: 800 km^2 / type: salt

In [5]:
!pip3 install Flask


Requirement already satisfied (use --upgrade to upgrade): Flask in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): click>=2.0 in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages (from Flask)
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages (from Flask)
Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.21 in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages (from Flask)
Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages (from Flask)
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages (from Jinja2>=2.4->Flask)

In [6]:
!curl http://localhost:5000/florb


curl: (7) Failed to connect to localhost port 5000: Connection refused

In [7]:
import requests
data = requests.get('http://localhost:5000/lakes').json()
print(len(data), "lakes")
for item in data[:10]:
    print(item['name'], "- elevation:", item['elevation'], "m / area:", item['area'], "km^2 / type:", item['type'])


---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connection.py in _new_conn(self)
    141             conn = connection.create_connection(
--> 142                 (self.host, self.port), self.timeout, **extra_kw)
    143 

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     90     if err is not None:
---> 91         raise err
     92 

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     80                 sock.bind(source_address)
---> 81             sock.connect(sa)
     82             return sock

ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

NewConnectionError                        Traceback (most recent call last)
/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, **response_kw)
    577                                                   body=body, headers=headers,
--> 578                                                   chunked=chunked)
    579 

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    361         else:
--> 362             conn.request(method, url, **httplib_request_kw)
    363 

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py in request(self, method, url, body, headers)
   1082         """Send a complete request to the server."""
-> 1083         self._send_request(method, url, body, headers)
   1084 

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py in _send_request(self, method, url, body, headers)
   1127             body = body.encode('iso-8859-1')
-> 1128         self.endheaders(body)
   1129 

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py in endheaders(self, message_body)
   1078             raise CannotSendHeader()
-> 1079         self._send_output(message_body)
   1080 

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py in _send_output(self, message_body)
    910 
--> 911         self.send(msg)
    912         if message_body is not None:

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py in send(self, data)
    853             if self.auto_open:
--> 854                 self.connect()
    855             else:

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connection.py in connect(self)
    166     def connect(self):
--> 167         conn = self._new_conn()
    168         self._prepare_conn(conn)

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connection.py in _new_conn(self)
    150             raise NewConnectionError(
--> 151                 self, "Failed to establish a new connection: %s" % e)
    152 

NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x10f63eb70>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    402                     retries=self.max_retries,
--> 403                     timeout=timeout
    404                 )

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, **response_kw)
    622             retries = retries.increment(method, url, error=e, _pool=self,
--> 623                                         _stacktrace=sys.exc_info()[2])
    624             retries.sleep()

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    280         if new_retry.is_exhausted():
--> 281             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    282 

MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /lakes (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10f63eb70>: Failed to establish a new connection: [Errno 61] Connection refused',))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
<ipython-input-7-70cfd1dcfd7a> in <module>()
      1 import requests
----> 2 data = requests.get('http://localhost:5000/lakes').json()
      3 print(len(data), "lakes")
      4 for item in data[:10]:
      5     print(item['name'], "- elevation:", item['elevation'], "m / area:", item['area'], "km^2 / type:", item['type'])

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/api.py in get(url, params, **kwargs)
     69 
     70     kwargs.setdefault('allow_redirects', True)
---> 71     return request('get', url, params=params, **kwargs)
     72 
     73 

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/api.py in request(method, url, **kwargs)
     55     # cases, and look like a memory leak in others.
     56     with sessions.Session() as session:
---> 57         return session.request(method=method, url=url, **kwargs)
     58 
     59 

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    473         }
    474         send_kwargs.update(settings)
--> 475         resp = self.send(prep, **send_kwargs)
    476 
    477         return resp

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/sessions.py in send(self, request, **kwargs)
    583 
    584         # Send the request
--> 585         r = adapter.send(request, **kwargs)
    586 
    587         # Total elapsed time of the request (approximately)

/Users/skkandrach/.virtualenvs/lede/lib/python3.5/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    465                 raise ProxyError(e, request=request)
    466 
--> 467             raise ConnectionError(e, request=request)
    468 
    469         except ClosedPoolError as e:

ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /lakes (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10f63eb70>: Failed to establish a new connection: [Errno 61] Connection refused',))

In [ ]:
!pip install requests

Problem set #2: Lakes of a certain type

The following code fetches all lakes of type salt and finds their average area and elevation.

Expected output:

average area: 18880
average elevation: 970

In [ ]:
import requests
data = requests.get('http://localhost:5000/lakes?type=salt').json()
avg_area = sum([x['area'] for x in data if x['area'] is not None]) / len(data)
avg_elev = sum([x['elevation'] for x in data if x['elevation'] is not None]) / len(data)
print("average area:", int(avg_area))
print("average elevation:", int(avg_elev))

In [ ]:
conn.rollback()

Problem set #3: Lakes in order

The following code fetches lakes in reverse order by their elevation and prints out the name of the first fifteen, excluding lakes with an empty elevation field.

Expected output:

* Licancabur Crater Lake
* Nam Co
* Lago Junin
* Lake Titicaca
* Poopo
* Salar de Uyuni
* Koli Sarez
* Lake Irazu
* Qinghai Lake
* Segara Anak
* Lake Tahoe
* Crater Lake
* Lake Tana
* Lake Van
* Issyk-Kul

In [ ]:
import requests
data = requests.get('http://localhost:5000/lakes?sort=elevation').json()
for item in [x['name'] for x in data if x['elevation'] is not None][:15]:
    print("*", item)

Problem set #4: Order and type

The following code prints the names of the largest caldera lakes, ordered in reverse order by area.

Expected output:

* Lake Nyos
* Lake Toba
* Lago Trasimeno
* Lago di Bolsena
* Lago di Bracciano
* Crater Lake
* Segara Anak
* Laacher Maar

In [ ]:
import requests
data = requests.get('http://localhost:5000/lakes?sort=area&type=caldera').json()
for item in data:
    print("*", item['name'])

Problem set #5: Error handling

Your API should work fine even when faced with potential error-causing inputs. For example, the expected output for this statement is an empty list ([]), not every row in the table.


In [ ]:
import requests
data = requests.get('http://localhost:5000/lakes', params={'type': "' OR true; --"}).json()
data

Specifying a field other than name, area or elevation for the sort parameter should fail silently, defaulting to sorting alphabetically. Expected output: ['Ammersee', 'Arresoe', 'Atlin Lake', 'Balaton', 'Barrage de Mbakaou']


In [ ]:
import requests
data = requests.get('http://localhost:5000/lakes', params={'sort': "florb"}).json()
[x['name'] for x in data[:5]]

Paste your code

Please paste the code for your entire Flask application in the cell below, in case we want to take a look when grading or debugging your assignment.


In [ ]:
# paste code here

from flask import Flask, request, jsonify
import pg8000
import decimal

app= Flask(__name__)


@app.route("/lakes")
def get_lakes():
    conn= pg8000.connect(database="mondial", user="skkandrach")
    cursor= conn.cursor()
    type_lakes = request.args.get('type', '')
    sort_lakes= request.args.get('sort', '')


    if (type_lakes != '') and (sort_lakes != '') and (sort_lakes != 'name'):
        cursor.execute("SELECT name, elevation, area, type FROM lake WHERE type = %s ORDER BY " + sort_lakes + " DESC", [type_lakes])

    elif type_lakes != '':
        cursor.execute("SELECT name, elevation, area, type FROM lake WHERE type = %s ORDER BY name", [type_lakes])

    elif sort_lakes not in ['name', 'elevation', 'area']:
        cursor.execute("SELECT name, elevation, area, type FROM lake ORDER BY name" )

    elif sort_lakes != '':
        cursor.execute("SELECT name, elevation, area, type FROM lake ORDER BY " + sort_lakes + " DESC")

    else:
        cursor.execute("SELECT name, elevation, area, type FROM lake ORDER BY name")




    def decimal_to_int(x):
        if isinstance(x, decimal.Decimal):
            return int(x)
        else:
            return None

    whole_dict = []
    for item in cursor.fetchall():
        whole_dict.append({'name': item[0], 'elevation': decimal_to_int(item[1]), 'area': decimal_to_int(item[2]), 'type': item[3]})
    return jsonify(everything_dict)

app.run(debug=True)

In [ ]: