The purpose of this tutorial is to show you how to make a basic web application. Why are we making a web application in a data journalism class? That's a good question! Our reasons are two-fold:
Having said all this, I want to state right off the bat that the applications we'll develop in this tutorial work fine, but might not work at scale. The difficulty with web applications is that they have to work not just for one person at one time, but for potentially thousands of people over the course of years. That means that there are a whole category of issues that come into play when developing web applications that we haven't bothered to look at in this class, and won't be able to address in full during this tutorial.
A web application is a computer program running on a server somewhere that responds to requests from web browsers (and other kinds of web clients, as we'll see).
Usually, the purpose of a web application is to provide an easily-accessible public-facing interface for some kind of data. The simplest kind of "web application" is a "server" that simply responds to requests for particular files on a computer—and there was a period in the history of the web in which this was by far the most dominant form of web application, and the idea of "web development" consisted mostly of writing HTML files and uploading them to such a server. (If you've ever done web programming with PHP or the Apache web server, then you've experienced this kind of web development.) In a contemporary context, most web applications are programs that respond in sophisticated ways to browser requests by dynamically composing responses from various sources, such as databases and real-time data feeds.
But wait, what is meant by "requests" and "responses"? To answer that, let's back up a bit and answer the question...
Any computer on the Internet is capable of opening a network connection to another. For our purposes, you can think of these network connections as kind of like UNIX standard input and standard output: one computer (the "client") opens a connection to another computer (the "server"). The client sends bytes to the server, and the server responds with bytes of its own.
A single "server" can have multiple programs running on it that respond to network connections. Such programs distinguish themselves from one another by listening to network connections on different "ports." So for example, the program on a server that responds to network connections related to sending e-mail might listen for connections on port 25, while the program that responds to connections related to database services might listen on port 27017. In this chapter, we'll write programs that listen for network connections, and we'll need to select a port for the programs to listen to. We'll need to select a port that isn't already in use by another program.
Some port numbers are associated by convention with particular services. Notably, port 80 is considered the "default" port for communicating with server programs that serve web pages.
As explained above, the Internet allows a client computer to send bytes to a server computer, and get bytes in return. But two programs can't meaningfully communicate with one another unless they speak a common language---unless they have a set of rules in common that dictate what the data sent between them should look like and what that data "means." Over the history of the Internet, industrious individuals have taken it upon themselves to invent such languages. The official word for these languages is "protocol."
There are hundreds of protocols in use on the Internet. As you're reading this, your computer is likely using dozens of them simultaneously to communicate with other computers on the Internet (or between programs on your own computer). There's a protocol for sending mail, a protocol for looking up domain names, a protocol for peer-to-peer file sharing and more.
The protocol used by the web---i.e., the protocol that web browsers use to talk
to web servers---is called HTTP (HyperText Transfer Protocol). In many ways,
HTTP is a very straightforward protocol---so straightforward that it's possible
for us to write requests by hand, essentially doing the work on our own that
we would normally delegate to a web browser. To do this, we're going to use
a utility called netcat (typed nc). Netcat allows us to connect to arbitrary
ports on arbitrary servers and type in whatever text we want. Here's how to
use netcat to speak HTTP to the server where the schedule for Reading and
Writing Electronic Text resides:
$ nc rwet.decontextualize.com 80
GET /schedule.html HTTP/1.1
Host: rwet.decontextualize.com
What you just typed in is known as the HTTP request. It's the information
that the client sends to the server, prompting the server to respond. After you
type the line beginning with Host:, hit enter twice, and you should see some
output that looks like this:
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 20 Apr 2015 05:14:43 GMT
Content-Type: text/html
Content-Length: 15433
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Wed, 15 Apr 2015 19:13:26 GMT
ETag: "188600cb-3c49-513c8243e4180"
Accept-Ranges: bytes
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
[...more html follows...]
This is the HTTP response. This is what the server sends to the client in response to its request. Let's talk a bit about the specific structure of both response and request.
An HTTP request consists of three things:
I'll point out these parts using the request we just made with netcat.
GET /schedule.html HTTP/1.1
Host: rwet.decontextualize.com
In this request, GET is the method; /schedule.html is the path, and
HTTP/1.1 is the HTTP version. There's only one header in this request:
Host. You have to include a Host header for every HTTP/1.1 request, and the
header must be set to the domain name of the server that you're sending the
request to. This request, translated into English, means essentially this:
"Hey, web server! I want to get the document at the path /schedule.html on
your server. I'm speaking HTTP/1.1 (and not a newer or older version of the
protocol). In case you didn't know, the host that I'm making this request to is
rwet.decontextualize.com."
This request has no body. A body is sent with a request only when the request is doing something like submitting data from a form, or uploading a file. In fact, you can't send a body with a GET request at all; bodies are only supported for POST and PUT requests.
But wait, what are all of these "methods"? What do they mean? Why is there more than one? That's a good question! In HTTP, the method is a verb that specifies what the client wants the server to do with the document named in the path. There is a closed set of valid HTTP verbs, meaning you can't just make new ones up on the fly. How exactly the server interprets these verbs is specific to each web application, but in general the verbs have the following uses:
GET: Client wants the server to return the content of the resource named in the path.PUT: Client wants to modify the contents of the resource named in the path, or create a new resource with the given path.POST: Client wants to add to or modify the resource.DELETE: Client wants to remove the resource.HEAD: Client wants to check to see whether or not the resource exists, and maybe get metadata about the resource, without necessarily fetching the contents of the resource.OPTIONS: Client wants to know which of the above methods are supported for the given resource.As you can see, the meanings of these verbs overlap in strange ways (e.g., it's
not clear what the difference between POST and PUT is, since both are used
to add and modify resources). Every web service uses these verbs slightly
differently, and when you implement a web service, you need to make decisions
about what the verbs "mean." It's best to stick to convention, but there's
nothing stopping you from making your application accept GET requests that
delete resources, or POST requests that return resources instead of modifying
them.
Most web applications make use only of the verbs GET and POST. This is
because most web browsers only support those two methods (though with the
advent of HTML5, the situation is changing). Even today, if you're developing a
web application---especially one that's designed for browsers, and not
necessarily for automated agents---you're safe making your service only support
GET and POST.
The response we received from the web server in the netcat example above looked like this:
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 20 Apr 2015 05:14:43 GMT
Content-Type: text/html
Content-Length: 15433
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Wed, 15 Apr 2015 19:13:26 GMT
ETag: "188600cb-3c49-513c8243e4180"
Accept-Ranges: bytes
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
[...more html follows...]
This is the response. The response consists of the status line, a series of
one or more headers, and then the body of the response. The status line has the
HTTP version and a number that indicates the status of the request---this is
the so-called HTTP "status code." 200 means "okay, here's your data"; 404
means "I couldn't find the resource you wanted." There are dozens of other
codes with various meanings and conventional uses.
The headers tell us information about the response, such as how many bytes are in it (the Content-Length header), what kind of document it is (the Content-Type header), and when the document was last modified (the Last-Modified header). Some of the headers are required; some are optional. The client needs to know
this information so that it can properly download the data and display it to
the user.
After the headers, we see the response body. The response body is simply the data contained in the resource that the client requested. On the web, response bodies are usually HTML documents, but they can also be other things, like images, PDFs, MP3s, etc. Web APIs generally return JSON documents, not HTML.
We've already discussed in this class how to make web
requests.
You could just type in all your web requests by hand, but it's easier to have
an application (like a web browser, or curl) or a library (like urllib) do
it for you. These tools abstract away all the work of formatting the request
correctly, including the right headers, etc. so you can just get your work
done.
It's the same situation with writing a web server---a program that receives web requests and then respond to them. You could write a program that listens on the network for incoming connections, parses HTTP requests according to the HTTP standard, then sends responses in the correct format. But that would be a hassle! It turns out that enterprising programmers have already done that work for us---the work of making a program that listens for HTTP requests and responds to them. That work takes the form of a web framework---a kind of "skeleton" for a web server that has all the difficult stuff already taken care of.
The framework gives you access to everything in the HTTP request in a clean, abstrated way: the path of the request, the HTTP method, the body of the request, any headers, and the query string. Your job is to write rules and code that specify what should happen when requests are received. What should the response code be? What should the response body look like? And so forth.
Web frameworks also commonly supply functionality to make writing web applications easier, such as HTML template rendering and database connection abstraction.
There are a number of "frameworks" for Python. They all have benefits and drawbacks. Here are a few well-known frameworks:
Today we're going to use a framework called Flask. I chose Flask because it's easy to get started with, but has a clear path for making more advanced web applications when you're ready to take off the training wheels. There's also an active developer community producing helpful Flask extensions and extensive, beginner-friendly documentation.
In particular, we're going to use Flask to take the text manipulation and text generation projects that we've made so far and turn them into simple web applications. In the process, we're just going to scratch the surface of what's possible with Flask, but hopefully it'll be enough to whet your appetite and help you understand the basic concepts.
Install Flask with pip like so:
In [ ]:
!pip3 install Flask
You may want to create a virtual environment and install Flask within the virtual environment instead of installing it globally on your machine.
Programs you write with Flask are designed to be executed from the command line, so I want to show you how to do that first. Here's a very simple Flask application. Copy this code and paste it into a file named hello.py:
from flask import Flask
app = Flask(__name__)
@app.route("/howdy")
def hello():
return "Hello, world!"
app.run()
On the command line, run this program like so:
$ python3 hello.py
You'll see something that looks like this:
* Running on http://127.0.0.1:5000/howdy (Press CTRL+C to quit)
... and then: nothing. It looks like the program is hanging, but it’s actually just waiting—waiting for someone to make a web request. You can go to the following URL in your browser in order to make a request to your application:
http://localhost:5000/howdy
Or you can use curl to perform the same request:
In [11]:
!curl http://localhost:5000/howdy
It's also possible to break this web application. Try fetching any path except /howdy and see what happens:
In [13]:
!curl http://localhost:5000/florb
Hey, great! We've made a simple "Hello, world" application. When you're done basking in the glory of being a back-end Python web developer, you can hit Ctrl-C in the window where you ran the Flask application in order to stop the web server.
Let's go through this example in detail. The important thing to keep in mind is the purpose of the Flask framework, which is to make it possible for web requests to a particular path (i.e., the stuff that comes after the hostname in the URL) to invoke particular Python functions.
The "core" part of the program—the part that defines what it does—is the function called hello:
def hello():
return "Hello, world!"
This is a very simple function, taking no parameters and returning a string ("Hello, world!"). This is the only thing this particular web request knows how to do. The reason that Flask knows to execute this particular function and display its return value to the web browser is the line that precedes the function definition:
@app.route("/howdy")
This is a special "decorator" that tells
Flask to run this function whenever the application gets a request to the path
/howdy. The main work of writing a Flask application is defining functions
and then associating them with paths to run when a particular path is in the
HTTP request. You can write as many functions that respond to as many paths as
you want. Note that the name of the function itself doesn't matter! You could name it hello or mortimer or cheese_whiz; the only thing Flask cares about is the path in the route decorator.
"Decorator" is the real, actual name for that funny line beginning with
@. You don't need to understand decorators in great detail in order to use Flask---you can just think of them as a funny bit of syntax that you have to put before a function definition in order to let Flask know what the function is for and when it should be called. But if you're interested, here's a good tutorial about decorators, what they're good for, and how to make your own.
The first two lines of the program look like this:
from flask import Flask
app = Flask(__name__)
The first line imports the Flask class from the flask module; the second creates a variable app that contains a Flask "application" value, which is the Flask object that manages the entire web application process from beginning to end. (Don't worry about what the __name__ parameter means for now, just accept it as part of the set syntax for the call.) The final line of the program:
app.run()
... simply instructs the object to start listening for web requests, and never stop. (At least until someone hits Ctrl-C.)
For the purposes of teaching and taking notes, it's a bit inconvenient to save every Flask app to disk as an individual Python file and run it there. As a kind of shortcut, I'm going to write some of the examples in this tutorial as Jupyter Notebook cells. For example, here's the previous example written in a cell:
In [5]:
from flask import Flask
app = Flask(__name__)
@app.route("/howdy")
def hello():
return "Hello, world!"
app.run(port=5001)
You'll notice after running the example that the left-column indicator just keeps showing [*]. That means that the server is running! You can access it through a browser window to see what it looks like. (This link should work). (If you get [Errno 48] Address already in use, make sure you've stopped the version of the example program that was running in your terminal window!)
To stop the server, make sure you're in Command Mode in Jupyter Notebook and hit I. (Alternatively, hit the Stop button in the toolbar.) You'll be unable to execute any further cells until you've stopped the current one. (You may need to hit Stop or i multiple times.)
The remaining examples in this notebook will use this technique. But remember that eventually you'll want to have your web applications running as stand-alone programs on your computer (or on a server), and in that case you'll need to copy and paste them into their own .py files.
One of the primary ways that we can make our web applications responsive to user input is by using query strings. Quick refresher: a query string is the strange thing you see at the end of URLs sometimes that begins with a question mark and has ampersands and equal signs, like this:
http://example.com/foo?color=green&texture=bumpy
Query strings are an easily implemented, standard interface for providing parameters and variables to your web application. Here's a simple web application for reversing the letters of a word given on the query string:
In [23]:
from flask import Flask, request
app = Flask(__name__)
@app.route("/reverse")
def reverser():
word_str = request.args['word']
word_reversed = ''.join(reversed(list(word_str)))
return word_reversed
app.run()
In [ ]:
!ls templates
In [ ]:
from flask import Flask, request
app = Flask(__name__)
greets = ["Hello", "Hi", "Salutations", "Greetings", "Hey", "Sup"]
places = ["region", "continent", "world", "solar system", "galaxy"]
@app.route('/hello')
def hello():
greeting = random.choice(greets) + ", " + random.choice(places)
return "<html><h1>Welcome to Greet-O-Tron 2K15</h1>" + \
"<h2>Your Greeting Is...</h2>" + \
"<p>" + greeting + "</p>";
app.run()
The browser (or proxy) sent a request that this server could not understand.
Oh gross. It looks like the application is returning an awful error message. How we can we get around this and display something nicer? One possibility is to use the dictionary object's `get` method, which attempts to fetch the value of a key but resorts to a default if the key isn't found. Here's a version of this app with such a solution in place.
In [25]:
from flask import Flask, request
app = Flask(__name__)
@app.route("/reverse")
def reverser():
word_str = request.args.get('word', None)
if word_str:
word_reversed = ''.join(reversed(list(word_str)))
return word_reversed
else:
return "no word specified :("
app.run()
In [ ]:
Here's what happens when no word parameter is supplied now:
$ curl -s http://localhost:5000/reverse
no word specified :(
But it still works as expected with a word parameter:
$ curl -s http://localhost:5000/reverse?word=slipup
pupilsWhen we're developing our web application, we can turn on "debug" mode to see
exactly where in our code the error occured. To turn on debug mode, pass a
debug parameter to the application's run method with a value of True,
like so:
app.run(debug=True)
You can define variables outside of your handler functions in a Flask app and then use them inside of your handler functions. You can also import whatever other Python modules you want. Here's an example using the random library:
In [27]:
from flask import Flask
import random
app = Flask(__name__)
greets = ["Hello", "Hi", "Salutations", "Greetings", "Hey", "Sup"]
places = ["region", "continent", "world", "solar system", "galaxy"]
@app.route('/hello')
def hello():
greeting = random.choice(greets) + ", " + random.choice(places)
return greeting + "\n"
if __name__ == '__main__':
app.run()
This example creates two lists and assigns them to variables greets and places, then sets up a route so that requests to /hello run a function that returns the results of selecting a random item from two lists (using the random.choice() function). Sample output:
$ curl -s http://localhost:5000/hello
Salutations, solar systemSo far, all of our examples have had as their output simple strings of text.
This is fine for testing, but what we'd actually like is to have our web
application display a nice pretty web page. Anything that is in the return
value for the function that handled the request will go straight to the
browser, so one way to display HTML would be to simply put some HTML code in a
big string and return it:
In [28]:
from flask import Flask, request
app = Flask(__name__)
greets = ["Hello", "Hi", "Salutations", "Greetings", "Hey", "Sup"]
places = ["region", "continent", "world", "solar system", "galaxy"]
@app.route('/hello')
def hello():
greeting = random.choice(greets) + ", " + random.choice(places)
return "<html><h1>Welcome to Greet-O-Tron 2K15</h1>" + \
"<h2>Your Greeting Is...</h2>" + \
"<p>" + greeting + "</p>";
app.run()
This would work, but it's a little ungainly and inconvenient. For one, the HTML is inside the code, so you have to modify the Python file if you want to tweak the way that the application looks. It's often the case that separate members of the team developing a web applications will work on different parts: one person will work on the Python code, while another will work on the HTML front-end. For both of these reasons, it's convenient to separate out the HTML content in a separate file.
Flask (as with other web frameworks) offers this functionality in the form of templates. A template is basically an HTML file that has a bit of extra syntax in it able to include data from the Python application in the HTML output. Templates live in separate files, so they can be edited apart from the Python code.
Let's make a version of our greeting generator that uses templates. The Python code looks mostly the same, with a few additions, which I will discuss below. (Note: this cell won't actually work until you run the cells that follow!)
In [ ]:
from flask import Flask, render_template
import random
app = Flask(__name__)
greets = ["Hello", "Hi", "Salutations", "Greetings", "Hey", "Sup"]
places = ["region", "continent", "world", "solar system", "galaxy"]
@app.route('/hello')
def hello():
return render_template("greeting.html",
greet=random.choice(greets), place=random.choice(places))
if __name__ == '__main__':
app.run()
The template file itself needs to live in a separate directory called
templates. Make this directory like so:
In [29]:
!mkdir templates
... and then put the following code into a file called greeting.html:
<!doctype html>
<html>
<head>
<title>Greetings!</title>
<style type="text/css">
body {
max-width: 600px;
font-family: 'Courier New', Courier, monospace;
margin: 1em auto;
}
</style>
</head>
<body>
<h1>I am Greet-O-Tron 2K15. I have prepared for you this greeting.</h1>
<p>{{ greet }}, {{ place }}</p>
</body>
</html>Try running the program and then viewing it in a browser. You'll see something that looks like this:
As you can see, the HTML page is now displaying our output! Awesome. Here's how
it all works. Let's look at what's happening in our Python file first. Step one
is to include the render_template function from the Flask library:
from flask import Flask, render_template
Then, instead of returning a string literal in our handler function, we instead
call render_template. This function takes a filename as its first
argument, which specifies which template file to use. (It looks for template files in a subdirectory called templates.) You can then give the function any
number of keyword parameters.
The render_template function does this, essentially:
{{ name }} (where name can be any Python variable name) in the contents of the template, it performs a replacement, inserting instead the value for the keyword argument name as passed to the function.Simple variable replacement like this will get you a long way in web development. But Flask's templating engine, Jinja2, supports much more sophisticated functionality---worth checking out if you want to build a non-toy web application with Flask.
We now have a fully functional, web-accessible text generator. The next task I want to set before us is to make a web application that acts on user-supplied text. The easiest way to do this is with an HTML form. A form in HTML looks like this:
<form action="/path" method="POST">
<!-- include input elements here, such as... -->
<input type="text" name="foo" value="Type text here"><br>
<textarea name="bar" rows="10" cols="40">Type even more text</textarea><br>
<input type="submit" value="Submit!">
</form>
This HTML code will look like this in the browser:
The key to understanding HTML forms is this: they instruct the browser on how
to make a request to your application---specifically, how to send the
information in the form to your web application when the user clicks "Submit."
The attributes of the <form> tag are key: the action attribute indicates
what path the browser should make a request to; the method attribute
indicates what HTTP method the browser should use. When the user clicks
Submit, the information in the form will be sent to your application, with
the name attributes of each form element specifying the keys and whatever the
user types in for those fields as the values.
(You usually want to use POST for forms, but you can also use GET. The only
difference is that a POST request will put the parameters in the HTTP request
body, while a GET request will put them in the query string.)
To make an application that uses an HTML form, we need to have two handler
functions. One to display the page that shows the form, and another to respond
to the request that the browser will do when the user clicks Submit on the
form.
Here's a simple web application that does this. Whatever the user types in the form will be parsed into words; only words with fewer than six letters will remain in the output.
In [2]:
from flask import Flask, request, render_template
app = Flask(__name__)
@app.route('/')
def home():
return render_template("simplify_home.html")
@app.route('/transformed', methods=["POST"])
def transformed():
text = request.form['text']
words = [w for w in text.split() if len(w) <= 5]
return render_template("simplify_transformed.html", output=' '.join(words))
app.run()
Now create these two files in your templates directory. First, simplify_home.html:
<!doctype html>
<html>
<head>
<title>Simplify your text</title>
<style type="text/css">
body { max-width: 600px; margin: 1em auto; }
</style>
</head>
<body>
<h1>Simplify</h1>
<p>Are you sick of text that has too many long
words? Me too! Here's a tool to get rid of them.
Just Enter some text below and click Submit.</p>
<form action="/transformed" method="POST">
<textarea name="text" rows="24" cols="66"></textarea><br>
<input type="submit" value="Submit!">
</form>
</body>
</html>
... and then simplify_transformed.html:
<!doctype html>
<html>
<head>
<title>Your text, simplified</title>
<style type="text/css">
body { max-width: 600px; margin: 1em auto; }
</style>
</head>
<body>
<h1>Here's your text!</h1>
<p>{{ output }}</p>
</body>
</html>
If you go to your application's root url (http://localhost:5000/), you'll
see the following (I've already pasted in Obama's State of the Union speech for
2016):
Paste in some text and click 'Submit'. You'll get something back that looks like this (excerpted):
Mr. Mr. Vice of my marks the year that I’ve come here to on the State of the And for this final one, I’m going to try to make it I know some of you are antsy to get back to Iowa. I've been I'll be hands if you want some tips. Now I that it’s an for what we’ll this year are low. But Mr. I the that you and the other took at the end of last year to pass a and make tax cuts for So I hope we can work this year on some like and who are drug abuse and So who we might the But I want to go easy on the list of for the year Don’t I’ve got from learn to write code to for And I will keep for on the work that still needs to be done.
There are a lot of moving parts in this example. Let's go over the program in detail. Structurally, this application is different from some of the previously-discussed applications in that it has two routes, not just one:
@app.route('/')
def home():
return render_template("simplify_home.html")
@app.route('/transformed', methods=["POST"])
def transformed():
text = request.form['text']
words = [w for w in text.split() if len(w) <= 5]
return render_template("simplify_transformed.html", output=' '.join(words))
... meaning that this application will respond to requests at two paths: / (i.e., the root) and /transformed. The home() function's route is easy to understand: requests to / will simply render the "home" template (i.e., the template with the text area).
The transformed() route is a bit more interesting. In this case, the route decorator has an extra parameter: methods=["POST"]. This extra parameter means that this particular route will work with only the POST HTTP method (see the discussion about HTTP methods earlier in this tutorial for more information on POST). If a web request arrives using the POST method to the path /transformed, the code inside transformed() will be run.
We're using the request object again to get extra information from the request. In the string-reversing application above, we used the .args attribute of the request object to read parameters passed in on the query string. In this case, because the request was submitted with POST using a form, we use the .form attribute of the request object. The .form attribute also behaves like a dictionary. The value for each key in the dictionary corresponds to whatever the user put into the HTML form widget with the corresponding name. Let's look at the <form> tag in simplify_home.html again:
<form action="/transformed" method="POST">
<textarea name="text" rows="24" cols="66"></textarea><br>
<input type="submit" value="Submit!">
</form>
The idea of an HTML form is that the user fills in some information, and then the web browser uses that information to make a web request. In this way, you can think of the <form> tag as a weird kind of hyperlink. The <form> tag has two important attributes: action, whose value specifies what URL the web request should be sent to, and method, which specifies which HTTP method to use. (This is almost always POST or GET.) Inside the <form> tag, you can use various HTML elements, including <textarea> and <input>, to add widgets to the form, like free-form text inputs, sliders, checkboxes, radio buttons, etc. (Here's the full list of native widgets and their functionality.).
The special <input type="submit"> widget creates a button; when you click the button, the web browser makes the request as specified by the form and the values that the user has filled in. You can actually use Chrome Developer Tools to see what this request looks like, as the browser sent it. Open Developer Tools, select the "Network" tab and reload the page after submitting the form, then select the only item in the left-hand panel. You'll see something that looks like this:
This shows exactly the data that the browser sent to our web application! Most of this data is filled in automatically by the browser for every request. But the "Request Header" section shows that the method (POST) and path (/transformed) are as we supplied them in the form; the "Form Data" (or request body) section shows the data from the form, as it was included in the request. (Forms sent via POST by default use a special form of encoding called application/x-www-form-urlencoded.)
Back in our program, Flask runs the transformed() function in response to this request. Flask automatically parses the form data in the request for us, making it available in the request.form dictionary (as discussed above). We use the value for the text key to perform some creative text transformation, and then render it in the template. Voila, an interactive web application!
EXERCISE: Add a field to the form so that the user can adjust the maximum length of words to be included in the respose.
Now that we know how to make a simple web application, let's try wiring it up to a PostgreSQL database. We're going to make a simple web application that allows users to view query data from the MONDIAL database and return the results as HTML. Before you proceed, make sure you've set up a local installation of PostgreSQL and imported the MONDIAL database as specified in a previous tutorial.
The first thing we're going to do is to make a simple web app that simply spits out a record from the database in response to a user's request. In this case, we'll ask the user for the name of a country and return information about the given country (if it's found in the database).
We're going to do a little trick where we only use one template instead of two, and use a bit of logic in the template to change the layout of the page based on where the user is in the query process. Use this template (save as country_lookup.html):
<!doctype html>
<html>
<head>
<title>Country Information</title>
<style type="text/css">
body { max-width: 600px; margin: 1em auto; }
</style>
</head>
<body>
<h1>Country Information</h1>
<p>This page uses data from MONDIAL to inform you about a country.</p>
{% if country %}
<p>You asked for information on {{ country['name'] }}. This country has
{{ country['population'] }} people and its capital is {{ country['capital'] }}.</p>
{% endif %}
<form action="/" method="GET">
Enter the name of a country here: <input type="text" name="lookup_name">
<input type="submit" value="Look up!">
</form>
<p>{{ output }}</p>
</body>
</html>
Now run the following:
In [11]:
from flask import Flask, request
import pg8000
app = Flask(__name__)
# you may need to include extra authorization details here...
conn = pg8000.connect(database="mondial")
@app.route("/")
def lookup():
country_info = None
if 'lookup_name' in request.args:
cursor = conn.cursor()
cursor.execute("SELECT name, capital, population FROM country WHERE name = %s",
[request.args['lookup_name']])
response = cursor.fetchone()
if response:
country_info = {
'name': response[0],
'capital': response[1],
'population': response[2]
}
return render_template('country_lookup.html', country=country_info)
app.run()
Point your browser at http://localhost:5000/. You should see a form that allows you to input the name of a country. If you put a name in and click "Submit," the page will show you information for that country.
This example shows how to connect to PostgreSQL and issue a query based on user-submitted information. We establish one database connection when the program begins, which will be used for all web requests over the duration of the program. In the lookup() function, the only thing the program is guaranteed to do is render the country_lookup template. That template has an interesting new bit of syntax in it:
{% if country %}
<p>You asked for information on {{ country['name'] }}. This country has
{{ country['population'] }} people and its capital is {{ country['capital'] }}.</p>
{% endif %}
The {% if ... %}/{% endif %} construction is specific to Jinja2, the library that Flask uses for its HTML templates. Anything between {% if ... %} and {% endif %} will be displayed in the output only if the condition next to if evaluates to True. In this case, we've exploited this fact to allow us to use the template for two different states of the application. (Those states are: (1) the user has not yet asked for information on a country and (2) the user has requested information and the application is displaying the results.)
If in lookup() the query string has a parameter lookup_name, then we assume the form has been submitted and we perform the database query by creating a cursor, executing the query and retrieving the results, stashing the retrieved data in a dictionary. This dictionary is then passed to the template, which checks to see if the dictionary is present (see above) and displays the data if so.
The Jinja2 template library has many other little bits of helpful syntax, including for loops. See the Jinja2 Template Designer Guide for more information.
EXERCISE #1: Add an error message to the template if the country that the user requested was not found.
EXERCISE #2: Write a Python script that submits requests to this web application and "scrapes" the results.
A web-based API is just a very specialized form of web application. The web applications we've written so far all return HTML, designed to be legible to humans, and expect the user to interact with them through form submissions. A web API, on the other hand, will tend to return data in a format designed to be legible to machines, and expect the user to interact with the data through a set of pre-defined rules.
In the following example, I've created a very simple web API that serves the data from the country table of the MONDIAL database. The way it works is by querying the database and then building a list of dictionaries from the results, then returning that list of dictionaries in JSON format (instead of HTML). Here's what it looks like:
In [16]:
from flask import Flask, jsonify
import pg8000
app = Flask(__name__)
conn = pg8000.connect(database="mondial")
@app.route("/countries")
def get_countries():
cursor = conn.cursor()
cursor.execute("SELECT name, capital, area, population FROM country ORDER BY name")
output = []
for item in cursor.fetchall():
output.append({'name': item[0],
'capital': item[1],
'area': int(item[2]),
'population': int(item[3])})
return jsonify(output)
app.run()
If you go to http://localhost:5000/countries, you should see a JSON response that looks like this:
[
{
"area": 647500,
"capital": "Kabul",
"name": "Afghanistan",
"population": 26023100
},
{
"area": 28750,
"capital": "Tirana",
"name": "Albania",
"population": 2800138
},
{
"area": 2381740,
"capital": "Algiers",
"name": "Algeria",
"population": 37062820
},
[...many countries omitted for brevity...]
{
"area": 390580,
"capital": "Harare",
"name": "Zimbabwe",
"population": 13061239
}
]
This is a JSON response, just like the one returned from fancy-shmancy APIs like the Spotify API. You could write a Python program to read this response and do computation with it. Pretty neat!
Flask provides a jsonify function (which must be imported separately) which converts its parameter to JSON and also sets various HTTP headers so that the response is returned as JSON and will be correctly interpreted as JSON by potential clients. Note that I had to explicitly convert some of the fields using int(); the numerical values returned from the database are technically Numeric objects, which (for whatever reason) can't automatically be converted to JSON by the jsonify function.
On its own, the API in the previous example isn't much more useful than just e-mailing someone a JSON dump. It might be a bit more helpful to allow the user to specify a subset of the data, rather than retrieve the whole thing. In the following example, the user can specify a parameter population_gt on the query string which, if supplied, will limit the records returned in the response to only those countries with populations greater than the given value.
In [19]:
from flask import Flask, jsonify, request
import pg8000
app = Flask(__name__)
conn = pg8000.connect(database="mondial")
@app.route("/countries")
def get_countries():
cursor = conn.cursor()
pop_gt = int(request.args.get('population_gt', 0))
cursor.execute(
"""SELECT name, capital, area, population
FROM country
WHERE population >= %s
ORDER BY name""",
[pop_gt])
output = []
for item in cursor.fetchall():
output.append({'name': item[0],
'capital': item[1],
'area': int(item[2]),
'population': int(item[3])})
return jsonify(output)
app.run()
A request to, e.g., http://localhost:5000/countries?population_gt=250000000, will return results like this:
[
{
"area": 9596960,
"capital": "Beijing",
"name": "China",
"population": 1360720000
},
{
"area": 3287590,
"capital": "Delhi",
"name": "India",
"population": 1210854977
},
{
"area": 1919440,
"capital": "Jakarta",
"name": "Indonesia",
"population": 252124458
},
{
"area": 9372610,
"capital": "Washington",
"name": "United States",
"population": 318857056
}
]
You're well on your way to being a backend web engineer. Great work!