This tutorial will walk you through the process of starting Clipper, creating and querying a Clipper application, and deploying models to Clipper. In the first part of the demo, you will set up Clipper and create an application without involving any machine learning, demonstrating how a frontend developer or dev-ops engineer can set up and query Clipper without having to know anything about the machine-learning models involved.
As an example, this tutorial will walk you through creating an application that labels images as either pictures of birds or planes. You will use the CIFAR-10 dataset as the source of these images.
As the first step in the tutorial, download the CIFAR dataset that your Clipper application will work with. You can do this by specifying a download location, cifar_loc, and running the below code. This will make use of the provided download_cifar.py.
This download can take some time. If it fails before you see the output "Finished downloading", go to the download location you specified, delete
cifar-10-python.tar.gz, and attempt the download again.
In [ ]:
cifar_loc = ""
%run ./download_cifar.py $cifar_loc
Now, we must extract the data into a format we can load. This will make use of the provided extract_cifar.py
This dataset has 50,000 training datapoints and 10,000 test datapoints. We don't need to use all of them to demonstrate how Clipper works. Feel free to adjust max_train_datapoints and max_test_datapoints or set them to None if you wish to use all the data available for training and testing. You can change these vaues and re-run this command in the future if you wish.
Using 10,000 training images (as opposed to the full 50,000 in the dataset) yields similar prediction accuracies and takes less time to extract into a readable format.
In [ ]:
max_train_datapoints = 10000
max_test_datapoints = 10000
%run ./extract_cifar.py $cifar_loc $max_train_datapoints $max_test_datapoints
The first step in building any application, using machine-learning or otherwise, is to understand the application requirements. Load the dataset into the notebook so you can examine it and better understand the dataset you will be working with. The cifar_utils library provides several utilities for working with CIFAR data – we will make use of one of them here.
In [ ]:
import cifar_utils
test_x, test_y = cifar_utils.filter_data(
*cifar_utils.load_cifar(cifar_loc, cifar_filename="cifar_test.data", norm=True))
no_norm_x, no_norm_y = cifar_utils.filter_data(
*cifar_utils.load_cifar(cifar_loc, cifar_filename="cifar_test.data", norm=False))
Take a look at the data you've loaded. The size and blurriness of these photos should give you a better understanding of the difficulty of the task you will ask of your machine learning models! If you'd like to see more images, increase the number of rows of images displayed -- the last argument to the function -- to a number greater than 2.
In [ ]:
%matplotlib inline
cifar_utils.show_example_images(no_norm_x, no_norm_y, 2)
Now you're ready to start Clipper! You will be using the clipper_admin client library to perform administrative commands.
Remember, Docker and Docker-Compose must be installed before deploying Clipper. Visit https://docs.docker.com/compose/install/ for instructions on how to do so. In addition, we recommend using Anaconda and Anaconda environments to manage Python.
Start by installing the library with pip:
pip install clipper_admin
Clipper uses Docker to manage application configurations and to deploy machine-learning models. Make sure your Docker daemon, local or remote, is up and running. You can check this by running docker ps in your command line – if your Docker daemon is not running, you will be told explicitly.
Starting Clipper will have the following effect on your setup:
If you'd like to deploy Clipper locally, you can leave the user and key variables blank and set host="localhost". Otherwise, you can deploy Clipper remotely to a machine that you have SSH access to. Set the user variable to your SSH username, the key variable to the path to your SSH key, and the host variable to the remote hostname or IP address.
If your SSH server is running on a non-standard port, you can specify the SSH port to use as another argument to the Clipper constructor. For example,
clipper = Clipper(host, user, key, ssh_port=9999).
In [ ]:
import sys
import os
from clipper_admin import Clipper
# Change the username if necessary
user = ""
# Set the path to the SSH key
key = ""
# Set the SSH host
host = ""
clipper = Clipper(host, user, key)
clipper.start()
Congratulations! You now have a running Clipper instance that you can start to interact with. Think of your clipper Python object as a vehicle for that interaction. Try using it to see the applications deployed to this Clipper instance:
In [ ]:
clipper.get_all_apps()
In order to query Clipper for predictions, you need to create an application. Each application specifies a name, a set of models it can query, the query input datatype, the selection policy, and a latency service level objective. Once you register an application with Clipper, the system will create two REST endpoints: one for requesting predictions and for providing feedback.
By associating the query interface with a specific application, Clipper allows frontend developers the flexibility to have multiple applications running in the same Clipper instance. Applications can request predictions from any model in Clipper. This allows a single Clipper instance to serve multiple machine-learning applications. It also provides a convenient mechanism for beta-testing or incremental rollout by creating experimental and stable applications for the same set of queries.
For this tutorial, you will create an application named "cifar_demo" and register a candidate model. Note that Clipper allows you to create the application before deploying the models. Clipper will be moving to a label-based model specification mechanism soon, so that in the future you won't have to explicitly enumerate all the models you want to query up front.
Registering the cifar_demo application with Clipper will have the following effect on your setup:
Don't worry if this command seems to take a long time. Before starting Clipper, the Docker containers must be downloaded from Docker Hub. These containers are fairly large and may take awhile to download depending on the speed of your internet connection.
In [ ]:
app_name = "cifar_demo"
model_name = "birds_vs_planes_classifier"
# If the model doesn't return a prediction in time, predict
# label 0 (bird) by default
default_output = "0"
clipper.register_application(
app_name,
model_name,
"doubles",
default_output,
slo_micros=20000)
Now when you list the applications registered with Clipper, you should see the newly registered "cifar_demo" application show up!
In [ ]:
clipper.get_all_apps(verbose=True)
Now that you have registered an application, you can start querying the application for predictions. In this case, Clipper has created two endpoints:
http://HOSTNAME:1337/cifar_demo/predict
http://HOSTNAME:1337/cifar_demo/updateYou will now start querying Clipper with a simple Python frontend app that computes the average accuracy of the responses after every 100 requests and updates a plot of the results with every iteration.
This diagram shows how the accuracy plot is receiving its test predictions:
In [ ]:
import seaborn as sns
sns.set_style("whitegrid")
sns.despine()
import matplotlib as mpl
%matplotlib notebook
cifar_utils.run_serving_workload(host, app_name, test_x, test_y)
Because you haven't deployed any models yet, Clipper will essentially be randomly guessing to respond to the predictions. Because you have deployed a binary classification model, random guessing should result in 50% accuracy, which is what the plot should be showing.
Now that you've created an application, it's time to move on to part 2 of the tutorial, where you will train and deploy some models to Clipper.