Prerequisites

pip install clipper_admin==0.4.1 \
            keras==2.2.4 \
            tensorflow==1.13.1 \
            numpy==1.16.4 \
            pandas==0.24.2 \
            pillow==6.0.0

Load a pretrained Keras model(ResNet50)


In [1]:
from keras.applications.resnet50 import ResNet50


Using TensorFlow backend.

In [2]:
# https://keras.io/applications/#classify-imagenet-classes-with-resnet50
model = ResNet50(weights='imagenet')


WARNING:tensorflow:From /Users/user/anaconda3/envs/clipper_test/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.

Initialize the Clipper cluster


In [3]:
from clipper_admin import ClipperConnection, DockerContainerManager

In [4]:
clipper_conn = ClipperConnection(DockerContainerManager())

In [5]:
clipper_conn.start_clipper(cache_size=1)  # Disable PredictionCache


19-06-09:00:01:18 INFO     [docker_container_manager.py:184] [default-cluster] Starting managed Redis instance in Docker
19-06-09:00:01:21 INFO     [docker_container_manager.py:276] [default-cluster] Metric Configuration Saved at /private/var/folders/3q/f41dn1j54z72yf8bs2_80l1r0000gn/T/tmpexo7e9ch.yml
19-06-09:00:01:21 INFO     [clipper_admin.py:162] [default-cluster] Clipper is running

In [6]:
!docker ps -a


CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                  PORTS                                            NAMES
cdffdb15fcf7        prom/prometheus:v2.9.2              "/bin/prometheus --c…"   1 second ago        Up Less than a second   0.0.0.0:9090->9090/tcp                           metric_frontend-96730
c0efd9245cfa        clipper/frontend-exporter:0.4.1     "python /usr/src/app…"   2 seconds ago       Up 1 second                                                              query_frontend_exporter-14432
6906442dee21        clipper/query_frontend:0.4.1        "/clipper/query_fron…"   3 seconds ago       Up 1 second             0.0.0.0:1337->1337/tcp, 0.0.0.0:7000->7000/tcp   query_frontend-14432
f9db4f85dbbd        clipper/management_frontend:0.4.1   "/clipper/mgmt_front…"   4 seconds ago       Up 2 seconds            0.0.0.0:1338->1338/tcp                           mgmt_frontend-1363
ffae60aa1362        redis:alpine                        "docker-entrypoint.s…"   4 seconds ago       Up 3 seconds            0.0.0.0:6379->6379/tcp                           redis-96320

Define 'predict' function


In [7]:
import io
import numpy as np
from PIL import Image
from keras.preprocessing.image import img_to_array
from keras.applications.resnet50 import preprocess_input, decode_predictions
    
def predict(model, inputs):
    def _predict_one(one_input_arr):
        try:
            image = Image.open(io.BytesIO(one_input_arr))
            if image.mode != "RGB":
                image = image.convert("RGB")
            image = image.resize((224, 224))
            image = img_to_array(image)
            image = np.expand_dims(image, axis=0)
            image = preprocess_input(image)
            return decode_predictions(preds=model.predict(image), top=3)[0]
        except Exception as e:
            print(e)
            return []
        
    return [_predict_one(i) for i in inputs]

Deploy Keras model and 'predict' function to the Clipper cluster


In [8]:
import clipper_admin.deployers.keras as keras_deployer

In [9]:
app_name = 'keras-test-app'
model_name = 'keras-test-model'

In [10]:
keras_deployer.deploy_keras_model(clipper_conn=clipper_conn,
                                  name=model_name,
                                  version='1',
                                  input_type='bytes',
                                  func=predict,
                                  model_path_or_object=model,
                                  num_replicas=1,
                                  batch_size=1,  # Disable adaptive batching policy
                                  pkgs_to_install=['pillow'])


19-06-09:00:01:22 INFO     [deployer_utils.py:41] Saving function to /var/folders/3q/f41dn1j54z72yf8bs2_80l1r0000gn/T/tmpfjihn7f7clipper
19-06-09:00:01:22 INFO     [deployer_utils.py:51] Serialized and supplied predict function
19-06-09:00:01:33 INFO     [keras.py:221] Using Python 3.6 base image
19-06-09:00:01:33 INFO     [clipper_admin.py:534] [default-cluster] Building model Docker image with model data from /var/folders/3q/f41dn1j54z72yf8bs2_80l1r0000gn/T/tmpfjihn7f7clipper
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster] Step 1/3 : FROM clipper/keras36-container:0.4.1
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster]  ---> 5410d028b48e
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster] Step 2/3 : RUN apt-get -y install build-essential && pip install pillow
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster]  ---> Using cache
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster]  ---> fcc57ca889cc
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster] Step 3/3 : COPY /var/folders/3q/f41dn1j54z72yf8bs2_80l1r0000gn/T/tmpfjihn7f7clipper /model/
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster]  ---> 1a36e69f6239
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster] Successfully built 1a36e69f6239
19-06-09:00:01:35 INFO     [clipper_admin.py:539] [default-cluster] Successfully tagged default-cluster-keras-test-model:1
19-06-09:00:01:35 INFO     [clipper_admin.py:541] [default-cluster] Pushing model Docker image to default-cluster-keras-test-model:1
19-06-09:00:01:38 INFO     [docker_container_manager.py:409] [default-cluster] Found 0 replicas for keras-test-model:1. Adding 1
19-06-09:00:01:39 INFO     [clipper_admin.py:724] [default-cluster] Successfully registered model keras-test-model:1
19-06-09:00:01:39 INFO     [clipper_admin.py:642] [default-cluster] Done deploying model keras-test-model:1.

In [11]:
clipper_conn.register_application(name=app_name,
                                  input_type="bytes",
                                  default_output="-1.0",
                                  slo_micros=10000000)  # 10s


19-06-09:00:01:39 INFO     [clipper_admin.py:236] [default-cluster] Application keras-test-app was successfully registered

In [12]:
clipper_conn.link_model_to_app(app_name=app_name,
                               model_name=model_name)


19-06-09:00:01:39 INFO     [clipper_admin.py:303] [default-cluster] Model keras-test-model is now linked to application keras-test-app

In [13]:
import time
time.sleep(30)

Download sample images


In [14]:
!wget https://harishnarayanan.org/images/writing/artistic-style-transfer/output_13_0.png -O elephant.jpg


--2019-06-09 00:02:09--  https://harishnarayanan.org/images/writing/artistic-style-transfer/output_13_0.png
Resolving harishnarayanan.org (harishnarayanan.org)... 46.101.13.93
Connecting to harishnarayanan.org (harishnarayanan.org)|46.101.13.93|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 111477 (109K) [image/png]
Saving to: ‘elephant.jpg’

elephant.jpg        100%[===================>] 108.86K   211KB/s    in 0.5s    

2019-06-09 00:02:11 (211 KB/s) - ‘elephant.jpg’ saved [111477/111477]


In [15]:
!wget http://kikei.github.io/images/plots/2018-08-05-rabbit2.jpg -O rabbit.jpg


--2019-06-09 00:02:11--  http://kikei.github.io/images/plots/2018-08-05-rabbit2.jpg
Resolving kikei.github.io (kikei.github.io)... 185.199.108.153, 185.199.110.153, 185.199.111.153, ...
Connecting to kikei.github.io (kikei.github.io)|185.199.108.153|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22976 (22K) [image/jpeg]
Saving to: ‘rabbit.jpg’

rabbit.jpg          100%[===================>]  22.44K  --.-KB/s    in 0.09s   

2019-06-09 00:02:11 (257 KB/s) - ‘rabbit.jpg’ saved [22976/22976]


In [16]:
from IPython.display import display
from PIL import Image

In [17]:
display(Image.open('elephant.jpg'))



In [18]:
display(Image.open('rabbit.jpg'))


Send some requests to the Clipper cluster


In [19]:
import json
import base64
import requests
from datetime import datetime
from keras.preprocessing import image

In [20]:
headers = {'Content-type': 'application/json'}
url = "http://{addr}/{app_name}/predict".format(
    addr=clipper_conn.get_query_addr(),
    app_name=app_name)

First request, which is so slow due to downloading a file from https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json.


In [21]:
start = datetime.now()
req_json = json.dumps({ "input": base64.b64encode(open('elephant.jpg', "rb").read()).decode() })
r = requests.post(url, headers=headers, data=req_json)
end = datetime.now()

In [22]:
latency = (end - start).total_seconds() * 1000.0
print("'%s', %f ms" % (r.text, latency))


'{"query_id":0,"output":"[('n02504458', 'African_elephant', 0.65733755), ('n01871265', 'tusker', 0.14728831), ('n02504013', 'Indian_elephant', 0.13633117)]","default":false}', 1571.285000 ms

Second request, which is moderate!


In [23]:
start = datetime.now()
req_json = json.dumps({ "input": base64.b64encode(open('rabbit.jpg', "rb").read()).decode() })
r = requests.post(url, headers=headers, data=req_json)
end = datetime.now()

In [24]:
latency = (end - start).total_seconds() * 1000.0
print("'%s', %f ms" % (r.text, latency))


'{"query_id":1,"output":"[('n02328150', 'Angora', 0.6669809), ('n02326432', 'hare', 0.2204211), ('n02325366', 'wood_rabbit', 0.093360566)]","default":false}', 475.778000 ms

Clean-up


In [25]:
clipper_conn.stop_all()


19-06-09:00:02:57 INFO     [clipper_admin.py:1424] [default-cluster] Stopped all Clipper cluster and all model containers

In [26]:
!docker rm -f $(docker ps -a -q) && docker image prune -f


82b99a3ebef2
cdffdb15fcf7
c0efd9245cfa
6906442dee21
f9db4f85dbbd
ffae60aa1362
Deleted Images:
deleted: sha256:ef861b780188c804738b677ed1513e37ee8cf1c841030bd63bb5544490511b08
deleted: sha256:762e306dffb80768e7c1fce6d6c957a9e8fbf2f77cef1e2294c6ad7b9b213319

Total reclaimed space: 103MB