Deploying Various MNIST Models on Kubernetes

Using:

kubeflow
seldon-core

Follow the main README to setup kubeflow and seldon-core. This notebook will show various rolling deployments of the trained models

Single model
AB Test between 2 models
Multi-Armed Bandit over 3 models

Dependencies

Tensorflow
grpcio package

Setup

Set kubectl to use the namespace where you installed kubeflow and seldon. In the README it is kubeflow.



In [ ]:

    
!kubectl config set-context $(kubectl config current-context) --namespace=kubeflow



In [ ]:

    
!make create_protos



In [ ]:

    
!python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto



In [ ]:

    
%matplotlib inline
import utils
from visualizer import get_graph
mnist = utils.download_mnist()

Ensure you have port forwarded the ambassador reverse proxy

kubectl port-forward $(kubectl get pods -n kubeflow -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n kubeflow 8002:80

Deploy Single Tensorflow Model



In [ ]:

    
get_graph("../k8s_serving/serving_model.json",'r')



In [ ]:

    
!pygmentize ../k8s_serving/serving_model.json



In [ ]:

    
!kubectl apply -f ../k8s_serving/serving_model.json



In [ ]:

    
!kubectl get seldondeployments mnist-classifier -o jsonpath='{.status}'



In [ ]:

    
utils.predict_rest_mnist(mnist)



In [ ]:

    
utils.predict_grpc_mnist(mnist)

Start load test



In [ ]:

    
!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust



In [ ]:

    
!helm install seldon-core-loadtesting --name loadtest  \
    --namespace kubeflow \
    --repo https://storage.googleapis.com/seldon-charts \
    --set locust.script=mnist_rest_locust.py \
    --set locust.host=http://mnist-classifier:8000 \
    --set oauth.enabled=false \
    --set oauth.key=oauth-key \
    --set oauth.secret=oauth-secret \
    --set locust.hatchRate=1 \
    --set locust.clients=1 \
    --set loadtest.sendFeedback=1 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=1 \
    --set data.size=784

Rolling update to AB Test

Run an AB Test between 2 models:

Tensorflow neural network model
Scikit-learn random forest.



In [ ]:

    
get_graph("../k8s_serving/ab_test_sklearn_tensorflow.json",'r')



In [ ]:

    
!pygmentize ../k8s_serving/ab_test_sklearn_tensorflow.json



In [ ]:

    
!kubectl apply -f ../k8s_serving/ab_test_sklearn_tensorflow.json



In [ ]:

    
!kubectl get seldondeployments mnist-classifier -o jsonpath='{.status}'



In [ ]:

    
utils.predict_rest_mnist(mnist)



In [ ]:

    
utils.evaluate_abtest(mnist,100)

Rolling Update to Multi-Armed Bandit

Run a epsilon-greey multi-armed bandit over 3 models:

Tensorflow neural network model
Scikit-learn random forest model
R least-squares model



In [ ]:

    
get_graph("../k8s_serving/epsilon_greedy_3way.json",'r')



In [ ]:

    
!pygmentize ../k8s_serving/epsilon_greedy_3way.json



In [ ]:

    
!kubectl apply -f ../k8s_serving/epsilon_greedy_3way.json



In [ ]:

    
!kubectl get seldondeployments mnist-classifier -o jsonpath='{.status}'



In [ ]:

    
utils.predict_rest_mnist(mnist)



In [ ]:

    
utils.evaluate_egreedy(mnist,100)



In [ ]: