This notebook shows the client code needed to consume a hybrid Keras-Tensorflow model served over Tensorflow Serving. The Tensorflow Serving Model Server needs to be started against our MNIST CNN test model at EXPORT_DIR_ROOT/EXPORT_MODEL_NAME using the following command:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \
--port=9000 --model_name=ktf-mnist-cnn \
--model_base_path=/home/sujit/Projects/polydlot/data/tf-export/ktf-mnist-cnn
Code for the client is adapted from the mnist_client.py code provided as part of the TF-Serving examples.
In [1]:
from __future__ import division, print_function
from grpc.beta import implementations
from sklearn.preprocessing import OneHotEncoder
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2
import os
import sys
import time
import numpy as np
import tensorflow as tf
In [2]:
# NUM_TESTS = 10
NUM_TESTS = 1
SERVER_HOST = "localhost"
SERVER_PORT = 9000
WORK_DIR = "/tmp"
DATA_DIR = "../../data"
TEST_FILE = os.path.join(DATA_DIR, "mnist_test.csv")
IMG_SIZE = 28
NUM_CLASSES = 10
BATCH_SIZE = 1
MODEL_NAME = "ktf-mnist-cnn"
In [3]:
def parse_file(filename):
xdata, ydata = [], []
fin = open(filename, "rb")
i = 0
for line in fin:
if i % 10000 == 0:
print("{:s}: {:d} lines read".format(
os.path.basename(filename), i))
cols = line.strip().split(",")
ydata.append(int(cols[0]))
xdata.append(np.reshape(np.array([float(x) / 255. for x in cols[1:]]),
(IMG_SIZE, IMG_SIZE, 1)))
i += 1
fin.close()
print("{:s}: {:d} lines read".format(os.path.basename(filename), i))
X = np.array(xdata, dtype="float32")
y = np.array(ydata, dtype="int32")
return X, y
Xtest, ytest = parse_file(TEST_FILE)
print(Xtest.shape, ytest.shape)
Since my Keras/TF hybrid model computes validation set accuracy during training, I need to pass in the labels in the original model as well, hence the labels input. In an actual prediction situation, you could simply pass a zero vector. However, the error message seems to indicate that it is looking for another placeholder (which in this case has been declared to be the output placeholder Y_. I believe it is looking for the Keras learning phase flag, which I cannot figure out how to pass in this model.
In [7]:
channel = implementations.insecure_channel(SERVER_HOST, SERVER_PORT)
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
for i in range(NUM_TESTS):
request = predict_pb2.PredictRequest()
request.model_spec.name = MODEL_NAME
request.model_spec.signature_name = "predict"
Xbatch, ybatch = Xtest[i], ytest[i]
Ybatch = np.zeros((NUM_CLASSES), dtype="int32")
print(Xbatch.shape, Ybatch.shape)
request.inputs["images"].CopyFrom(
tf.contrib.util.make_tensor_proto(Xbatch, shape=[1, IMG_SIZE, IMG_SIZE, 1]))
request.inputs["labels"].CopyFrom(
tf.contrib.util.make_tensor_proto(Ybatch, shape=[1, NUM_CLASSES]))
result = stub.Predict(request, 10.0)
In [ ]: