Serving a Keras Resnet Model

Keras is a high level API that can be used to build deep neural nets with only a few lines of code, and supports a number of backends for computation (TensorFlow, Theano, and CNTK). Keras also contains a library of pre-trained models, including a Resnet model with 50-layers, trained on the ImageNet dataset, which we will use for this exercise.

This notebook teaches how to create a servable version of the Imagenet Resnet50 model in Keras using the TensorFlow backend. The servable model can be served using TensorFlow Serving, which runs very efficiently in C++ and supports multiple platforms (different OSes, as well as hardware with different types of accelerators such as GPUs). The model will need to handle RPC prediction calls coming from a client that sends requests containing a batch of jpeg images.

See https://github.com/keras-team/keras/blob/master/keras/applications/resnet50.py for the implementation of ResNet50.

Preamble

Import the required libraries.

Constants

Setting the Output Directory

Set a version number and directory for the output of the servable model. Note that with TensorFlow, if you've successfully saved the servable in a directory, trying to save another servable will fail hard. You always want to increment your version number, or otherwise delete the output directory and re-run the servable creation code.

Build the Servable Model from Keras

Keras has a prepackaged ImageNet-trained ResNet50 model which takes in a 4d input tensor (i.e. a tensor of RGB-color images) and outputs a list of class probabilities for all of the classes.

We will create a servable model whose input takes in a batch of jpeg-encoded images, and outputs a dictionary containing the top k classes and probabilities for each image in the batch. We've refactored the input preprocessing and output postprocessing into helper functions.

Helper Functions for Building a TensorFlow Graph

TensorFlow is essentially a computation graph with variables and states. The graph must be built before it can ingest and process data. Typically, a TensorFlow graph will contain a set of input nodes (called placeholders) from which data can be ingested, and a set of TensorFlow functions that take existing nodes as inputs and produces a dependent node that performs a computation on the input nodes. Each node can be referenced as an "output" node through which processed data can be read.

It is often useful to create helper functions for building a TensorFlow graphs for two reasons:

1. Modularity: you can reuse functions in different places; for instance, a different image model or ResNet architecture can reuse functions.
2. Testability: you can attach placeholders at the input of graph building helper functions, and read the output to ensure that your result matches expected behavior.

Helper function: convert JPEG strings to Normalized 3D Tensors

The client (resnet_client.py) sends jpeg encoded images into an array of jpegs (each entry a string) to send to the server. These jpegs are all appropriately resized to 224x224x3, and do not need resizing on the server side to enter into the ResNet model. However, the ResNet50 model was trained with pixel values normalized using a predefined method in the Keras resnet50 package (resnet50.preprocess_input()). We will need to extract the raw 3D tensor from each jpeg string and normalize the values.

Exercise: Add a command in the helper function to build a node that decodes a jpeg string into a 3D RGB image tensor.

Useful References:

• tf.image module

Unit test the helper function

Exercise: We are going to construct an input placeholder node in our TensorFlow graph to read data into TensorFlow, and use the helper function to attach computational elements to the input node, resulting in an output node where data is collected. Next, we will then run the graph by providing sample input into the placeholder (Input data can be python floats, ints, strings, numpy arrays, ndarrays, etc.), and returning the value at the output node.

A placeholder can store a Tensor of arbitrary dimension, and arbitrary length in any dimension.

An example of a placeholder that holds a 1d tensor of floating values is:

x = tf.placeholder(dtype=tf.float32, shape=[10], 'my_input_node')

An example of a 2d tensor (matrix) of dimensions 10x20 holding string values is:

x = tf.placeholder(dtype=tf.string, shape=[10, 20], 'my_string_matrix')

Note that we assigned a Python variable x to be a pointer to the placeholder, but simply calling tf.placeholder() with a named element would create an element in the TensorFlow graph that can be referenced in a global dictionary as 'my_input_node'. However, it helps to keep a Python pointer to keep track of the element without having to and pass it into helper functions.

Any dependent node in the graph can serve as an output node. For instance, passing an input node x through y = build_jpeg_to_image_graph(x) would return a node referenced by python variable y which is the result of processing the input through the graph built by the helper function. When we run the test graph with real data below, you will see how to return the output of y.

Remember: TensorFlow helper functions are used to help construct a computational graph! build_jpeg_to_image_graph() does not return a 3D array. It returns a graph node that returns a 3D array after processing a jpeg-encoded string!**

Useful References:

TensorFlow shapes, TensorFlow data types

Run the Test Graph

To run data through a test graph, a TensorFlow session must be created. TensorFlow will only run a portion of the graph that is required to map a set of inputs defined by a dictionary with entries of type {placeholder_node : python data}, and an output graph node (or list of graph nodes), e.g.:

with tf.Session() as sess:
    sess.run(output_node,
             {placeholder_1: input_data_1, placeholder_2: input_data_2, ...})

or

with tf.Session() as sess:
    sess.run([output_node_1, output_node_2, ...],
             {placeholder_1: input_data_1, placeholder_2: input_data_2, ...})

Exercise: Let's read a jpeg image as an encoded string, pass it into the input placeholder, and return a 3D tensor result which is the normalized image. The expected image size is 224x224x3. Please add more potentially useful assert statements to test the output.

Hint: The ??? below are just examples of possible assertions you can make about the range of output values in the preprocessed image. The more robust the assertions, the better you can validate your code! See the documentation for build_jpeg_to_image_graph() above for a description of how the resnet50.preprocess_image() function transforms images to fill in the ??? below.

Remarks

The approach above uses vanilla TensorFlow to perform unit testing. You may notice that the code is more verbose than ideal, since you have to create a session, feed input through a dictionary, etc. We encourage the student to investigate some options below at a later time:

TensorFlow Eager was introduced in TensorFlow 1.5 as a way to execute TensorFlow graphs in a way similar to numpy operations. After testing individual parts of the graph using Eager, you will need to rebuild a graph with the Eager option turned off in order to build a performance optimized TensorFlow graph. Also, keep in mind that you will need another virtual environment with TensorFlow 1.5 in order to run eager execution, which may not be compatible with TensorFlow Serving 1.4 used in this tutorial.

TensorFlow unit testing is a more software engineer oriented approach to run tests. By writing test classes that can be invoked individually when building the project, calling tf.test.main() will run all tests and return a list of ones that succeeded and failed, allowing you to inspect errors. Because we are in a notebook environment, such a test would not succeed due to an already running kernel that tf.test cannot access. The tests must be run from the command line, e.g. python test_my_graph.py.

We've provided both eager execution and unit test examples in the testing directory showing how to unit test various components in this notebook. Note that because these examples contain the solution to exercises below, please complete all notebook exercises prior to reading through these examples.

Now that we know how to run TensorFlow tests, let's create and test more helper functions!

Helper Function: Preprocessing Server Input

The server receives a client request in the form of a dictionary {'images': 1D_tensor_of_jpeg_encoded_strings}, which must be preprocessed into a 4D tensor before feeding into the Keras ResNet50 model.

Exercise: We will be using a ResNet client to send requests to our server. You will need to create a graph to preprocess client requests to be compliant with the client. Using tf.map_fn and build_jpeg_to_image_graph, fill in the missing line (marked ???) to convert the client request into an array of 3D floating-point, preprocessed tensor. The following lines stack and reshape this array into a 4D tensor.

Useful References:

• tf.map_fn
• tf.DType

Unit Test the Input Preprocessing Helper Function

Exercise: Construct a TensorFlow unit test graph for the input function.

Hint: the input node test_jpeg_tensor should be a tf.placeholder. You need to define the shape parameter in tf.placeholder. None inside an array indicates that the length can vary along that dimension.

Helper Function: Postprocess Server Output

Exercise: The Keras model returns a 1D tensor of probabilities for each class. We want to wrote a postprocess_output() that returns only the top k classes and probabilities.

Useful References:

• tf.nn.top_k

Unit Test the Output Postprocessing Helper Function

Exercise: Fill in the shape field for the model output, which should be a 1D tensor of probabilities.

Hint: how many image classes are there?

Load the Keras Model and Build the Graph

The Keras Model uses TensorFlow as its backend, and therefore its inputs and outputs can be treated as elements of a TensorFlow graph. In other words, you can provide an input that is a TensorFlow tensor, and read the model output like a TensorFlow tensor!

Exercise: Build the end to end network by filling in the TODOs below.

Useful References:

• Keras ResNet50 API
• Keras Model class API: ResNet50 model inherits this class.

Creating the Input-Output Signature

Exercise: The final step to creating a servable model is to define the end-to-end input and output API. Edit the inputs and outputs parameters to predict_signature_def below to ensure that the signature correctly handles client request. The inputs parameter should be a dictionary {'images': tensor_of_strings}, and the outputs parameter a dictionary {'classes': tensor_of_top_k_classes, 'probabilities': tensor_of_top_k_probs}.

Export the Servable Model