

In [ ]:

    
# Copyright 2019 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

Train tensorflow or keras model on GCP or Kubeflow from Notebooks

This notebook introduces you to using Kubeflow Fairing to train the model to Kubeflow on Google Kubernetes Engine (GKE), and Google Cloud AI Platform training. This notebook demonstrate how to:

Train an Keras model in a local notebook,
Use Kubeflow Fairing to train an Keras model remotely on Kubeflow cluster,
Use Kubeflow Fairing to train an Keras model remotely on AI Platform training,
Use Kubeflow Fairing to deploy a trained model to Kubeflow, and Call the deployed endpoint for predictions.

You need Python 3.6 to use Kubeflow Fairing.

Setups

Pre-conditions
- Deployed a kubeflow cluster through https://deploy.kubeflow.cloud/
- Have the following environment variable ready:
  - PROJECT_ID # project host the kubeflow cluster or for running AI platform training
  - DEPLOYMENT_NAME # kubeflow deployment name, the same the cluster name after delpoyed
  - GCP_BUCKET # google cloud storage bucket

Create service account

export SA_NAME = [service account name]
gcloud iam service-accounts create ${SA_NAME}
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
  --role 'roles/editor'
gcloud iam service-accounts keys create ~/key.json \
  --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

Authorize for Source Repository
```
gcloud auth configure-docker
```

Update local kubeconfig (for submiting job to kubeflow cluster)

export CLUSTER_NAME=${DEPLOYMENT_NAME} # this is the deployment name or the kubenete cluster name
export ZONE=us-central1-c
gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${ZONE}

Set the environmental variable: GOOGLE_APPLICATION_CREDENTIALS

export GOOGLE_APPLICATION_CREDENTIALS = ....

os.environ['GOOGLE_APPLICATION_CREDENTIALS']=...

Install the lastest version of fairing

pip install git+https://github.com/kubeflow/fairing@master

Please not that the above configuration is required for notebook service running outside Kubeflow environment. And the examples demonstrated in the notebook is fully tested on notebook service outside Kubeflow cluster also.

The environemt variables, e.g. service account, projects and etc, should have been pre-configured while setting up the cluster.



In [1]:

    
import os
import logging
import tensorflow as tf
import fairing
import numpy as np
from datetime import datetime
from fairing.cloud import gcp



In [3]:

    
# Setting up google container repositories (GCR) for storing output containers
# You can use any docker container registry istead of GCR
# For local notebook, GCP_PROJECT should be set explicitly
GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
GCP_Bucket = os.environ['GCP_BUCKET'] # e.g., 'gs://kubeflow-demo-g/'

# This is for local notebook instead of that in kubeflow cluster
# os.environ['GOOGLE_APPLICATION_CREDENTIALS']=

Define the model logic



In [4]:

    
def gcs_copy(src_path, dst_path):
    import subprocess
    print(subprocess.run(['gsutil', 'cp', src_path, dst_path], stdout=subprocess.PIPE).stdout[:-1].decode('utf-8'))
    
def gcs_download(src_path, file_name):
    import subprocess
    print(subprocess.run(['gsutil', 'cp', src_path, file_name], stdout=subprocess.PIPE).stdout[:-1].decode('utf-8'))



In [9]:

    
class TensorflowModel(object):
    
    def __init__(self):
        self.model_file = "mnist_model.h5"
        self.model = None    
    
    def build(self):
        self.model = tf.keras.models.Sequential([
          tf.keras.layers.Flatten(input_shape=(28, 28)),
          tf.keras.layers.Dense(512, activation=tf.nn.relu),
          tf.keras.layers.Dropout(0.2),
          tf.keras.layers.Dense(10, activation=tf.nn.softmax)
        ])
        self.model.compile(optimizer='adam',
                      loss='sparse_categorical_crossentropy',
                      metrics=['accuracy'])
        print(self.model.summary())
    
    def save_model(self):
        self.model.save(self.model_file)
        gcs_copy(self.model_file, GCP_Bucket + self.model_file)
    
    def train(self):
        self.build()
        
        mnist = tf.keras.datasets.mnist
        (x_train, y_train),(x_test, y_test) = mnist.load_data()
        x_train, x_test = x_train / 255.0, x_test / 255.0
        
        callbacks = [
          # Interrupt training if `val_loss` stops improving for over 2 epochs
          tf.keras.callbacks.EarlyStopping(patience=2, monitor='val_loss'),
          # Write TensorBoard logs to `./logs` directory
          tf.keras.callbacks.TensorBoard(log_dir=GCP_Bucket + 'logs/' 
                                         + datetime.now().date().__str__())
        ]
        self.model.fit(x_train, y_train, batch_size=32, epochs=5, callbacks=callbacks,
                  validation_data=(x_test, y_test))
        self.save_model()
        
    def predict(self, X):
        if not self.model:
            self.model = tf.keras.models.load_model(self.model_file)
        # Do any preprocessing
        prediction = self.model.predict(data=X)

Train an Keras model in a notebook



In [10]:

    
TensorflowModel().train()









    



_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten_2 (Flatten)          (None, 784)               0         
_________________________________________________________________
dense_4 (Dense)              (None, 512)               401920    
_________________________________________________________________
dropout_2 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 10)                5130      
=================================================================
Total params: 407,050
Trainable params: 407,050
Non-trainable params: 0
_________________________________________________________________
None
Train on 60000 samples, validate on 10000 samples
Epoch 1/5
60000/60000 [==============================] - 7s 115us/sample - loss: 0.2222 - acc: 0.9352 - val_loss: 0.1027 - val_acc: 0.9686
Epoch 2/5
60000/60000 [==============================] - 7s 122us/sample - loss: 0.0969 - acc: 0.9700 - val_loss: 0.0921 - val_acc: 0.9716
Epoch 3/5
60000/60000 [==============================] - 7s 114us/sample - loss: 0.0688 - acc: 0.9785 - val_loss: 0.0665 - val_acc: 0.9787
Epoch 4/5
60000/60000 [==============================] - 7s 124us/sample - loss: 0.0522 - acc: 0.9833 - val_loss: 0.0704 - val_acc: 0.9793
Epoch 5/5
60000/60000 [==============================] - 7s 109us/sample - loss: 0.0440 - acc: 0.9860 - val_loss: 0.0638 - val_acc: 0.9808

Spicify a image registry that will hold the image built by fairing



In [ ]:

    
# In this demo, I use gsutil, therefore i compile a special image to install GoogleCloudSDK as based image
base_image = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)
!docker build --build-arg PY_VERSION=3.6.4 . -t {base_image}
!docker push {base_image}



In [11]:

    
GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
BASE_IMAGE = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)
DOCKER_REGISTRY = 'gcr.io/{}/fairing-job-tf'.format(GCP_PROJECT)

Deploy the training job to kubeflow cluster



In [19]:

    
from fairing import TrainJob
from fairing.backends import GKEBackend

train_job = TrainJob(TensorflowModel, BASE_IMAGE, input_files=["requirements.txt"],
                     docker_registry=DOCKER_REGISTRY, backend=GKEBackend())
train_job.submit()









    



Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>
Using docker registry: gcr.io/gojek-kubeflow/fairing-job-tf
Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>
Building the docker image.
Building image using docker
Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Creating docker context: /tmp/fairing_context_2n45lxud
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Context: /tmp/fairing_context_2n45lxud, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_2n45lxud, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_2n45lxud, Adding requirements.txt at /app/requirements.txt
Context: /tmp/fairing_context_2n45lxud, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py
Context: /tmp/fairing_context_2n45lxud, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py
Context: /tmp/fairing_context_2n45lxud, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py
Context: /tmp/fairing_context_2n45lxud, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp6n3z2tqq at /app/pickled_fn.p
Context: /tmp/fairing_context_2n45lxud, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp4xybue2z at /app/TensorflowModel.py
Context: /tmp/fairing_context_2n45lxud, Adding /tmp/fairing_dockerfile_zb_zih63 at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F3B46B24...
Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/7 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/7 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/7 : COPY /app//requirements.txt /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 05e78d5eb908
Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> e31aa3ffcc59
Build output: Step 6/7 : COPY /app/ /app/
Build output: 
Build output: ---> 3ae86853d828
Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p
Build output: 
Build output: ---> Running in 5d8b52dd1d5b
Build output: ---> e1d8afd59e79
Push finished: {'ID': 'sha256:e1d8afd59e79d2284cbe145805c30d1e5ed6cf6e540a96260349df454942b78f'}
Build output: Successfully built e1d8afd59e79
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F3B46B24
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F3B46B24...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Pushing [>                                                  ]     512B/45kB
Push output: Pushing [==================================================>]  53.76kB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushed None
Push output: F3B46B24: digest: sha256:7ece47c2d77d36e628d66a49b507471f5452516a513ec126b66250db54a4cf11 size: 3473 None
Push finished: {'Tag': 'F3B46B24', 'Digest': 'sha256:7ece47c2d77d36e628d66a49b507471f5452516a513ec126b66250db54a4cf11', 'Size': 3473}
Not able to find gcp credentials secret: user-gcp-sa
Training job fairing-job-2xzs8 launched.
Waiting for fairing-job-2xzs8-zmwp8 to start...
Pod started running True






    



From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
From /usr/local/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
2019-05-10 05:43:28.440970: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-10 05:43:28.448538: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-05-10 05:43:28.449685: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x406eeb0 executing computations on platform Host. Devices:
2019-05-10 05:43:28.449747: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-05-10 05:43:28.692467: E tensorflow/core/util/events_writer.cc:108] Write failed because file could not be opened.
Train on 60000 samples, validate on 10000 samples
Epoch 1/5
 1184/60000 [..............................] - ETA: 26s - loss: 1.2242 - acc: 0.6363
 2336/60000 [>.............................] - ETA: 21s - loss: 0.8205 - acc: 0.75
 3456/60000 [>.............................] - ETA: 19s - loss: 0.6894 - acc: 0.79
 4512/60000 [=>............................] - ETA: 18s - loss: 0.6282 - acc: 0.81
 5632/60000 [=>............................] - ETA: 17s - loss: 0.5725 - acc: 0.82
 6880/60000 [==>...........................] - ETA: 16s - loss: 0.5282 - acc: 0.84
 8320/60000 [===>..........................] - ETA: 15s - loss: 0.4944 - acc: 0.85
 9664/60000 [===>..........................] - ETA: 14s - loss: 0.4602 - acc: 0.86
11104/60000 [====>.........................] - ETA: 13s - loss: 0.4370 - acc: 0.86
12448/60000 [=====>........................] - ETA: 13s - loss: 0.4213 - acc: 0.87
13856/60000 [=====>........................] - ETA: 12s - loss: 0.4023 - acc: 0.88
15264/60000 [======>.......................] - ETA: 11s - loss: 0.3866 - acc: 0.88
16608/60000 [=======>......................] - ETA: 11s - loss: 0.3756 - acc: 0.88
18016/60000 [========>.....................] - ETA: 10s - loss: 0.3624 - acc: 0.89
19552/60000 [========>.....................] - ETA: 10s - loss: 0.3498 - acc: 0.89
20960/60000 [=========>....................] - ETA: 9s - loss: 0.3405 - acc: 0.899
22592/60000 [==========>...................] - ETA: 9s - loss: 0.3321 - acc: 0.902
24096/60000 [===========>..................] - ETA: 9s - loss: 0.3217 - acc: 0.906
25440/60000 [===========>..................] - ETA: 8s - loss: 0.3146 - acc: 0.907
26752/60000 [============>.................] - ETA: 8s - loss: 0.3061 - acc: 0.910
28192/60000 [=============>................] - ETA: 7s - loss: 0.3000 - acc: 0.911
29600/60000 [=============>................] - ETA: 7s - loss: 0.2930 - acc: 0.913
30976/60000 [==============>...............] - ETA: 7s - loss: 0.2881 - acc: 0.914
32352/60000 [===============>..............] - ETA: 6s - loss: 0.2826 - acc: 0.916
33984/60000 [===============>..............] - ETA: 6s - loss: 0.2783 - acc: 0.917
35296/60000 [================>.............] - ETA: 6s - loss: 0.2726 - acc: 0.919
36672/60000 [=================>............] - ETA: 5s - loss: 0.2676 - acc: 0.920
38080/60000 [==================>...........] - ETA: 5s - loss: 0.2625 - acc: 0.921
39424/60000 [==================>...........] - ETA: 5s - loss: 0.2591 - acc: 0.922
40864/60000 [===================>..........] - ETA: 4s - loss: 0.2547 - acc: 0.924
42208/60000 [====================>.........] - ETA: 4s - loss: 0.2509 - acc: 0.924
43808/60000 [====================>.........] - ETA: 3s - loss: 0.2469 - acc: 0.926
45280/60000 [=====================>........] - ETA: 3s - loss: 0.2446 - acc: 0.927
46784/60000 [======================>.......] - ETA: 3s - loss: 0.2419 - acc: 0.928
48064/60000 [=======================>......] - ETA: 2s - loss: 0.2386 - acc: 0.929
49440/60000 [=======================>......] - ETA: 2s - loss: 0.2355 - acc: 0.930
50784/60000 [========================>.....] - ETA: 2s - loss: 0.2328 - acc: 0.930
52128/60000 [=========================>....] - ETA: 1s - loss: 0.2305 - acc: 0.931
53568/60000 [=========================>....] - ETA: 1s - loss: 0.2281 - acc: 0.932
55168/60000 [==========================>...] - ETA: 1s - loss: 0.2257 - acc: 0.933
56544/60000 [===========================>..] - ETA: 0s - loss: 0.2236 - acc: 0.933
57952/60000 [===========================>..] - ETA: 0s - loss: 0.2212 - acc: 0.934
59360/60000 [============================>.] - ETA: 0s - loss: 0.2191 - acc: 0.935
60000/60000 [==============================] - 15s 250us/sample - loss: 0.2179 - acc: 0.9355 - val_loss: 0.1225 - val_acc: 0.9619
Epoch 2/5
 1376/60000 [..............................] - ETA: 14s - loss: 0.0829 - acc: 0.97
 2720/60000 [>.............................] - ETA: 13s - loss: 0.0942 - acc: 0.97
 4096/60000 [=>............................] - ETA: 13s - loss: 0.1059 - acc: 0.96
 5408/60000 [=>............................] - ETA: 12s - loss: 0.1030 - acc: 0.96
 6688/60000 [==>...........................] - ETA: 12s - loss: 0.1015 - acc: 0.96
 8128/60000 [===>..........................] - ETA: 12s - loss: 0.1002 - acc: 0.96
 9504/60000 [===>..........................] - ETA: 11s - loss: 0.0975 - acc: 0.97
10816/60000 [====>.........................] - ETA: 11s - loss: 0.0963 - acc: 0.97
12160/60000 [=====>........................] - ETA: 11s - loss: 0.0982 - acc: 0.969
13504/60000 [=====>........................] - ETA: 11s - loss: 0.0983 - acc: 0.969
14880/60000 [======>.......................] - ETA: 10s - loss: 0.0969 - acc: 0.969
16192/60000 [=======>......................] - ETA: 10s - loss: 0.0969 - acc: 0.969
17600/60000 [=======>......................] - ETA: 10s - loss: 0.0957 - acc: 0.970
18880/60000 [========>.....................] - ETA: 9s - loss: 0.0937 - acc: 0.9704
20424/60000 [=========>....................] - ETA: 9s - loss: 0.0940 - acc: 0.971
21760/60000 [=========>....................] - ETA: 9s - loss: 0.0939 - acc: 0.971
23072/60000 [==========>...................] - ETA: 8s - loss: 0.0929 - acc: 0.971
24416/60000 [===========>..................] - ETA: 8s - loss: 0.0938 - acc: 0.971
25728/60000 [===========>..................] - ETA: 8s - loss: 0.0936 - acc: 0.971
27072/60000 [============>.................] - ETA: 7s - loss: 0.0952 - acc: 0.971
28384/60000 [=============>................] - ETA: 7s - loss: 0.0953 - acc: 0.970
29696/60000 [=============>................] - ETA: 7s - loss: 0.0948 - acc: 0.971
31200/60000 [==============>...............] - ETA: 6s - loss: 0.0940 - acc: 0.971
32544/60000 [===============>..............] - ETA: 6s - loss: 0.0940 - acc: 0.971
33888/60000 [===============>..............] - ETA: 6s - loss: 0.0938 - acc: 0.971
35232/60000 [================>.............] - ETA: 5s - loss: 0.0939 - acc: 0.970
36544/60000 [=================>............] - ETA: 5s - loss: 0.0940 - acc: 0.970
37920/60000 [=================>............] - ETA: 5s - loss: 0.0938 - acc: 0.970
39328/60000 [==================>...........] - ETA: 4s - loss: 0.0939 - acc: 0.970
40640/60000 [===================>..........] - ETA: 4s - loss: 0.0946 - acc: 0.970
42240/60000 [====================>.........] - ETA: 4s - loss: 0.0950 - acc: 0.970
43648/60000 [====================>.........] - ETA: 3s - loss: 0.0957 - acc: 0.970
44928/60000 [=====================>........] - ETA: 3s - loss: 0.0958 - acc: 0.970
46368/60000 [======================>.......] - ETA: 3s - loss: 0.0961 - acc: 0.970
47712/60000 [======================>.......] - ETA: 2s - loss: 0.0958 - acc: 0.970
49056/60000 [=======================>......] - ETA: 2s - loss: 0.0955 - acc: 0.970
50464/60000 [========================>.....] - ETA: 2s - loss: 0.0962 - acc: 0.970
52192/60000 [========================>.....] - ETA: 1s - loss: 0.0959 - acc: 0.970
53568/60000 [=========================>....] - ETA: 1s - loss: 0.0953 - acc: 0.970
54944/60000 [==========================>...] - ETA: 1s - loss: 0.0952 - acc: 0.970
56256/60000 [===========================>..] - ETA: 0s - loss: 0.0956 - acc: 0.970
57536/60000 [===========================>..] - ETA: 0s - loss: 0.0955 - acc: 0.970
58880/60000 [============================>.] - ETA: 0s - loss: 0.0953 - acc: 0.970
60000/60000 [==============================] - 15s 247us/sample - loss: 0.0953 - acc: 0.9705 - val_loss: 0.0795 - val_acc: 0.9763
Epoch 3/5
 1376/600
 2688/60000 [>.....
 4000/60000 [=>..............
 5408/60000 [=>........................
 6720/60000 [==>...........................] - ET
 8096/60000 [===>..........................] - ETA: 12s - l
 9472/60000 [===>..........................] - ETA: 12s - loss: 0.068
10848/60000 [====>.........................] - ETA: 11s - loss: 0.0665 - acc: 0
12224/60000 [=====>........................] - ETA: 11s - loss: 0.0656 - acc: 0.9804
13632/60000 [=====>........................] - ETA: 11s - loss: 0.0678 - acc: 0.9800
15008/60000 [======>.......................] - ETA: 10s - loss: 0.0703 - acc: 0.9787
16384/60000 [=======>......................] - ETA: 10s - loss: 0.0696 - acc: 0.9787
17792/60000 [=======>......................] - ETA: 9s - loss: 0.0697 - acc: 0.9789 
19136/60000 [========>.....................] - ETA: 9s - loss: 0.0687 - acc: 0.9793
20704
22144/60000 [==========>...
23488/60000 [==========>...................] - ET
24800/60000 [===========>..................] - ETA: 8s - loss: 0.0667 -
26144/60000 [============>.................] - ETA: 8s - loss: 0.0660 - acc: 0.9803
27392/60000 [============>.................] - ETA: 7s - loss: 0.0655 - acc: 0.9804
28704/60000 [=============>................] - ETA: 7s - loss: 0.0652 - acc: 0.9802
30016/60000 [==============>...............] - ETA: 7s - loss: 0.0653 - acc: 0.9801
31616/60000 [=
32896/60000 [===============>.......
34272/60000 [================>.............] - ETA: 6s - l
35744/60000 [================>.............] - ETA: 5s - loss: 0.0662 - acc: 0.9
37120/60000 [=================>............] - ETA: 5s - loss: 0.0661 - acc: 0.9799
38624/60000 [==================>...........] - ETA: 5s - loss: 0.0668 - acc: 0.9797
40032/60000 [===================>..........] - ETA: 4s - loss: 0.0675 - acc: 0.9796
4
42752/60000 [==========
44064/60000 [=====================>........]
45376/60000 [=====================>........] - ETA: 3s - loss: 0.06
46816/60000 [======================>.......] - ETA: 3s - loss: 0.0665 - acc: 0.9798
48192/60000 [=======================>......] - ETA: 2s - loss: 0.0662 - acc: 0.9800
49504/60000 [=======================>......] - ETA: 2s - loss: 0.0667 - acc: 0.9799
50816/60000 [========================>.....] - ETA: 2s - loss: 0.0669 - acc: 0.9797
52384/6000
53664/60000 [===================
55072/60000 [==========================>...] - ETA: 1s
56448/60000 [===========================>..] - ETA: 0s - loss: 0.0673 - acc:
57760/60000 [===========================>..] - ETA: 0s - loss: 0.0669 - acc: 0.9798
59136/60000 [============================>.] - ETA: 0s - loss: 0.0671 - acc: 0.9798
60000/60000 [==============================] - 15s 246us/sample - loss: 0.0672 - acc: 0.9797 - val_loss: 0.0690 - val_acc: 0.9782
Epoch 4/5
 1568/600
 3008/60000 [>.....
 4320/60000 [=>..............
 5664/60000 [=>........................
 7008/60000 [==>...........................] - ET
 8384/60000 [===>..........................] - ETA: 11s - l
 9760/60000 [===>..........................] - ETA: 11s - loss: 0.046
11136/60000 [====>.........................] - ETA: 11s - loss: 0.0451 - acc: 0
12448/60000 [=====>........................] - ETA: 11s - loss: 0.0482 - acc: 0.9851
13856/60000 [=====>........................] - ETA: 10s - loss: 0.0472 - acc: 0.9856
15296/60000 [======>.......................] - ETA: 10s - loss: 0.0479 - acc: 0.9852
16704/60000 [=======>......................] - ETA: 9s - loss: 0.0473 - acc: 0.9854 
18144/60000 [========>.....................] - ETA: 9s - loss: 0.0482 - acc: 0.9850
19584/60000 [========>.....................] - ETA: 9s - loss: 0.0498 - acc: 0.9839
21184/60000 [====
22624/60000 [==========>...............
24128/60000 [===========>..................] - ETA: 8s - loss
25440/60000 [===========>..................] - ETA: 7s - loss: 0.0482 - acc: 0.9841
26752/60000 [============>.................] - ETA: 7s - loss: 0.0478 - acc: 0.9843
28128/60000 [=============>................] - ETA: 7s - loss: 0.0483 - acc: 0.9843
29472/60000 [=============>................] - ETA: 7s - loss: 0.0487 - acc: 0.9840
3107
32448/60000 [=============
33760/60000 [===============>..............] - E
35104/60000 [================>.............] - ETA: 5s - loss: 0.0502
36576/60000 [=================>............] - ETA: 5s - loss: 0.0505 - acc: 0.9835
37824/60000 [=================>............] - ETA: 5s - loss: 0.0508 - acc: 0.9833
39200/60000 [==================>...........] - ETA: 4s - loss: 0.0510 - acc: 0.9831
40672/60000 [===================>..........] - ETA: 4s - loss: 0.0507 - acc: 0.9832
42304/60000 [
43680/60000 [====================>.
45152/60000 [=====================>........] - ETA: 3s -
46528/60000 [======================>.......] - ETA: 3s - loss: 0.0520 - acc: 0.
47840/60000 [======================>.......] - ETA: 2s - loss: 0.0520 - acc: 0.9827
49312/60000 [=======================>......] - ETA: 2s - loss: 0.0517 - acc: 0.9828
50656/60000 [========================>.....] - ETA: 2s - loss: 0.0519 - acc: 0.9827
51968/60000 [========================>.....] - ETA: 1s - loss: 0.0520 - acc: 0.9828
53536/60000 [=========
54912/60000 [==========================>...]
56256/60000 [===========================>..] - ETA: 0s - loss: 0.0
57664/60000 [===========================>..] - ETA: 0s - loss: 0.0523 - acc: 0.9828
59008/60000 [============================>.] - ETA: 0s - loss: 0.0526 - acc: 0.9827
60000/60000 [==============================] - 15s 242us/sample - loss: 0.0527 - acc: 0.9826 - val_loss: 0.0686 - val_acc: 0.9785
Epoch 5/5
 1472/600
 2912/60000 [>.....
 4384/60000 [=>..............
 5792/60000 [=>........................
 7200/60000 [==>...........................] - ET
 8704/60000 [===>..........................] - ETA: 11s - l
10048/60000 [====>.........................] - ETA: 11s - loss: 0.037
11456/60000 [====>.........................] - ETA: 11s - loss: 0.0354 - acc: 0
12864/60000 [=====>........................] - ETA: 10s - loss: 0.0357 - acc: 0.9883
14208/60000 [======>.......................] - ETA: 10s - loss: 0.0358 - acc: 0.9882
15648/60000 [======>.......................] - ETA: 10s - loss: 0.0368 - acc: 0.9881
17120/60000 [=======>......................] - ETA: 9s - loss: 0.0366 - acc: 0.9881
18528/60000 [========>.....................] - ETA: 9s - loss: 0.0374 - acc: 0.9881
2
21536/60000 [=========>
22944/60000 [==========>...................]
24288/60000 [===========>..................] - ETA: 8s - loss: 0.03
25760/60000 [===========>..................] - ETA: 7s - loss: 0.0392 - acc: 0.9876
27200/60000 [============>.................] - ETA: 7s - loss: 0.0392 - acc: 0.9875
28384/60000 [=============>................] - ETA: 7s - loss: 0.0404 - acc: 0.9873
29728/60000 [=============>................] - ETA: 7s - loss: 0.0409 - acc: 0.9871
31488/6000
32960/60000 [===============>...
34400/60000 [================>.............] - ETA: 5s
35776/60000 [================>.............] - ETA: 5s - loss: 0.0403 - acc:
36960/60000 [=================>............] - ETA: 5s - loss: 0.0408 - acc: 0.9867
38304/60000 [==================>...........] - ETA: 5s - loss: 0.0405 - acc: 0.9868
39712/60000 [==================>...........] - ETA: 4s - loss: 0.0408 - acc: 0.9867
41088/60000 [===================>..........] - ETA: 4s - loss: 0.0412 - acc: 0.9866
42752/60000 [======
44224/60000 [=====================>......
45568/60000 [=====================>........] - ETA: 3s - loss:
46976/60000 [======================>.......] - ETA: 3s - loss: 0.0416 - acc: 0.9866
48288/60000 [=======================>......] - ETA: 2s - loss: 0.0416 - acc: 0.9865
49568/60000 [=======================>......] - ETA: 2s - loss: 0.0419 - acc: 0.9863
51008/60000 [========================>.....] - ETA: 2s - loss: 0.0420 - acc: 0.9864
52576/
53920/60000 [===============
55264/60000 [==========================>...] - ETA
56672/60000 [===========================>..] - ETA: 0s - loss: 0.0424 -
57984/60000 [===========================>..] - ETA: 0s - loss: 0.0425 - acc: 0.9861
59360/60000 [============================>.] - ETA: 0s - loss: 0.0427 - acc: 0.9861
60000/60000 [==============================] - 14s 241us/sample - loss: 0.0426 - acc: 0.9861 - val_loss: 0.0733 - val_acc: 0.9784






    



Cleaning up job fairing-job-2xzs8...

Deploy distributed training job to kubeflow cluster



In [20]:

    
fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, 
                           base_image=BASE_IMAGE, push=True)
fairing.config.set_deployer(name='tfjob', worker_count=1, ps_count=1)
run_fn = fairing.config.fn(TensorflowModel)



In [21]:

    
run_fn()









    



Using preprocessor: <fairing.preprocessors.function.FunctionPreProcessor object at 0x13ec8b8d0>
Using builder: <fairing.builders.docker.docker.DockerBuilder object at 0x13db86be0>
Building image using docker
Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Creating docker context: /tmp/fairing_context_umpxhanz
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Context: /tmp/fairing_context_umpxhanz, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_umpxhanz, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_umpxhanz, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py
Context: /tmp/fairing_context_umpxhanz, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py
Context: /tmp/fairing_context_umpxhanz, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py
Context: /tmp/fairing_context_umpxhanz, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmps3h5tzpc at /app/pickled_fn.p
Context: /tmp/fairing_context_umpxhanz, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmp6qz0qc45 at /app/TensorflowModel.py
Context: /tmp/fairing_context_umpxhanz, Adding /tmp/fairing_dockerfile_0u6n5c26 at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:D4321F3C...
Build output: Step 1/6 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/6 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/6 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/6 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> bad4d4b2ac20
Build output: Step 5/6 : COPY /app/ /app/
Build output: 
Build output: ---> 361038c651a2
Build output: Step 6/6 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p
Build output: 
Build output: ---> Running in 541c9b9d7a4e
Build output: ---> b0c230291103
Push finished: {'ID': 'sha256:b0c23029110344ac31205f7d195760abec1a316568fdb12044235fe4f8655904'}
Build output: Successfully built b0c230291103
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:D4321F3C
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:D4321F3C...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Pushing [>                                                  ]     512B/44.95kB
Push output: Pushing [==================================================>]  52.74kB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushed None
Push output: D4321F3C: digest: sha256:4d1c9337dac7cc439f680b54496b7bed8d6a0b9df2a79d7d4be99037e34fa480 size: 3266 None
Push finished: {'Tag': 'D4321F3C', 'Digest': 'sha256:4d1c9337dac7cc439f680b54496b7bed8d6a0b9df2a79d7d4be99037e34fa480', 'Size': 3266}
Training job fairing-tfjob-zsst2 launched.
Waiting for fairing-tfjob-zsst2-worker-0 to start...
Pod started running True






    



From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
From /usr/local/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
2019-05-10 05:47:42.392693: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-10 05:47:42.399617: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-05-10 05:47:42.400345: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x3584220 executing computations on platform Host. Devices:
2019-05-10 05:47:42.400386: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-05-10 05:47:42.659055: E tensorflow/core/util/events_writer.cc:108] Write failed because file could not be opened.
Train on 60000 samples, validate on 10000 samples
Epoch 1/5
 1216/60000 [..............................] - ETA: 25s - loss: 1.1786 - acc: 0.6568
 2368/60000 [>.............................] - ETA: 20s - loss: 0.8382 - acc: 0.75
 3552/60000 [>.............................] - ETA: 18s - loss: 0.6999 - acc: 0.79
 4704/60000 [=>............................] - ETA: 17s - loss: 0.6185 - acc: 0.81
 5856/60000 [=>............................] - ETA: 16s - loss: 0.5724 - acc: 0.82
 6976/60000 [==>...........................] - ETA: 16s - loss: 0.5280 - acc: 0.84
 8064/60000 [===>..........................] - ETA: 16s - loss: 0.4968 - acc: 0.85
 9216/60000 [===>..........................] - ETA: 15s - loss: 0.4655 - acc: 0.86
10368/60000 [====>.........................] - ETA: 15s - loss: 0.4471 - acc: 0.86
11424/60000 [====>.........................] - ETA: 14s - loss: 0.4292 - acc: 0.87
12544/60000 [=====>........................] - ETA: 14s - loss: 0.4115 - acc: 0.87
13696/60000 [=====>........................] - ETA: 14s - loss: 0.4005 - acc: 0.88
14816/60000 [======>.......................] - ETA: 13s - loss: 0.3900 - acc: 0.88
15904/60000 [======>.......................] - ETA: 13s - loss: 0.3765 - acc: 0.88
17056/60000 [=======>......................] - ETA: 12s - loss: 0.3639 - acc: 0.89
18112/60000 [========>.....................] - ETA: 12s - loss: 0.3544 - acc: 0.89
19200/60000 [========>.....................] - ETA: 12s - loss: 0.3467 - acc: 0.89
20544/60000 [=========>....................] - ETA: 11s - loss: 0.3383 - acc: 0.89
21632/60000 [=========>....................] - ETA: 11s - loss: 0.3336 - acc: 0.89
22720/60000 [==========>...................] - ETA: 11s - loss: 0.3263 - acc: 0.90
23872/60000 [==========>...................] - ETA: 10s - loss: 0.3200 - acc: 0.90
25024/60000 [===========>..................] - ETA: 10s - loss: 0.3153 - acc: 0.90
26176/60000 [============>.................] - ETA: 10s - loss: 0.3108 - acc: 0.90
27264/60000 [============>.................] - ETA: 9s - loss: 0.3051 - acc: 0.9073
28352/60000 [=============>................] - ETA: 9s - loss: 0.3016 - acc: 0.909
29504/60000 [=============>................] - ETA: 9s - loss: 0.2959 - acc: 0.910
30464/60000 [==============>...............] - ETA: 8s - loss: 0.2918 - acc: 0.912
31360/60000 [==============>...............] - ETA: 8s - loss: 0.2887 - acc: 0.913
32416/60000 [===============>..............] - ETA: 8s - loss: 0.2853 - acc: 0.914
33504/60000 [===============>..............] - ETA: 8s - loss: 0.2821 - acc: 0.915
34624/60000 [================>.............] - ETA: 7s - loss: 0.2779 - acc: 0.916
35776/60000 [================>.............] - ETA: 7s - loss: 0.2733 - acc: 0.9181
36896/60000 [=================>............] - ETA: 7s - loss: 0.2695 - acc: 0.919
38016/60000 [==================>...........] - ETA: 6s - loss: 0.2658 - acc: 0.920
39168/60000 [==================>...........] - ETA: 6s - loss: 0.2637 - acc: 0.921
40510/60000 [===================>..........] - ETA: 5s - loss: 0.2603 - acc: 0.922
41664/60000 [===================>..........] - ETA: 5s - loss: 0.2563 - acc: 0.923
42816/60000 [====================>.........] - ETA: 5s - loss: 0.2537 - acc: 0.924
43968/60000 [====================>.........] - ETA: 4s - loss: 0.2508 - acc: 0.925
45088/60000 [=====================>........] - ETA: 4s - loss: 0.2477 - acc: 0.926
46208/60000 [======================>.......] - ETA: 4s - loss: 0.2448 - acc: 0.927
47328/60000 [======================>.......] - ETA: 3s - loss: 0.2432 - acc: 0.927
48448/60000 [=======================>......] - ETA: 3s - loss: 0.2413 - acc: 0.928
49792/60000 [=======================>......] - ETA: 3s - loss: 0.2391 - acc: 0.929
50912/60000 [========================>.....] - ETA: 2s - loss: 0.2366 - acc: 0.929
52064/60000 [=========================>....] - ETA: 2s - loss: 0.2344 - acc: 0.930
53216/60000 [=========================>....] - ETA: 2s - loss: 0.2318 - acc: 0.931
54368/60000 [==========================>...] - ETA: 1s - loss: 0.2298 - acc: 0.931
55488/60000 [==========================>...] - ETA: 1s - loss: 0.2279 - acc: 0.931
56608/60000 [===========================>..] - ETA: 1s - loss: 0.2261 - acc: 0.932
57760/60000 [===========================>..] - ETA: 0s - loss: 0.2245 - acc: 0.933
59104/60000 [============================>.] - ETA: 0s - loss: 0.2227 - acc: 0.933
60000/60000 [==============================] - 19s 310us/sample - loss: 0.2206 - acc: 0.9342 - val_loss: 0.1089 - val_acc: 0.9665
Epoch 2/5
 1056/60000 [..............................] - ETA: 19s - loss: 0.1339 - acc: 0.96
 2208/60000 [>.............................] - ETA: 17s - loss: 0.1089 - acc: 0.96
 3328/60000 [>.............................] - ETA: 16s - loss: 0.1035 - acc: 0.96
 4480/60000 [=>............................] - ETA: 16s - loss: 0.0987 - acc: 0.96
 5568/60000 [=>............................] - ETA: 16s - loss: 0.1008 - acc: 0.96
 6528/60000 [==>...........................] - ETA: 16s - loss: 0.1036 - acc: 0.96
 7520/60000 [==>...........................] - ETA: 16s - loss: 0.1048 - acc: 0.96
 8640/60000 [===>..........................] - ETA: 15s - loss: 0.1044 - acc: 0.96
 9696/60000 [===>..........................] - ETA: 15s - loss: 0.1014 - acc: 0.968
10784/60000 [====>.........................] - ETA: 15s - loss: 0.1013 - acc: 0.967
11904/60000 [====>.........................] - ETA: 14s - loss: 0.1043 - acc: 0.967
12960/60000 [=====>........................] - ETA: 14s - loss: 0.1064 - acc: 0.966
14112/60000 [======>.......................] - ETA: 14s - loss: 0.1091 - acc: 0.965
15232/60000 [======>.......................] - ETA: 13s - loss: 0.1078 - acc: 0.966
16384/60000 [=======>......................] - ETA: 13s - loss: 0.1079 - acc: 0.966
17472/60000 [=======>......................] - ETA: 13s - loss: 0.1075 - acc: 0.966
18624/60000 [========>.....................] - ETA: 12s - loss: 0.1069 - acc: 0.96
19936/60000 [========>.....................] - ETA: 12s - loss: 0.1065 - acc: 0.96
21088/60000 [=========>....................] - ETA: 11s - loss: 0.1055 - acc: 0.96
22240/60000 [==========>...................] - ETA: 11s - loss: 0.1050 - acc: 0.96
23328/60000 [==========>...................] - ETA: 11s - loss: 0.1052 - acc: 0.96
24448/60000 [===========>..................] - ETA: 10s - loss: 0.1046 - acc: 0.96
25632/60000 [===========>..................] - ETA: 10s - loss: 0.1051 - acc: 0.96
26688/60000 [============>.................] - ETA: 10s - loss: 0.1045 - acc: 0.96
27776/60000 [============>.................] - ETA: 9s - loss: 0.1036 - acc: 0.9676
28928/60000 [=============>................] - ETA: 9s - loss: 0.1039 - acc: 0.967
30016/60000 [==============>...............] - ETA: 9s - loss: 0.1041 - acc: 0.967
31104/60000 [==============>...............] - ETA: 8s - loss: 0.1037 - acc: 0.967
32416/60000 [===============>..............] - ETA: 8s - loss: 0.1030 - acc: 0.968
33504/60000 [===============>..............] - ETA: 8s - loss: 0.1018 - acc: 0.968
34656/60000 [================>.............] - ETA: 7s - loss: 0.1011 - acc: 0.968
35808/60000 [================>.............] - ETA: 7s - loss: 0.1001 - acc: 0.969
36960/60000 [=================>............] - ETA: 6s - loss: 0.0995 - acc: 0.969
38112/60000 [==================>...........] - ETA: 6s - loss: 0.0995 - acc: 0.969
39264/60000 [==================>...........] - ETA: 6s - loss: 0.1007 - acc: 0.968
40352/60000 [===================>..........] - ETA: 5s - loss: 0.1009 - acc: 0.9688
41728/60000 [====
42880/60000 [====================>.....
43904/60000 [====================>.........] - ETA: 4s - loss
44864/60000 [=====================>........] - ETA: 4s - loss: 0.1012 - acc: 0.9687
45920/60000 [=====================>........] - ETA: 4s - loss: 0.1008 - acc: 0.9688
46976/60000 [======================>.......] - ETA: 3s - loss: 0.1005 - acc: 0.9689
48128/60000 [=======================>......] - ETA: 3s - loss: 0.1007 - acc: 0.9689
4953
50656/60000 [=============
51808/60000 [========================>.....] - E
52960/60000 [=========================>....] - ETA: 2s - loss: 0.1002
54112/60000 [==========================>...] - ETA: 1s - loss: 0.1000 - acc: 0.9693
55264/60000 [==========================>...] - ETA: 1s - loss: 0.0998 - acc: 0.9694
56480/60000 [===========================>..] - ETA: 1s - loss: 0.0996 - acc: 0.9695
57664/60000 [===========================>..] - ETA: 0s - loss: 0.0996 - acc: 0.9694
59008/60000 [
60000/60000 [==============================] - 18s 305us/sample - loss: 0.0989 - acc: 0.9697 - val_loss: 0.0827 - val_acc: 0.9745
Epoch 3/5
 1216/600
 2368/60000 [>.....
 3584/60000 [>...............
 4736/60000 [=>........................
 5952/60000 [=>............................] - ET
 7168/60000 [==>...........................] - ETA: 14s - l
 8320/60000 [===>..........................] - ETA: 14s - loss: 0.066
 9472/60000 [===>..........................] - ETA: 13s - loss: 0.0652 - acc: 0
10656/60000 [====>.........................] - ETA: 13s - loss: 0.0669 - acc: 0.9789
11840/60000 [====>.........................] - ETA: 13s - loss: 0.0658 - acc: 0.9795
12992/60000 [=====>........................] - ETA: 12s - loss: 0.0664 - acc: 0.9791
14176/60000 [======>.......................] - ETA: 12s - loss: 0.0668 - acc: 0.9784
15360/60000 [======>.......................] - ETA: 12s - loss: 0.0651 - acc: 0.9788
16512/60000 [=======>......................] - ETA: 11s - loss: 0.0662 - acc: 0.9785
17696/60000 [=======>......................] - ETA: 11s - loss: 0.0677 - acc: 0.9779
18848/60000 [========>.....................] - ETA: 11s - loss: 0.0676 - acc: 0.9779
20000/60000 [=========>....................] - ETA: 11s - loss: 0.0687 - acc: 0.9777
21248/6000
22240/60000 [=======
23296/60000 [==========>......
24352/60000 [===========>...............
25440/60000 [===========>..................] - ETA: 9
26624/60000 [============>.................] - ETA: 9s - loss: 0.0694 - acc
27872/60000 [============>.................] - ETA: 9s - loss: 0.0693 - acc: 0.9775
29024/60000 [=============>................] - ETA: 8s - loss: 0.0695 - acc: 0.9775
30208/60000 [==============>...............] - ETA: 8s - loss: 0.0695 - acc: 0.9775
31392/60000 [==============>...............] - ETA: 8s - loss: 0.0699 - acc: 0.9775
32768/60000 [=====
33984/60000 [===============>...........
35104/60000 [================>.............] - ETA: 7s - loss:
36224/60000 [=================>............] - ETA: 6s - loss: 0.0699 - acc: 0.9775
37344/60000 [=================>............] - ETA: 6s - loss: 0.0705 - acc: 0.9773
38528/60000 [==================>...........] - ETA: 6s - loss: 0.0709 - acc: 0.9771
39712/60000 [==================>...........] - ETA: 5s - loss: 0.0708 - acc: 0.9772
41056
42272/60000 [==============
43456/60000 [====================>.........] - ET
44640/60000 [=====================>........] - ETA: 4s - loss: 0.0704 -
45792/60000 [=====================>........] - ETA: 3s - loss: 0.0700 - acc: 0.9777
46944/60000 [======================>.......] - ETA: 3s - loss: 0.0699 - acc: 0.9776
48096/60000 [=======================>......] - ETA: 3s - loss: 0.0699 - acc: 0.9776
49280/60000 [=======================>......] - ETA: 3s - loss: 0.0703 - acc: 0.9776
50656/60000 [=
51840/60000 [=======================
52960/60000 [=========================>....] - ETA: 1s - l
54144/60000 [==========================>...] - ETA: 1s - loss: 0.0710 - acc: 0.9
55328/60000 [==========================>...] - ETA: 1s - loss: 0.0710 - acc: 0.9774
56512/60000 [===========================>..] - ETA: 0s - loss: 0.0710 - acc: 0.9773
57696/60000 [===========================>..] - ETA: 0s - loss: 0.0714 - acc: 0.9772
5
60000/60000 [==============================] - 17s 291us/sample - loss: 0.0713 - acc: 0.9772 - val_loss: 0.0655 - val_acc: 0.9786
Epoch 4/5
  992/600
 2080/60000 [>.....
 3168/60000 [>...............
 4352/60000 [=>........................
 5536/60000 [=>............................] - ET
 6688/60000 [==>...........................] - ETA: 15s - l
 7872/60000 [==>...........................] - ETA: 15s - loss: 0.044
 9024/60000 [===>..........................] - ETA: 14s - loss: 0.0439 - acc: 0
10176/60000 [====>.........................] - ETA: 14s - loss: 0.0455 - acc: 0.9857
11392/60000 [====>.........................] - ETA: 13s - loss: 0.0469 - acc: 0.9852
12576/60000 [=====>........................] - ETA: 13s - loss: 0.0521 - acc: 0.9842
13824/60000 [=====>........................] - ETA: 13s - loss: 0.0525 - acc: 0.9840
15008/60000 [======>.......................] - ETA: 12s - loss: 0.0526 - acc: 0.9837
16256/60000 [=======>......................] - ETA: 12s - loss: 0.0519 - acc: 0.9839
17440/60000 [=======>......................] - ETA: 11s - loss: 0.0521 - acc: 0.9837
18592/60000 [========>.....................] - ETA: 11s - loss: 0.0517 - acc: 0.9839
19744/60000 [========>.....................] - ETA: 11s - loss: 0.0525 - acc: 0.9835
21152/6000
22304/60000 [=======
23488/60000 [==========>......
24672/60000 [===========>..................
25888/60000 [===========>..................] - ETA: 9s - loss: 0.
27072/60000 [============>.................] - ETA: 9s - loss: 0.0526 - acc: 0.9835
28224/60000 [=============>................] - ETA: 8s - loss: 0.0523 - acc: 0.9836
29408/60000 [=============>................] - ETA: 8s - loss: 0.0521 - acc: 0.9835
30624/60000 [==============>...............] - ETA: 8s - loss: 0.0523 - acc: 0.9835
32032/60
33280/60000 [===============>.
34464/60000 [================>.............] - ETA:
35616/60000 [================>.............] - ETA: 6s - loss: 0.0524 - ac
36768/60000 [=================>............] - ETA: 6s - loss: 0.0523 - acc: 0.9833
37952/60000 [=================>............] - ETA: 6s - loss: 0.0521 - acc: 0.9835
39072/60000 [==================>...........] - ETA: 5s - loss: 0.0531 - acc: 0.9833
40096/60000 [===================>..........] - ETA: 5s - loss: 0.0527 - acc: 0.9834
41280/60000 [====
42368/60000 [====================>.....
43488/60000 [====================>.........] - ETA: 4s - loss
44640/60000 [=====================>........] - ETA: 4s - loss: 0.0531 - acc: 0.9832
45856/60000 [=====================>........] - ETA: 3s - loss: 0.0529 - acc: 0.9833
46976/60000 [======================>.......] - ETA: 3s - loss: 0.0532 - acc: 0.9830
48224/60000 [=======================>......] - ETA: 3s - loss: 0.0534 - acc: 0.9829
4960
50784/60000 [=============
52000/60000 [=========================>....] - E
53184/60000 [=========================>....] - ETA: 1s - loss: 0.0542
54368/60000 [==========================>...] - ETA: 1s - loss: 0.0543 - acc: 0.9827
55552/60000 [==========================>...] - ETA: 1s - loss: 0.0543 - acc: 0.9827
56736/60000 [===========================>..] - ETA: 0s - loss: 0.0543 - acc: 0.9827
57920/60000 [===========================>..] - ETA: 0s - loss: 0.0537 - acc: 0.9829
59360/60000 [
60000/60000 [==============================] - 17s 288us/sample - loss: 0.0532 - acc: 0.9829 - val_loss: 0.0719 - val_acc: 0.9782
Epoch 5/5
 1216/600
 2368/60000 [>.....
 3520/60000 [>...............
 4704/60000 [=>........................
 5888/60000 [=>............................] - ET
 7136/60000 [==>...........................] - ETA: 14s - l
 8320/60000 [===>..........................] - ETA: 14s - loss: 0.038
 9472/60000 [===>..........................] - ETA: 13s - loss: 0.0385 - acc: 0
10624/60000 [====>.........................] - ETA: 13s - loss: 0.0383 - acc: 0.9874
11776/60000 [====>.........................] - ETA: 13s - loss: 0.0382 - acc: 0.9873
12928/60000 [=====>........................] - ETA: 12s - loss: 0.0388 - acc: 0.9874
14080/60000 [======>.......................] - ETA: 12s - loss: 0.0386 - acc: 0.9876
15200/60000 [======>.......................] - ETA: 12s - loss: 0.0398 - acc: 0.9872
16384/60000 [=======>......................] - ETA: 12s - loss: 0.0405 - acc: 0.9869
17632/60000 [=======>......................] - ETA: 11s - loss: 0.0414 - acc: 0.9867
18688/60000 [========>.....................] - ETA: 11s - loss: 0.0411 - acc: 0.9866
19616/60000 [========>.....................] - ETA: 11s - loss: 0.0408 - acc: 0.9868
20832/6000
21920/60000 [=======
23104/60000 [==========>......
24256/60000 [===========>...............
25376/60000 [===========>..................] - ETA: 9s
26528/60000 [============>.................] - ETA: 9s - loss: 0.0405 - acc:
27744/60000 [============>.................] - ETA: 9s - loss: 0.0410 - acc: 0.9870
28896/60000 [=============>................] - ETA: 8s - loss: 0.0412 - acc: 0.9868
30048/60000 [==============>...............] - ETA: 8s - loss: 0.0415 - acc: 0.9868
31296/60000 [==============>...............] - ETA: 8s - loss: 0.0410 - acc: 0.9869
32672/60000 [=======
33856/60000 [===============>.............
35104/60000 [================>.............] - ETA: 6s - loss: 0
36288/60000 [=================>............] - ETA: 6s - loss: 0.0407 - acc: 0.9870
37472/60000 [=================>............] - ETA: 6s - loss: 0.0405 - acc: 0.9870
38624/60000 [==================>...........] - ETA: 5s - loss: 0.0405 - acc: 0.9870
39808/60000 [==================>...........] - ETA: 5s - loss: 0.0407 - acc: 0.9868
41216/6
42432/60000 [================
43616/60000 [====================>.........] - ETA:
44800/60000 [=====================>........] - ETA: 4s - loss: 0.0418 - a
46016/60000 [======================>.......] - ETA: 3s - loss: 0.0419 - acc: 0.9866
47136/60000 [======================>.......] - ETA: 3s - loss: 0.0422 - acc: 0.9865
48352/60000 [=======================>......] - ETA: 3s - loss: 0.0422 - acc: 0.9865
49536/60000 [=======================>......] - ETA: 2s - loss: 0.0421 - acc: 0.9865
50880/60000 [===
52032/60000 [=========================
53280/60000 [=========================>....] - ETA: 1s - los
54496/60000 [==========================>...] - ETA: 1s - loss: 0.0435 - acc: 0.986
55712/60000 [==========================>...] - ETA: 1s - loss: 0.0440 - acc: 0.9859
56960/60000 [===========================>..] - ETA: 0s - loss: 0.0442 - acc: 0.9859
58144/60000 [============================>.] - ETA: 0s - loss: 0.0445 - acc: 0.9858
593
60000/60000 [==============================] - 17s 289us/sample - loss: 0.0453 - acc: 0.9855 - val_loss: 0.0597 - val_acc: 0.9799
rpc error: code = Unknown desc = Error: No such container: 902d4d41f1f6986dbd1383869a361f63c4bd57ec3a4b4f45f6d22e86bb841162

Deploy the training job as CMLE training job

Doesn’t support CMLE distributed training



In [22]:

    
from fairing import TrainJob
from fairing.backends import GCPManagedBackend
train_job = TrainJob(TensorflowModel, BASE_IMAGE, input_files=["requirements.txt"],
                     docker_registry=DOCKER_REGISTRY, backend=GCPManagedBackend())
train_job.submit()









    



Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>
Using docker registry: gcr.io/gojek-kubeflow/fairing-job-tf
Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>
Building the docker image.
Building image using docker
Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Creating docker context: /tmp/fairing_context_ql6o52sy
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Context: /tmp/fairing_context_ql6o52sy, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_ql6o52sy, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_ql6o52sy, Adding requirements.txt at /app/requirements.txt
Context: /tmp/fairing_context_ql6o52sy, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py
Context: /tmp/fairing_context_ql6o52sy, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py
Context: /tmp/fairing_context_ql6o52sy, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py
Context: /tmp/fairing_context_ql6o52sy, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpr7pp4yyz at /app/pickled_fn.p
Context: /tmp/fairing_context_ql6o52sy, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpwbebk4cv at /app/TensorflowModel.py
Context: /tmp/fairing_context_ql6o52sy, Adding /tmp/fairing_dockerfile_m8l47yuv at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:258D8D01...
Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/7 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/7 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/7 : COPY /app//requirements.txt /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 05e78d5eb908
Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> e31aa3ffcc59
Build output: Step 6/7 : COPY /app/ /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 3ae86853d828
Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p
Build output: 
Build output: ---> Using cache
Build output: ---> e1d8afd59e79
Push finished: {'ID': 'sha256:e1d8afd59e79d2284cbe145805c30d1e5ed6cf6e540a96260349df454942b78f'}
Build output: Successfully built e1d8afd59e79
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:258D8D01
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:258D8D01...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: 258D8D01: digest: sha256:7ece47c2d77d36e628d66a49b507471f5452516a513ec126b66250db54a4cf11 size: 3473 None
Push finished: {'Tag': '258D8D01', 'Digest': 'sha256:7ece47c2d77d36e628d66a49b507471f5452516a513ec126b66250db54a4cf11', 'Size': 3473}
file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest
URL being requested: POST https://ml.googleapis.com/v1/projects/gojek-kubeflow/jobs?alt=json






    



Creating training job with the following options: {'jobId': 'fairing_job_64b00bf8', 'trainingInput': {'scaleTier': 'BASIC', 'masterConfig': {'imageUri': 'gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:258D8D01'}, 'region': 'us-central1'}}
Job submitted successfully.
Access job logs at the following URL:
https://console.cloud.google.com/mlengine/jobs/fairing_job_64b00bf8?project=gojek-kubeflow

Inspect training process with tensorboard



In [22]:

    
# ! tensorboard --logdir=gs://kubeflow-demo-g/logs --host=localhost --port=8777

Deploy the trained model to Kubeflow for predictions



In [13]:

    
from fairing import PredictionEndpoint
from fairing.backends import KubeflowGKEBackend
# The trained_ames_model.joblib is exported during the above local training
endpoint = PredictionEndpoint(TensorflowModel, BASE_IMAGE, input_files=['mnist_model.h5', "requirements.txt"],
                              docker_registry=DOCKER_REGISTRY, backend=KubeflowGKEBackend())
endpoint.create()









    



Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>
Using docker registry: gcr.io/gojek-kubeflow/fairing-job-tf
Using builder: <class 'fairing.builders.docker.docker.DockerBuilder'>
Building the docker image.
Building image using docker
Docker command: ['python', '/app/function_shim.py', '--serialized_fn_file', '/app/pickled_fn.p']
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Creating docker context: /tmp/fairing_context_ftqfzvuc
/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...
Context: /tmp/fairing_context_ftqfzvuc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_ftqfzvuc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_ftqfzvuc, Adding mnist_model.h5 at /app/mnist_model.h5
Context: /tmp/fairing_context_ftqfzvuc, Adding requirements.txt at /app/requirements.txt
Context: /tmp/fairing_context_ftqfzvuc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/functions/function_shim.py at /app/function_shim.py
Context: /tmp/fairing_context_ftqfzvuc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py
Context: /tmp/fairing_context_ftqfzvuc, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py
Context: /tmp/fairing_context_ftqfzvuc, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpemtc_cv2 at /app/pickled_fn.p
Context: /tmp/fairing_context_ftqfzvuc, Adding /var/folders/1c/dnk5c85905ngk3qvcc9fhlnm00hm6n/T/tmpuam9o33d at /app/TensorflowModel.py
Context: /tmp/fairing_context_ftqfzvuc, Adding /tmp/fairing_dockerfile_lxnsduaj at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:B242A591...
Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/7 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/7 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/7 : COPY /app//requirements.txt /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 05e78d5eb908
Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> e31aa3ffcc59
Build output: Step 6/7 : COPY /app/ /app/
Build output: 
Build output: ---> 9ba9f57fa27d
Build output: Step 7/7 : CMD python /app/function_shim.py --serialized_fn_file /app/pickled_fn.p
Build output: 
Build output: ---> Running in eab23dbc43f2
Build output: ---> 0889349c4cae
Push finished: {'ID': 'sha256:0889349c4caecb63661a2d9912e732f3b2e2155838b3aecac89ac9a87c26c857'}
Build output: Successfully built 0889349c4cae
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:B242A591
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:B242A591...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [>                                                  ]  49.64kB/4.956MB
Push output: Layer already exists None
Push output: Pushing [=========>                                         ]  902.1kB/4.956MB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [=============>                                     ]  1.295MB/4.956MB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [=============>                                     ]  1.361MB/4.956MB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [============================>                      ]  2.803MB/4.956MB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [====================================>              ]  3.655MB/4.956MB
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Pushing [================================================>  ]  4.769MB/4.956MB
Push output: Pushing [==================================================>]  4.965MB
Push output: Pushed None
Push output: B242A591: digest: sha256:cac8f31164f75a4958ff97e8689eacbdb420818f45579c45a3b001c9ee6c99ac size: 3475 None
Push finished: {'Tag': 'B242A591', 'Digest': 'sha256:cac8f31164f75a4958ff97e8689eacbdb420818f45579c45a3b001c9ee6c99ac', 'Size': 3475}
Deploying the endpoint.






    



Waiting for prediction endpoint to come up...






    



Cluster endpoint: http://35.184.251.118:5000/predict
Prediction endpoint: http://35.184.251.118:5000/predict



In [14]:

    
endpoint.delete()









    



Deleted service: kubeflow/fairing-service-vrhnq
Deleted deployment: kubeflow/fairing-deployer-fd2bz



In [ ]:



In [ ]:



In [ ]:



In [ ]: