

In [ ]:

    
# Copyright 2019 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

Train tensorflow or keras model in .py files on GCP or Kubeflow from Notebooks

This notebook introduces you to using Kubeflow Fairing to train the model, which is developed using tensorflow or keras and enclosed in python files, to Kubeflow on Google Kubernetes Engine (GKE), and Google Cloud AI Platform training. This notebook demonstrate how to:

Use Kubeflow Fairing to train an Tensorflow model remotely on Kubeflow cluster,
Use Kubeflow Fairing to train an Tensorflow model remotely on AI Platform training,

You need Python 3.6 to use Kubeflow Fairing.

Setups

Pre-conditions
- Deployed a kubeflow cluster through https://deploy.kubeflow.cloud/
- Have the following environment variable ready:
  - PROJECT_ID # project host the kubeflow cluster or for running AI platform training
  - DEPLOYMENT_NAME # kubeflow deployment name, the same the cluster name after delpoyed
  - GCP_BUCKET # google cloud storage bucket

Create service account

export SA_NAME = [service account name]
gcloud iam service-accounts create ${SA_NAME}
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \
  --role 'roles/editor'
gcloud iam service-accounts keys create ~/key.json \
  --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

Authorize for Source Repository
```
gcloud auth configure-docker
```

Update local kubeconfig (for submiting job to kubeflow cluster)

export CLUSTER_NAME=${DEPLOYMENT_NAME} # this is the deployment name or the kubenete cluster name
export ZONE=us-central1-c
gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${ZONE}

Set the environmental variable: GOOGLE_APPLICATION_CREDENTIALS

export GOOGLE_APPLICATION_CREDENTIALS = ....

os.environ['GOOGLE_APPLICATION_CREDENTIALS']=...

Install the lastest version of fairing

pip install git+https://github.com/kubeflow/fairing@master

Please not that the above configuration is required for notebook service running outside Kubeflow environment. And the examples demonstrated in the notebook is fully tested on notebook service outside Kubeflow cluster also.

The environemt variables, e.g. service account, projects and etc, should have been pre-configured while setting up the cluster.



In [5]:

    
import os
import fairing
from fairing.cloud import gcp



In [ ]:

    
# Setting up google container repositories (GCR) for storing output containers
# You can use any docker container registry istead of GCR
# For local notebook, GCP_PROJECT should be set explicitly
GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
GCP_Bucket = os.environ['GCP_BUCKET'] # e.g., 'gs://kubeflow-demo-g/'

# This is for local notebook instead of that in kubeflow cluster
# os.environ['GOOGLE_APPLICATION_CREDENTIALS']=



In [ ]:

    
# In this demo, I use gsutil, therefore i compile a special image to install GoogleCloudSDK as based image
base_image = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)
!docker build --build-arg PY_VERSION=3.6.4 . -t {base_image}
!docker push {base_image}



In [6]:

    
GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
DOCKER_REGISTRY = 'gcr.io/{}/fairing-job-tf'.format(GCP_PROJECT)
BASE_IMAGE = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)









    



gojek-kubeflow



In [7]:

    
file_name = 'model.py'

Deploy the training job to AI platform training



In [8]:

    
fairing.config.set_preprocessor('python', executable=file_name, input_files=[file_name, 'requirements.txt'])
fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, base_image=BASE_IMAGE, push=True)
fairing.config.set_deployer(name='gcp')
fairing.config.run()









    



Using preprocessor: <fairing.preprocessors.base.BasePreProcessor object at 0x11f4fe128>
Using builder: <fairing.builders.docker.docker.DockerBuilder object at 0x11f4fee48>
file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest
Building image using docker
Docker command: ['python', '/app/model.py']
Creating docker context: /tmp/fairing_context_6ulb5qmv
Context: /tmp/fairing_context_6ulb5qmv, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_6ulb5qmv, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_6ulb5qmv, Adding model.py at /app/model.py
Context: /tmp/fairing_context_6ulb5qmv, Adding requirements.txt at /app/requirements.txt
Context: /tmp/fairing_context_6ulb5qmv, Adding /tmp/fairing_dockerfile_ju8v9_bp at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E...
Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/7 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/7 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/7 : COPY /app//requirements.txt /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 05e78d5eb908
Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> e31aa3ffcc59
Build output: Step 6/7 : COPY /app/ /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 710caec21dce
Build output: Step 7/7 : CMD python /app/model.py
Build output: 
Build output: ---> Using cache
Build output: ---> d52847d8c0d9
Push finished: {'ID': 'sha256:d52847d8c0d9ba6599a6f0f8c85a21fbe0d03ac17093ec6792ed92ab82dfe5fb'}
Build output: Successfully built d52847d8c0d9
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: F0D4918E: digest: sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157 size: 3472 None
Push finished: {'Tag': 'F0D4918E', 'Digest': 'sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157', 'Size': 3472}
URL being requested: POST https://ml.googleapis.com/v1/projects/gojek-kubeflow/jobs?alt=json






    



Creating training job with the following options: {'jobId': 'fairing_job_5310fcf1', 'trainingInput': {'masterConfig': {'imageUri': 'gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E'}, 'region': 'us-central1'}}
Job submitted successfully.
Access job logs at the following URL:
https://console.cloud.google.com/mlengine/jobs/fairing_job_5310fcf1?project=gojek-kubeflow






    Out[8]:





(<fairing.preprocessors.base.BasePreProcessor at 0x11f4fe128>,
 <fairing.builders.docker.docker.DockerBuilder at 0x11f4fee48>,
 <fairing.deployers.gcp.gcp.GCPJob at 0x11f4fefd0>)

Deploy the training job to kubeflow cluster



In [9]:

    
fairing.config.set_preprocessor('python', executable=file_name, input_files=[file_name, 'requirements.txt'])
fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, base_image=BASE_IMAGE, push=True)
fairing.config.set_deployer(name='job')
fairing.config.run()









    



Using preprocessor: <fairing.preprocessors.base.BasePreProcessor object at 0x11f5e9cf8>
Using builder: <fairing.builders.docker.docker.DockerBuilder object at 0x11f5e95c0>
Building image using docker
Docker command: ['python', '/app/model.py']
Creating docker context: /tmp/fairing_context_f3f4quru
Context: /tmp/fairing_context_f3f4quru, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py
Context: /tmp/fairing_context_f3f4quru, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py
Context: /tmp/fairing_context_f3f4quru, Adding model.py at /app/model.py
Context: /tmp/fairing_context_f3f4quru, Adding requirements.txt at /app/requirements.txt
Context: /tmp/fairing_context_f3f4quru, Adding /tmp/fairing_dockerfile_wpuoc5id at Dockerfile
Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB...
Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest
Build output: 
Build output: ---> 07b0c0a773a2
Build output: Step 2/7 : WORKDIR /app/
Build output: 
Build output: ---> Using cache
Build output: ---> e38aad2dc182
Build output: Step 3/7 : ENV FAIRING_RUNTIME 1
Build output: 
Build output: ---> Using cache
Build output: ---> 597bd070338a
Build output: Step 4/7 : COPY /app//requirements.txt /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 05e78d5eb908
Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi
Build output: 
Build output: ---> Using cache
Build output: ---> e31aa3ffcc59
Build output: Step 6/7 : COPY /app/ /app/
Build output: 
Build output: ---> Using cache
Build output: ---> 710caec21dce
Build output: Step 7/7 : CMD python /app/model.py
Build output: 
Build output: ---> Using cache
Build output: ---> d52847d8c0d9
Push finished: {'ID': 'sha256:d52847d8c0d9ba6599a6f0f8c85a21fbe0d03ac17093ec6792ed92ab82dfe5fb'}
Build output: Successfully built d52847d8c0d9
Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB
Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB...
Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Preparing None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Waiting None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: Layer already exists None
Push output: BA147BAB: digest: sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157 size: 3472 None
Push finished: {'Tag': 'BA147BAB', 'Digest': 'sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157', 'Size': 3472}
Training job fairing-job-s6pfs launched.
Waiting for fairing-job-s6pfs-wr9gb to start...
Pod started running True






    



INFO:tensorflow:TF_CONFIG {}
INFO:tensorflow:cluster=None job_name=None task_index=None
INFO:tensorflow:Will export model
WARNING:tensorflow:From /app/model.py:179: load_mnist (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:300: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tensorflow/logs', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fdeb8464390>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:62: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /app/model.py:87: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From /app/model.py:89: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
WARNING:tensorflow:From /app/model.py:105: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:From /app/model.py:109: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dropout instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2019-05-10 05:58:07.339398: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-10 05:58:07.346743: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-05-10 05:58:07.348307: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x43f44b0 executing computations on platform Host. Devices:
2019-05-10 05:58:07.348362: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tensorflow/logs/model.ckpt.
INFO:tensorflow:loss = 2.316252, step = 1
INFO:tensorflow:global_step/sec: 6.55602
INFO:tensorflow:loss = 2.0959373, step = 101 (15.253 sec)
INFO:tensorflow:Saving checkpoints for 200 into /tmp/tensorflow/logs/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2019-05-10T05:58:38Z
INFO:tensorflow:Graph was finalized.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2019-05-10-05:58:38
INFO:tensorflow:Saving dict for global step 200: accuracy = 0.8046875, global_step = 200, loss = 0.94937694
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 200: /tmp/tensorflow/logs/model.ckpt-200
INFO:tensorflow:Performing the final export in the end of training.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classes', 'serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tensorflow/model/temp-b'1557467918'/saved_model.pb
INFO:tensorflow:Loss for final step: 0.95952034.
WARNING:tensorflow:Directory b'/tmp/tensorflow/model/1557467918' already exists; retrying (attempt 1/10)
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classes', 'serving_default']
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: None
INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tensorflow/model/temp-b'1557467919'/saved_model.pb

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting /tmp/tensorflow/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting /tmp/tensorflow/input_data/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/input_data/t10k-labels-idx1-ubyte.gz
Train and evaluate
Training done
Export saved model
Done exporting the model






    



Cleaning up job fairing-job-s6pfs...






    Out[9]:





(<fairing.preprocessors.base.BasePreProcessor at 0x11f5e9cf8>,
 <fairing.builders.docker.docker.DockerBuilder at 0x11f5e95c0>,
 <fairing.deployers.job.job.Job at 0x120600630>)



In [ ]: