

In [ ]:

    
import glob, re, os
import logging
import fairing
GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
DOCKER_REGISTRY = 'gcr.io/{}'.format(GCP_PROJECT) # every Google Cloud Platform project comes with a private Docker registry
base_image = "{}/{}".format(DOCKER_REGISTRY, "fairing:latest")
logging.getLogger('googleapiclient.discovery_cache').setLevel(logging.ERROR) # suppress nagging bug about a library incompatibility
print(base_image)

Authenticate with the docker registry first

gcloud auth configure-docker

If using TPUs please also authorize Cloud TPU to access your project as described here.

Set up your output bucket



In [ ]:

    
BUCKET = "gs://"  # your bucket here
assert re.search(r'gs://.+', BUCKET), 'A GCS bucket is required to store your results.'

Build a base image to work with fairing



In [1]:

    
!cat Dockerfile









    



# The base image for AI Platform Notebook instances
FROM gcr.io/deeplearning-platform-release/tf-gpu.1-13
# Stuff that should have been configured in the base image (bug)
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
# Upgrading to Tensorflow 2.0
RUN pip install tensorflow-gpu==2.0.0-beta1



In [ ]:

    
!docker build . -t {base_image}



In [ ]:

    
!docker push {base_image}

Start an AI Platform job



In [ ]:

    
additional_files = '' # If your code requires additional files, you can specify them here (or include everything in the current folder with glob.glob('./**', recursive=True))
# If your code does not require any dependencies or config changes, you can directly start from an official Tensorflow docker image
#fairing.config.set_builder('docker', registry=DOCKER_REGISTRY, base_image='gcr.io/deeplearning-platform-release/tf-gpu.1-13')

# base image
fairing.config.set_builder('docker', registry=DOCKER_REGISTRY, base_image=base_image)
# AI Platform job hardware config
fairing.config.set_deployer('gcp', job_config={'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard_p100'}})
# input and output notebooks
fairing.config.set_preprocessor('full_notebook',
                                notebook_file="05K_MNIST_TF20Keras_Tensorboard_playground.ipynb",
                                input_files=additional_files,
                                output_file=os.path.join(BUCKET, 'fairing-output', 'mnist-001.ipynb'))


# GPU settings for single K80, single p100 respectively
# job_config={'trainingInput': {'scaleTier': 'BASIC_GPU'}}
# job_config={'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'standard_p100'}}

# These job_config settings for TPUv2
#job_config={'trainingInput': {'scaleTier': 'BASIC_GPU'}}
#job_config={'trainingInput': {'scaleTier': 'CUSTOM', 'masterType': 'n1-standard-8', 'workerType': 'cloud_tpu', 'workerCount': 1,
#                                               'workerConfig': {'accelerator_config': {'type': 'TPU_V2','count': 8}}}})
# On AI Platform, TPUv3 support is alpha and available to whitelisted customers only



In [ ]:

    
fairing.config.run()

License

author: Martin Gorner
twitter: @martin_gorner

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This is not an official Google product but sample code provided for an educational purpose