Kubeflow pipelines

This notebook goes through the steps of using Kubeflow pipelines using the Python3 interpreter (command-line) to preprocess, train, tune and deploy the babyweight model.

1. Start Hosted Pipelines and Notebook

To try out this notebook, first launch Kubeflow Hosted Pipelines and an AI Platform Notebooks instance. Follow the instructions in this README.md file.

2. Install necessary packages



In [ ]:

    
%pip install --quiet kfp python-dateutil --upgrade

Make sure to restart the kernel to pick up new packages (look for button in the ribbon of icons above this notebook)

3. Connect to the Hosted Pipelines

Visit https://console.cloud.google.com/ai-platform/pipelines/clusters and get the hostname for your cluster. You can get it by clicking on the Settings icon. Alternately, click on the Open Pipelines Dashboard link and look at the URL. Change the settings in the following cell



In [1]:

    
# CHANGE THESE
PIPELINES_HOST='447cdd24f70c9541-dot-us-central1.notebooks.googleusercontent.com'
PROJECT='ai-analytics-solutions'
BUCKET='ai-analytics-solutions-kfpdemo'



In [2]:

    
import kfp
import os
client = kfp.Client(host=PIPELINES_HOST)
#client.list_pipelines()

4. [Optional] Build Docker containers

I have made my containers public (See https://cloud.google.com/container-registry/docs/access-control on how to do this), so you can simply use my images.



In [ ]:

    
%%bash
cd pipelines/containers
#bash build_all.sh

Check that the Docker images work properly ...



In [ ]:

    
#!docker run -t gcr.io/ai-analytics-solutions/babyweight-pipeline-bqtocsv:latest --project $PROJECT  --bucket $BUCKET --local

5. Upload and execute pipeline

Upload to the Kubeflow pipeline cluster



In [3]:

    
from pipelines.containers.pipeline import mlp_babyweight

args = {
    'project' : PROJECT, 
    'bucket' : BUCKET
}

#pipeline = client.create_run_from_pipeline_func(mlp_babyweight.preprocess_train_and_deploy, args)

os.environ['HPARAM_JOB'] = 'babyweight_200207_231639' # change to job from complete step
pipeline = client.create_run_from_pipeline_func(mlp_babyweight.train_and_deploy, args)









    




Experiment link here






    




Run link here



In [ ]:

    
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.



In [ ]: