In [3]:
!mkdir -p ~/agave/funwave-tvd-docker-automation
%cd ~/agave
!pip3 install --upgrade setvar
import re
import os
import sys
from setvar import *
from time import sleep
# This cell enables inline plotting in the notebook
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
!auth-tokens-refresh
We saw in the last notebook how we can build images of our funwave-tvd code and use Agave to make the process a bit easier. We can take some lessons learned from the devops community to automate the building of our images and implement basic benchmarking and testing.
While the Agave fork app we created is handy, it doesn't provide particularly good visibility, let along security. We certainly do not want to share an app like that for others to use. So, let's start start by creating a new Agave app that will build our Docker container. First up, our updated app assets.
While functional, our previous Dockerfile didn't give us much info we could use for things like attribution, discovery, etc. Let's add in some additional fields to give our Dockerfile meaning. We will use a couple new Dockerfile directives to do this
ARG a runtime argument supplied to the docker build command
LABEL one or more terms applied to the image as metadata
In [15]:
writefile("funwave-tvd-docker-automation/Dockerfile","""
FROM stevenrbrandt/science-base
MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>
ARG BUILD_DATE
ARG VERSION
LABEL org.agaveplatform.ax.architecture="x86_64" \
org.agaveplatform.ax.build-date="\$BUILD_DATE" \
org.agaveplatform.ax.version="\$VERSION" \
org.agaveplatform.ax.name="${AGAVE_USERNAME}/funwave-tvd" \
org.agaveplatform.ax.summary="Funwave-TVD is a code to simulate the shallow water and Boussinesq equations written by Dr. Fengyan Shi." \
org.agaveplatform.ax.vcs-type="git" \
org.agaveplatform.ax.vcs-url="https://github.com/fengyanshi/FUNWAVE-TVD" \
org.agaveplatform.ax.license="BSD 3-clause"
USER root
RUN mkdir -p /home/install
RUN chown jovyan /home/install
USER jovyan
RUN cd /home/install && \
git clone https://github.com/fengyanshi/FUNWAVE-TVD && \
cd FUNWAVE-TVD/src && \
perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile && \
make
WORKDIR /home/install/FUNWAVE-TVD/src
RUN mkdir -p /home/jovyan/rundir
WORKDIR /home/jovyan/rundir
""")
In our previous wrapper script, we simply took whatever was given to us an ran it. Here we will restrict the wrapper to run a specific build comman.
Note that we mix an match a couple variable types. The
Agave_*variable are template variables resolved by Agave at runtime with the vales from the job etails. Theversionvariable is a parameter we will define in our app description.
In [62]:
writefile("funwave-tvd-docker-automation/funwave-build-wrapper.txt","""
sudo docker build \
--build-arg "BUILD_DATE=\${AGAVE_JOB_SUBMIT_TIME}" \
--build-arg "VERSION=\${code_version}" \
--rm -t funwave-tvd:\${code_version} .
docker inspect funwave-tvd:\${code_version}
""")
Now we need to create some JSON to tell Agave how to run and advertise our app. This app definition will look a lot like the fork app definition with a few changes. First, we are updating the app id so a new app will be created. Second we change the parameter.
code_version is a string parameter describing the version of the code. We have also removed the data file input from the previous app description. This is because our deployment folder contains the Dockerfile to build our image. No other info is needed to run our build app.
In [63]:
writefile("funwave-tvd-docker-automation/funwave-build-app.txt","""
{
"name":"${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild",
"version":"1.0",
"label":"Builds the funwave docker image",
"shortDescription":"Funwave docker build",
"longDescription":"",
"deploymentSystem":"${AGAVE_STORAGE_SYSTEM_ID}",
"deploymentPath":"automation/funwave-tvd-docker-automation",
"templatePath":"funwave-build-wrapper.txt",
"testPath":"test.txt",
"executionSystem":"${AGAVE_EXECUTION_SYSTEM_ID}",
"executionType":"CLI",
"parallelism":"SERIAL",
"modules":[],
"inputs":[],
"parameters":[{
"id" : "code_version",
"value" : {
"visible":true,
"required":true,
"type":"string",
"order":0,
"enquote":false,
"default":"latest"
},
"details":{
"label": "Version of the code",
"description": "If true, output will be packed and compressed",
"argument": null,
"showArgument": false,
"repeatArgument": false
},
"semantics":{
"argument": null,
"showArgument": false,
"repeatArgument": false
}
}],
"outputs":[]
}
""")
Here is our default test file
In [64]:
writefile("funwave-tvd-docker-automation/test.txt","""
code_version=latest
""")
In [65]:
!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N automation
!files-upload -S ${AGAVE_STORAGE_SYSTEM_ID} -F funwave-tvd-docker-automation automation
In [51]:
!apps-addupdate -F funwave-tvd-docker-automation/funwave-build-app.txt
In [52]:
requestbin_url = !requestbin-create
os.environ['REQUESTBIN_URL'] = requestbin_url[0]
setvar("""
WEBHOOK_URL=${REQUESTBIN_URL}
""")
Now we'll run our build using the following job request. This is very similar to before.
In [53]:
writefile("funwave-tvd-docker-automation/job.json","""
{
"name":"funwave-build",
"appId": "${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild-1.0",
"maxRunTime":"00:10:00",
"archive": false,
"notifications": [
{
"url":"${WEBHOOK_URL}",
"event":"*",
"persistent":"true"
}
],
"parameters": {
"code_version":"latest"
}
}
""")
Because the setvar() command can evalute $() style bash shell substitutions, we will use it to submit our job. This will capture the output of the submit command, and allow us to parse it for the JOB_ID. We'll use the JOB_ID in several subsequent steps.
In [69]:
setvar("""
# Capture the output of the job submit command
OUTPUT=$(jobs-submit -F funwave-tvd-docker-automation/job.json)
# Parse out the job id from the output
JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)
""")
In [70]:
for iter in range(20):
setvar("STAT=$(jobs-status $JOB_ID)")
stat = os.environ["STAT"]
sleep(5.0)
if stat == "FINISHED" or stat == "FAILED":
break
In [74]:
!jobs-output-get -P ${JOB_ID} funwave-build.out
To rebuild our Docker image, we can rerun our job submission command, or simply resumbit the previous job.
In [75]:
setvar("""
# Capture the output of the job submit command
OUTPUT=$(jobs-resubmit ${JOB_ID})
# Parse out the job id from the output
JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)
""")
In [ ]: