In [ ]:


In [4]:
%reset -f
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import pandas as pd
%matplotlib inline
import seaborn as sns
sns.set(style="whitegrid", palette="muted", color_codes=True)


import tensorflow as tf; 
print('TF version:' + tf.__version__)


TF version:1.0.1

Versions


In [3]:
import os
print("Path at terminal when executing this file")
print(os.getcwd() + "\n")

%matplotlib inline
%load_ext watermark
%watermark -d -v -m -p pandas,scipy,matplotlib


Path at terminal when executing this file
/root/sharedfolder/dev/cto-gamma_2472/CTO/data-science/jupyter/docker

2017-05-21 

CPython 2.7.6
IPython 5.3.0

pandas 0.18.1
scipy 0.19.0
matplotlib 2.0.0

compiler   : GCC 4.8.4
system     : Linux
release    : 3.13.0-112-generic
machine    : x86_64
processor  : x86_64
CPU cores  : 40
interpreter: 64bit

Docker commands building a CPU based docker

Install Docker

Linux

apt-get install docker.io

OSX

https://download.docker.com/mac/stable/Docker.dmg

OR

Install Homebrew

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install Cask

brew install caskroom/cask/brew-cask

Install docker toolbox

brew cask install docker-toolbox

To allow your user to run docker commands without the sudo prefix

sudo usermod -aG docker shlomo

Base Image


In [ ]:
Base images are images that have no parent image, usually images with an OS like ubuntu, busybox or debian.
We start with specifying our base image. Use the FROM keyword to do that
FROM ubuntu:16.04

docker pull ubuntu:12.04

The next step usually is to write the commands of copying the files and installing the dependencies.

Install TensorFlow CPU version from central repo


In [ ]:
RUN pip --no-cache-dir install \
    http://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.0.0-cp27-none-linux_x86_64.whl

Pick up TF dependencies


In [ ]:
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        curl \
        libfreetype6-dev \
        libpng12-dev \
        libzmq3-dev \
        pkg-config \
        python \
        python-dev \
        rsync \
        software-properties-common \
        unzip \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Install PIP


In [ ]:
RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

Install Python libs


In [ ]:
RUN pip --no-cache-dir install \
        ipykernel \
        jupyter \
        matplotlib \
        numpy \
        scipy \
        sklearn \
        pandas \
        Pillow \
        && \
    python -m ipykernel.kernelspec

Set up our notebook config


In [ ]:
COPY jupyter_notebook_config.py /root/.jupyter/

Open ports


In [ ]:
# TensorBoard
EXPOSE 6006
# IPython
EXPOSE 8888

Jupyter config


In [ ]:
# jupyter_notebook_config.py

import os
from IPython.lib import passwd

c.NotebookApp.ip = '*'
c.NotebookApp.port = int(os.getenv('PORT', 8888))
c.NotebookApp.open_browser = False

# sets a password if PASSWORD is set in the environment
if 'PASSWORD' in os.environ:
  c.NotebookApp.password = passwd(os.environ['PASSWORD'])
  del os.environ['PASSWORD']

Docker components

Docker consists of the following components:

Images

Containers

Daemon

Clients

Registries

Images

Images are read-only templates which provide functionality for running an instance of this image (container). An example for a image is the latest release of Ubuntu. Images are defined as layers, for example, you can add Java to the Ubuntu image and get another image based on this.

The Docker hub provides lot of pre-configured images. You can modify existing images and save these modifications as new image.

Containers

Container are the started components based on images. They contain the actual application and dependencies but share the same kernel. They can be started, stopped, paused, deleted. Containers are immutable and disposable.

Docker Daemon

Is used to manage the container. It runs natively on Linux and inside a VM on Windows and Mac OS X. To start it use the docker command.

Docker Clients

Clients (CLI, IDE) run on host VM. They provide the tools to interact with container, i.e., to start them.

Registries

Images are saved in a registry and have an ID with consists of a repository and a tag. For example, fedora:22, is an image which contains the Fedora 22 OS from the fedora repository.

To use an image you have to pull it from a registry, to share an image with others you have to push it to one. The default Docker registry is the Docker Hub. You can upload your personal images to Github, in this case you add your user name as prefix to the image, e.g., vogella/fedore:22

Docker commands - after the docker script is ready

Search for a docker image


In [ ]:
root@shlomo::~# docker search --stars=5 "postgresql-9.3"
Flag --stars has been deprecated, use --filter=stars=3 instead
NAME                     DESCRIPTION                     STARS     OFFICIAL   AUTOMATED
helmi03/docker-postgis   PostGIS 2.1 in PostgreSQL 9.3   24                   [OK]

Build an image


In [ ]:
docker build -t quantscientist/deep-ml-meetups -f Dockerfile.cpu .

View available images


In [ ]:
root@shlomo:~# docker images
REPOSITORY                          TAG                     IMAGE ID            CREATED              VIRTUAL SIZE
quantscientist/deep-ml-meetups      latest                  3871333c6375        5 weeks ago          5.146 GB

In [ ]:
root@shlomo:~# docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
docker-airflow         latest              8117652bf8e8        7 weeks ago         1.22 GB
debian                 jessie              8cedef9d7368        2 months ago        123 MB
tylerfowler/superset   latest              224639f0ff97        4 months ago        879 MB
uifd/ui-for-docker     latest              965940f98fa5        8 months ago        8.1 MB
cloudera/clusterdock   latest              3e15a0e12577        8 months ago        463 MB
zhicwu/presto          latest              a1fe6f0241a4        12 months ago       2.05 GB
cloudera/quickstart    latest              4239cd2958c6        13 months ago       6.34 GB

Stop all docker containers - Bash


In [ ]:
#!/bin/bash
# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q)

In [ ]:
# Delete all stopped containers
docker rm $( docker ps -q -f status=exited)
# Delete all dangling (unused) images
docker rmi $( docker images -q -f dangling=true)

"Login" into a container


In [ ]:
docker exec -it <containerIdOrName> bash
( Docker version 1.3 or greater)

In [ ]:
docker network ls

In [ ]:
FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04

Verbatim bash script


In [ ]:
shlomokashani@tf-docker-meetup:~/dev/deep-ml-meetups/nice-docker$ history
    1  mkdir dev
    2  cd dev/
    3  apt-get install docker.io
    4  sudo apt-get install docker.io
    5  git clone git@github.com:QuantScientist/deep-ml-meetups.git
    6  ssh-keygen -t rsa -b 4096 -C "shlomokashani@gmail.com"
    7  cat /home/shlomokashani/.ssh/id_rsa.pub
    8  ssh-keygen -t rsa -b 4096 -C "shlomokashani@gmail.com"
    9  git clone git@github.com:QuantScientist/deep-ml-meetups.git
   10  cd deep-ml-meetups/nice-docker/
   11  docker build -t quantscientist/deep-ml-meetups -f Dockerfile.cpu .
   12  sudo docker build -t quantscientist/deep-ml-meetups -f Dockerfile.cpu .
   13  docker ps
   14  sudo usermod -aG docker shlomokashani
   15  history

In [ ]:
docker run -it -p 5555:5555 -p 7842:7842 -p 8787:8787 -p 8786:8786 -p 8788:8788 -v ~/dev/:/root/sharedfolder  quantscientist/deep-ml-meetups bash

Data Lab - ON GOOGLE


In [ ]:
shlomokashani@cloudshell:~$ gcloud config set project tf-docker
Updated property [core/project].

shlomokashani@tf-docker:~$ datalab create tf-docker-datalab

Data Lab - LOCAL


In [ ]:
git clone https://github.com/GoogleCloudPlatform/datalab.git
cd datalab/containers/datalab
# Replace the MyProjectID value in the next line with your project ID
PROJECT_ID=MyProjectID
./build.sh && ./run.sh