Who Am I?

Brian Spiering

What Do I Do?

Professor @

Keras - Neural Networks for humans

A high-level, intuitive API for Deep Learning.

Easy to define neural networks, then automatically handles execution.

A simple, modular interface which allows focus on learning and enables fast experimentation

Goals

General introduction to Deep Learning
Overview of keras library
An end-to-end example in keras

Anti-Goals

Understanding of Deep Learning (there will be no equations)
Building neural networks from scratch
Complete survey of keras library

Deep Learning 101

Deep Learning (DL) are Neural networks (NN) with >1 hidden layer

Neural Networks are Nodes & Edges

Nonlinear function allows learning of nonlinear relationships

Groups of nodes all the way down

Deep Learning isn't magic, it is just very good at finding patterns

Deep Learning has fewer steps than traditional Machine Learning

If you want to follow along…

GitHub repo: bit.ly/pybay-keras

If you want to type along…

Run a local Jupyter Notebook
Binder: In-Browser Jupyter Notebook
: "Google Docs for Jupyter Notebooks"



In [84]:

    
reset -fs



In [85]:

    
import keras



In [86]:

    
# What is the backend / execution engine?



In [87]:

    
keras.backend.backend()









    Out[87]:





'tensorflow'

"An open-source software library for Machine Intelligence"

Numerical computation using data flow graphs.

TensorFlow: A great backend

A very flexible architecture which allows you to do almost any numerical operation.

Then deploy the computation to CPUs or GPUs (one or more) across desktop, cloud, or mobile device.

MNIST handwritten digit database:
The “Hello World!” of Computer Vision



In [88]:

    
# Import data



In [89]:

    
from keras.datasets import mnist



In [90]:

    
# Setup train and test splits



In [91]:

    
(x_train, y_train), (x_test, y_test) = mnist.load_data()



In [92]:

    
from random import randint
from matplotlib import pyplot

%matplotlib inline



In [93]:

    
pyplot.imshow(x_train[randint(0, x_train.shape[0])], cmap='gray_r');

Munge data

Convert image matrix into vector to feed into first layer



In [94]:

    
# Munge Data
# Transform from matrix to vector, cast, and normalize



In [95]:

    
image_size = 784 # 28 x 28

x_train = x_train.reshape(x_train.shape[0], image_size) # Transform from matrix to vector
x_train = x_train.astype('float32') # Cast as 32 bit integers
x_train /= 255 # Normalize inputs from 0-255 to 0.0-1.0

x_test = x_test.reshape(x_test.shape[0], image_size) # Transform from matrix to vector
x_test = x_test.astype('float32') # Cast as 32 bit integers
x_test /= 255 # Normalize inputs from 0-255 to 0.0-1.0



In [96]:

    
# Convert class vectors to binary class matrices



In [97]:

    
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)



In [98]:

    
# Import the most common type of neural network



In [99]:

    
from keras.models import Sequential

RTFM - https://keras.io/layers/



In [100]:

    
# Define model instance



In [101]:

    
model = Sequential()



In [102]:

    
# Import the most common type of network layer, fully interconnected



In [103]:

    
from keras.layers import Dense



In [104]:

    
# Define input layer



In [105]:

    
layer_input = Dense(units=512,            # Number of nodes
                    activation='sigmoid', # The nonlinearity
                    input_shape=(image_size,)) 
model.add(layer_input)



In [106]:

    
# Define another layer



In [107]:

    
model.add(Dense(units=512, activation='sigmoid'))



In [108]:

    
# Define output layers



In [109]:

    
layer_output = Dense(units=10,             # Number of digits (0-9)
                     activation='softmax') # Convert neural activation to probability of category

model.add(layer_output)



In [110]:

    
# Print summary



In [111]:

    
model.summary()









    



_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_9 (Dense)              (None, 512)               401920    
_________________________________________________________________
dense_10 (Dense)             (None, 512)               262656    
_________________________________________________________________
dense_11 (Dense)             (None, 10)                5130      
=================================================================
Total params: 669,706
Trainable params: 669,706
Non-trainable params: 0
_________________________________________________________________



In [112]:

    
# Yes - we compile the model to run it



In [113]:

    
model.compile(loss='categorical_crossentropy', 
              optimizer='sgd',
              metrics=['accuracy'])



In [114]:

    
# Train the model



In [115]:

    
training = model.fit(x_train, 
                     y_train,
                     epochs=5, # Number of passes over complete dataset
                     verbose=True, 
                     validation_split=0.1)









    



Train on 54000 samples, validate on 6000 samples
Epoch 1/5
54000/54000 [==============================] - 15s 285us/step - loss: 2.1522 - acc: 0.3213 - val_loss: 1.8987 - val_acc: 0.5315
Epoch 2/5
54000/54000 [==============================] - 14s 262us/step - loss: 1.5000 - acc: 0.6548 - val_loss: 1.0769 - val_acc: 0.7430
Epoch 3/5
54000/54000 [==============================] - 15s 285us/step - loss: 0.9003 - acc: 0.7860 - val_loss: 0.6709 - val_acc: 0.8560
Epoch 4/5
54000/54000 [==============================] - 14s 266us/step - loss: 0.6515 - acc: 0.8317 - val_loss: 0.5121 - val_acc: 0.8778
Epoch 5/5
54000/54000 [==============================] - 18s 340us/step - loss: 0.5385 - acc: 0.8549 - val_loss: 0.4268 - val_acc: 0.8940



In [116]:

    
# Let's see how well our model performs



In [117]:

    
loss, accuracy = model.evaluate(x_test, 
                                y_test, 
                                verbose=True)
print(f"Test loss: {loss:.3}")
print(f"Test accuracy: {accuracy:.3%}")









    



10000/10000 [==============================] - 1s 106us/step
Test loss: 0.476
Test accuracy: 87.140%

Keras' Other Features

Common built-in functions (e.g., activation functions and optimitizers)
Convolutional neural network (CNN or ConvNet)
Recurrent neural network (RNN) & Long-short term memory (LSTM)
Pre-trained models

Summary

Keras is designed for human beings, not computers.
Easier to try out Deep Learning (focus on the what, not the how).
Simple to define neural networks.

Futher Study - Keras

Keras docs
Keras blog
Keras courses
- edX
- Coursera

Futher Study - Deep Learning

Prerequisites: Linear Algebra, Probability, Machine Learning
fast.ai Course
Deep Learning Book

Bonus Material



In [118]:

    
# reset -fs



In [119]:

    
# from keras import *



In [120]:

    
# whos



In [121]:

    
# from keras.datasets import fashion_mnist



In [122]:

    
# # Setup train and test splits
# (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()



In [123]:

    
# from random import randint
# from matplotlib import pyplot

# %matplotlib inline



In [124]:

    
# pyplot.imshow(x_train[randint(0, x_train.shape[0])], cmap='gray_r');



In [125]:

    
# # Define CNN model

# # Redefine input dimensions to make sure conv works
# img_rows, img_cols = 28, 28
# x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
# x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
# input_shape = (img_rows, img_cols, 1)



In [126]:

    
# import keras



In [127]:

    
# # Convert class vectors to binary class matrices
# y_train = keras.utils.to_categorical(y_train, 10)
# y_test = keras.utils.to_categorical(y_test, 10)



In [128]:

    
# from keras.layers import Conv2D, Dense, Flatten, MaxPooling2D



In [129]:

    
# # Define model
# model = Sequential()
# model.add(Conv2D(32, 
#              kernel_size=(3, 3),
#              activation='sigmoid',
#              input_shape=input_shape))
# model.add(Conv2D(64, (3, 3), activation='sigmoid'))
# model.add(MaxPooling2D(pool_size=(2, 2)))
# model.add(Flatten())
# model.add(Dense(128, activation='sigmoid'))
# model.add(Dense(10, activation='softmax'))



In [130]:

    
# model.compile(loss='categorical_crossentropy', 
#               optimizer='adam',
#               metrics=['accuracy'])



In [131]:

    
# # Define training
# training = model.fit(x_train, 
#                      y_train,
#                      epochs=5,
#                      verbose=True, 
#                      validation_split=0.1)



In [132]:

    
# loss, accuracy = model.evaluate(x_test, 
#                                 y_test, 
#                                 verbose=True)
# print(f"Test loss: {loss:.3}")
# print(f"Test accuracy: {accuracy:.3%}")

What is `keras`?

Keras (κέρας) means horn in Greek.

It is a reference to a literary image from ancient Greek and Latin literature.

First found in the Odyssey, where dream spirits (Oneiroi, singular Oneiros) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn.

It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).

Source