Autoencoders

I've been exploring how useful autoencoders are and how painfully simple they are to implement in Keras. In this post, my goal is to better understand them myself, so I borrow heavily from the Keras blog on the same topic. So rather than sprinkling references to the Keras blog throughout the post, just assume I borrowed it from Francois Chollet. Thanks to Francois for making his code available!

For instance, I thought about drawing a diagram overviewing autoencoders, but it's hard to beat the effective simplicity of this diagram.

So, autoencoders are legit. They perform data compression but not in the JPEG or MPEG way, which make some broad assumptions about images, sound, and video and apply compression based on the assumptions. Instead, autoencoders learn (automatically) a lossy compression based on the data examples fed in. So the compression is specific to those examples.

What's Required

Autoencoders require 3 things:

  1. Encoding function
  2. Decoding function
  3. Loss function describing the amount of information loss between the compressed and decompressed representations of the data examples and the decompressed representation (i.e. a "loss" function).

The encoding/decoding functions are typically (parametric) neural nets and are differentiable with respect to the distance function. The differentiable part enables optimizing the parameters of the encoding/decoding functions to minimize the reconstruction loss.

What Are They Good For

  1. Data Denoising
  2. Dimension Reduction
  3. Data Visualization (basically the same as 2, but plots)

For data denoising, think PCA, but nonlinear. In fact, if the encoder/decoder functions are linear, the result spans the space of the PCA solution. The nonlinear part is useful because they can capture, for example, multimodality in the feature space, which PCA can't.

Dimension reduction is a direct result of the lossy compression of the algorithm. It can help with denoising and pre-training before building another ML algorithm. But is the compression good enough to replace JPEG or MPEG? Possibly. Check out this post based on a recent paper.

But this post is not about the cutting edge stuff. Instead, we're going to focus on more of the basics and do the following:

  • Simple Autoencoder
  • Deep Autoencoder
  • Convolution Autoencoder
  • Build a Second Convolution Autoencoder to Denoise Images

Data Loading and Preprocessing

For this post, I'm going to use the MNIST data set. To get started, let's start with the boilerplate imports.


In [1]:
from IPython.display import Image, SVG
import matplotlib.pyplot as plt

%matplotlib inline

import numpy as np
import keras
from keras.datasets import mnist
from keras.models import Model, Sequential
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Flatten, Reshape
from keras import regularizers


/Users/ramey/miniconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.

With that out of the way, let's load the MNIST data set and scale the images to a range between 0 and 1. If you haven't already downloaded the data set, the Keras load_data function will download the data directly from S3 on AWS.


In [2]:
# Loads the training and test data sets (ignoring class labels)
(x_train, _), (x_test, _) = mnist.load_data()

# Scales the training and test data to range between 0 and 1.
max_value = float(x_train.max())
x_train = x_train.astype('float32') / max_value
x_test = x_test.astype('float32') / max_value

The data set consists 3D arrays with 60K training and 10K test images. The images have a resolution of 28 x 28 (pixels).


In [3]:
x_train.shape, x_test.shape


Out[3]:
((60000, 28, 28), (10000, 28, 28))

To work with the images as vectors, let's reshape the 3D arrays as matrices. In doing so, we'll reshape the 28 x 28 images into vectors of length 784


In [4]:
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

(x_train.shape, x_test.shape)


Out[4]:
((60000, 784), (10000, 784))

Simple Autoencoder

Let's start with a simple autoencoder for illustration. The encoder and decoder functions are each fully-connected neural layers. The encoder function uses a ReLU activation function), while the decoder function uses a sigmoid activation function.

So what are the encoder and the decoder layers doing?

  • The encoder layer "encodes" the input image as a compressed representation in a reduced dimension. The compressed image typically looks garbled, nothing like the original image.
  • The decoder layer "decodes" the encoded image back to the original dimension. The decoded image is a lossy reconstruction of the original image.

In our example, the compressed image has a dimension of 32. The encoder model reduces the dimension from the original 784-dimensional vector to the encoded 32-dimensional vector. The decoder model restores the dimension from the encoded 32-dimensional representation back to the original 784-dimensional vector.

The compression factor is the ratio of the input dimension to the encoded dimension. In our case, the factor is 24.5 = 784 / 32.

The autoencoder model maps an input image to its reconstructed image.


In [5]:
# input dimension = 784
input_dim = x_train.shape[1]
encoding_dim = 32

compression_factor = float(input_dim) / encoding_dim
print("Compression factor: %s" % compression_factor)

autoencoder = Sequential()
autoencoder.add(
    Dense(encoding_dim, input_shape=(input_dim,), activation='relu')
)
autoencoder.add(
    Dense(input_dim, activation='sigmoid')
)

autoencoder.summary()


Compression factor: 24.5
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 32)                25120     
_________________________________________________________________
dense_2 (Dense)              (None, 784)               25872     
=================================================================
Total params: 50,992
Trainable params: 50,992
Non-trainable params: 0
_________________________________________________________________

Encoder Model

We can extract the encoder model from the first layer of the autoencoder model. The reason we want to extract the encoder model is to examine what an encoded image looks like.


In [6]:
input_img = Input(shape=(input_dim,))
encoder_layer = autoencoder.layers[0]
encoder = Model(input_img, encoder_layer(input_img))

encoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 784)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 32)                25120     
=================================================================
Total params: 25,120
Trainable params: 25,120
Non-trainable params: 0
_________________________________________________________________

Okay, now we're ready to train our first autoencoder. We'll iterate on the training data in batches of 256 in 50 epochs. Let's also use the Adam optimizer and per-pixel binary crossentropy loss. The purpose of the loss function is to reconstruct an image similar to the input image.

I want to call out something that may look like a typo or may not be obvious at first glance. Notice the repeat of x_train in autoencoder.fit(x_train, x_train, ...). This implies that x_train is both the input and output, which is exactly what we want for image reconstruction.

I'm running this code on a laptop, so you'll notice the training times are a bit slow (no GPU).


In [7]:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
                epochs=50,
                batch_size=256,
                shuffle=True,
                validation_data=(x_test, x_test))


Train on 60000 samples, validate on 10000 samples
Epoch 1/50
60000/60000 [==============================] - 2s 35us/step - loss: 0.2786 - val_loss: 0.1903
Epoch 2/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1713 - val_loss: 0.1543
Epoch 3/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1449 - val_loss: 0.1342
Epoch 4/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1291 - val_loss: 0.1220
Epoch 5/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1190 - val_loss: 0.1133
Epoch 6/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1115 - val_loss: 0.1071
Epoch 7/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1060 - val_loss: 0.1025
Epoch 8/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.1020 - val_loss: 0.0992
Epoch 9/50
60000/60000 [==============================] - 2s 33us/step - loss: 0.0993 - val_loss: 0.0971
Epoch 10/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0975 - val_loss: 0.0955
Epoch 11/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0962 - val_loss: 0.0945
Epoch 12/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0954 - val_loss: 0.0939
Epoch 13/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0949 - val_loss: 0.0934
Epoch 14/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0945 - val_loss: 0.0931
Epoch 15/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0942 - val_loss: 0.0928
Epoch 16/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0940 - val_loss: 0.0926
Epoch 17/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0939 - val_loss: 0.0925
Epoch 18/50
60000/60000 [==============================] - 2s 33us/step - loss: 0.0937 - val_loss: 0.0923
Epoch 19/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0936 - val_loss: 0.0923
Epoch 20/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0935 - val_loss: 0.0922
Epoch 21/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0934 - val_loss: 0.0921
Epoch 22/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0933 - val_loss: 0.0921
Epoch 23/50
60000/60000 [==============================] - 2s 34us/step - loss: 0.0933 - val_loss: 0.0920
Epoch 24/50
60000/60000 [==============================] - 2s 32us/step - loss: 0.0932 - val_loss: 0.0920
Epoch 25/50
60000/60000 [==============================] - 2s 35us/step - loss: 0.0932 - val_loss: 0.0919
Epoch 26/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0931 - val_loss: 0.0919
Epoch 27/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0931 - val_loss: 0.0919
Epoch 28/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0930 - val_loss: 0.0918
Epoch 29/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0930 - val_loss: 0.0918
Epoch 30/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0930 - val_loss: 0.0918
Epoch 31/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0929 - val_loss: 0.0917
Epoch 32/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0929 - val_loss: 0.0917
Epoch 33/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0929 - val_loss: 0.0917
Epoch 34/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0929 - val_loss: 0.0917
Epoch 35/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0928 - val_loss: 0.0917
Epoch 36/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0928 - val_loss: 0.0916
Epoch 37/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0928 - val_loss: 0.0916
Epoch 38/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0928 - val_loss: 0.0917
Epoch 39/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0928 - val_loss: 0.0916
Epoch 40/50
60000/60000 [==============================] - 2s 38us/step - loss: 0.0928 - val_loss: 0.0918
Epoch 41/50
60000/60000 [==============================] - 2s 38us/step - loss: 0.0928 - val_loss: 0.0916
Epoch 42/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0916
Epoch 43/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0916
Epoch 44/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0916
Epoch 45/50
60000/60000 [==============================] - 2s 36us/step - loss: 0.0927 - val_loss: 0.0916
Epoch 46/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0915
Epoch 47/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0915
Epoch 48/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0915
Epoch 49/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0927 - val_loss: 0.0915
Epoch 50/50
60000/60000 [==============================] - 2s 37us/step - loss: 0.0926 - val_loss: 0.0916
Out[7]:
<keras.callbacks.History at 0x181b5c4cf8>

We've successfully trained our first autoencoder. With a mere 50,992 parameters, our autoencoder model can compress an MNIST digit down to 32 floating-point digits. Not that impressive, but it works.

To check out the encoded images and the reconstructed image quality, we randomly sample 10 test images. I really like how the encoded images look. Do they make sense? No. Are they eye candy though? Most definitely.

However, the reconstructed images are quite lossy. You can see the digits clearly, but notice the loss in image quality.


In [8]:
num_images = 10
np.random.seed(42)
random_test_images = np.random.randint(x_test.shape[0], size=num_images)

encoded_imgs = encoder.predict(x_test)
decoded_imgs = autoencoder.predict(x_test)

plt.figure(figsize=(18, 4))

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(3, num_images, i + 1)
    plt.imshow(x_test[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    
    # plot encoded image
    ax = plt.subplot(3, num_images, num_images + i + 1)
    plt.imshow(encoded_imgs[image_idx].reshape(8, 4))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # plot reconstructed image
    ax = plt.subplot(3, num_images, 2*num_images + i + 1)
    plt.imshow(decoded_imgs[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()



Deep Autoencoder

Above, we used single fully-connected layers for both the encoding and decoding models. Instead, we can stack multiple fully-connected layers to make each of the encoder and decoder functions deep. You know because deep learning.

In this next model, we'll use 3 fully-connected layers for the encoding model with decreasing dimensions from 128 to 64 32 again. Likewise, we'll add 3 fully-connected decoder layers that reconstruct the image back to 784 dimensions. Except for the last layer, we'll use ReLU activation functions again.

In Keras, this model is painfully simple to do, so let's get started. We'll use the same training configuration: Adam + 50 epochs + batch size of 256.


In [9]:
autoencoder = Sequential()

# Encoder Layers
autoencoder.add(Dense(4 * encoding_dim, input_shape=(input_dim,), activation='relu'))
autoencoder.add(Dense(2 * encoding_dim, activation='relu'))
autoencoder.add(Dense(encoding_dim, activation='relu'))

# Decoder Layers
autoencoder.add(Dense(2 * encoding_dim, activation='relu'))
autoencoder.add(Dense(4 * encoding_dim, activation='relu'))
autoencoder.add(Dense(input_dim, activation='sigmoid'))

autoencoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_3 (Dense)              (None, 128)               100480    
_________________________________________________________________
dense_4 (Dense)              (None, 64)                8256      
_________________________________________________________________
dense_5 (Dense)              (None, 32)                2080      
_________________________________________________________________
dense_6 (Dense)              (None, 64)                2112      
_________________________________________________________________
dense_7 (Dense)              (None, 128)               8320      
_________________________________________________________________
dense_8 (Dense)              (None, 784)               101136    
=================================================================
Total params: 222,384
Trainable params: 222,384
Non-trainable params: 0
_________________________________________________________________

Encoder Model

Like we did above, we can extract the encoder model from the autoencoder. The encoder model consists of the first 3 layers in the autoencoder, so let's extract them to visualize the encoded images.


In [10]:
input_img = Input(shape=(input_dim,))
encoder_layer1 = autoencoder.layers[0]
encoder_layer2 = autoencoder.layers[1]
encoder_layer3 = autoencoder.layers[2]
encoder = Model(input_img, encoder_layer3(encoder_layer2(encoder_layer1(input_img))))

encoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 784)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 128)               100480    
_________________________________________________________________
dense_4 (Dense)              (None, 64)                8256      
_________________________________________________________________
dense_5 (Dense)              (None, 32)                2080      
=================================================================
Total params: 110,816
Trainable params: 110,816
Non-trainable params: 0
_________________________________________________________________

In [11]:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
                epochs=50,
                batch_size=256,
                validation_data=(x_test, x_test))


Train on 60000 samples, validate on 10000 samples
Epoch 1/50
60000/60000 [==============================] - 4s 59us/step - loss: 0.2502 - val_loss: 0.1741
Epoch 2/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1580 - val_loss: 0.1422
Epoch 3/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1357 - val_loss: 0.1275
Epoch 4/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1245 - val_loss: 0.1190
Epoch 5/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1178 - val_loss: 0.1137
Epoch 6/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1133 - val_loss: 0.1104
Epoch 7/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1098 - val_loss: 0.1070
Epoch 8/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1068 - val_loss: 0.1042
Epoch 9/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1042 - val_loss: 0.1018
Epoch 10/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.1018 - val_loss: 0.0997
Epoch 11/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0999 - val_loss: 0.0980
Epoch 12/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0982 - val_loss: 0.0966
Epoch 13/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0967 - val_loss: 0.0949
Epoch 14/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0956 - val_loss: 0.0941
Epoch 15/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0947 - val_loss: 0.0935
Epoch 16/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0939 - val_loss: 0.0929
Epoch 17/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0933 - val_loss: 0.0921
Epoch 18/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0927 - val_loss: 0.0915
Epoch 19/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0921 - val_loss: 0.0911
Epoch 20/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0917 - val_loss: 0.0905
Epoch 21/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0912 - val_loss: 0.0904
Epoch 22/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0907 - val_loss: 0.0902
Epoch 23/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0903 - val_loss: 0.0892
Epoch 24/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0898 - val_loss: 0.0889
Epoch 25/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0895 - val_loss: 0.0887
Epoch 26/50
60000/60000 [==============================] - 3s 53us/step - loss: 0.0892 - val_loss: 0.0887
Epoch 27/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0890 - val_loss: 0.0880
Epoch 28/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0886 - val_loss: 0.0879
Epoch 29/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0883 - val_loss: 0.0876
Epoch 30/50
60000/60000 [==============================] - 3s 56us/step - loss: 0.0882 - val_loss: 0.0875
Epoch 31/50
60000/60000 [==============================] - 3s 57us/step - loss: 0.0879 - val_loss: 0.0871
Epoch 32/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0876 - val_loss: 0.0869
Epoch 33/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0875 - val_loss: 0.0867
Epoch 34/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0872 - val_loss: 0.0866
Epoch 35/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0870 - val_loss: 0.0866
Epoch 36/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0868 - val_loss: 0.0862
Epoch 37/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0867 - val_loss: 0.0860
Epoch 38/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0865 - val_loss: 0.0859
Epoch 39/50
60000/60000 [==============================] - 3s 56us/step - loss: 0.0864 - val_loss: 0.0858
Epoch 40/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0862 - val_loss: 0.0856
Epoch 41/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0861 - val_loss: 0.0856
Epoch 42/50
60000/60000 [==============================] - 3s 56us/step - loss: 0.0859 - val_loss: 0.0856
Epoch 43/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0859 - val_loss: 0.0853
Epoch 44/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0857 - val_loss: 0.0854
Epoch 45/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0855 - val_loss: 0.0851
Epoch 46/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0854 - val_loss: 0.0850
Epoch 47/50
60000/60000 [==============================] - 3s 56us/step - loss: 0.0853 - val_loss: 0.0849
Epoch 48/50
60000/60000 [==============================] - 3s 54us/step - loss: 0.0852 - val_loss: 0.0848
Epoch 49/50
60000/60000 [==============================] - 4s 59us/step - loss: 0.0851 - val_loss: 0.0848
Epoch 50/50
60000/60000 [==============================] - 3s 55us/step - loss: 0.0850 - val_loss: 0.0845
Out[11]:
<keras.callbacks.History at 0x181e2eb940>

As with the simple autoencoder, we randomly sample 10 test images (the same ones as before). The reconstructed digits look much better than those from the single-layer autoencoder. This observation aligns with the reduction in validation loss after adding multiple layers to the autoencoder.


In [12]:
num_images = 10
np.random.seed(42)
random_test_images = np.random.randint(x_test.shape[0], size=num_images)

encoded_imgs = encoder.predict(x_test)
decoded_imgs = autoencoder.predict(x_test)

plt.figure(figsize=(18, 4))

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(3, num_images, i + 1)
    plt.imshow(x_test[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    
    # plot encoded image
    ax = plt.subplot(3, num_images, num_images + i + 1)
    plt.imshow(encoded_imgs[image_idx].reshape(8, 4))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # plot reconstructed image
    ax = plt.subplot(3, num_images, 2*num_images + i + 1)
    plt.imshow(decoded_imgs[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()



Convolutional Autoencoder

Now that we've explored deep autoencoders, let's use a convolutional autoencoder instead, given that the input objects are images. What this means is our encoding and decoding models will be convolutional neural networks instead of fully-connected networks.

Again, Keras makes this very easy for us. Before we get started though, we need to reshapes the images back to 28 x 28 x 1 for the convnets. The 1 is for 1 channel because black and white. If we had RGB color, there would be 3 channels.


In [13]:
x_train = x_train.reshape((len(x_train), 28, 28, 1))
x_test = x_test.reshape((len(x_test), 28, 28, 1))

To build the convolutional autoencoder, we'll make use of Conv2D and MaxPooling2D layers for the encoder and Conv2D and UpSampling2D layers for the decoder. The encoded images are transformed to a 3D array of dimensions 4 x 4 x 8, but to visualize the encoding, we'll flatten it to a vector of length 128. I tried to use an encoding dimension of 32 like above, but I kept getting subpar results.

After the flattening layer, we reshape the image back to a 4 x 4 x 8 array before upsampling back to a 28 x 28 x 1 image.


In [14]:
autoencoder = Sequential()

# Encoder Layers
autoencoder.add(Conv2D(16, (3, 3), activation='relu', padding='same', input_shape=x_train.shape[1:]))
autoencoder.add(MaxPooling2D((2, 2), padding='same'))
autoencoder.add(Conv2D(8, (3, 3), activation='relu', padding='same'))
autoencoder.add(MaxPooling2D((2, 2), padding='same'))
autoencoder.add(Conv2D(8, (3, 3), strides=(2,2), activation='relu', padding='same'))

# Flatten encoding for visualization
autoencoder.add(Flatten())
autoencoder.add(Reshape((4, 4, 8)))

# Decoder Layers
autoencoder.add(Conv2D(8, (3, 3), activation='relu', padding='same'))
autoencoder.add(UpSampling2D((2, 2)))
autoencoder.add(Conv2D(8, (3, 3), activation='relu', padding='same'))
autoencoder.add(UpSampling2D((2, 2)))
autoencoder.add(Conv2D(16, (3, 3), activation='relu'))
autoencoder.add(UpSampling2D((2, 2)))
autoencoder.add(Conv2D(1, (3, 3), activation='sigmoid', padding='same'))

autoencoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 28, 28, 16)        160       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 8)         1160      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 8)           0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 4, 4, 8)           584       
_________________________________________________________________
flatten_1 (Flatten)          (None, 128)               0         
_________________________________________________________________
reshape_1 (Reshape)          (None, 4, 4, 8)           0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 4, 4, 8)           584       
_________________________________________________________________
up_sampling2d_1 (UpSampling2 (None, 8, 8, 8)           0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 8, 8, 8)           584       
_________________________________________________________________
up_sampling2d_2 (UpSampling2 (None, 16, 16, 8)         0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 14, 14, 16)        1168      
_________________________________________________________________
up_sampling2d_3 (UpSampling2 (None, 28, 28, 16)        0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 28, 28, 1)         145       
=================================================================
Total params: 4,385
Trainable params: 4,385
Non-trainable params: 0
_________________________________________________________________

Encoder Model

To extract the encoder model for the autoencoder, we're going to use a slightly different approach than before. Rather than extracting the first 6 layers, we're going to create a new Model with the same input as the autoencoder, but the output will be that of the flattening layer. As a side note, this is a very useful technique for grabbing submodels for things like transfer learning.

As I mentioned before, the encoded image is a vector of length 128.


In [15]:
encoder = Model(inputs=autoencoder.input, outputs=autoencoder.get_layer('flatten_1').output)
encoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1_input (InputLayer)  (None, 28, 28, 1)         0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 28, 28, 16)        160       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 8)         1160      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 8)           0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 4, 4, 8)           584       
_________________________________________________________________
flatten_1 (Flatten)          (None, 128)               0         
=================================================================
Total params: 1,904
Trainable params: 1,904
Non-trainable params: 0
_________________________________________________________________

In [16]:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
                epochs=100,
                batch_size=128,
                validation_data=(x_test, x_test))


Train on 60000 samples, validate on 10000 samples
Epoch 1/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.1986 - val_loss: 0.1361
Epoch 2/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.1252 - val_loss: 0.1153
Epoch 3/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.1128 - val_loss: 0.1081
Epoch 4/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.1072 - val_loss: 0.1039
Epoch 5/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.1033 - val_loss: 0.1004
Epoch 6/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.1004 - val_loss: 0.0979
Epoch 7/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0983 - val_loss: 0.0963
Epoch 8/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0966 - val_loss: 0.0944
Epoch 9/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0952 - val_loss: 0.0933
Epoch 10/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0940 - val_loss: 0.0921
Epoch 11/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0929 - val_loss: 0.0912
Epoch 12/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0921 - val_loss: 0.0906
Epoch 13/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0912 - val_loss: 0.0896
Epoch 14/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0905 - val_loss: 0.0891
Epoch 15/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0899 - val_loss: 0.0885
Epoch 16/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0894 - val_loss: 0.0884
Epoch 17/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0888 - val_loss: 0.0875
Epoch 18/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0884 - val_loss: 0.0871
Epoch 19/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0880 - val_loss: 0.0868
Epoch 20/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0876 - val_loss: 0.0863
Epoch 21/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0872 - val_loss: 0.0861
Epoch 22/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0869 - val_loss: 0.0859
Epoch 23/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0867 - val_loss: 0.0855
Epoch 24/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0864 - val_loss: 0.0855
Epoch 25/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0862 - val_loss: 0.0853
Epoch 26/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0860 - val_loss: 0.0854
Epoch 27/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0858 - val_loss: 0.0848
Epoch 28/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0856 - val_loss: 0.0846
Epoch 29/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0855 - val_loss: 0.0845
Epoch 30/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0853 - val_loss: 0.0842
Epoch 31/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0852 - val_loss: 0.0843
Epoch 32/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0851 - val_loss: 0.0843
Epoch 33/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0849 - val_loss: 0.0841
Epoch 34/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0848 - val_loss: 0.0837
Epoch 35/100
60000/60000 [==============================] - 72s 1ms/step - loss: 0.0847 - val_loss: 0.0837
Epoch 36/100
60000/60000 [==============================] - 70s 1ms/step - loss: 0.0846 - val_loss: 0.0838
Epoch 37/100
60000/60000 [==============================] - 71s 1ms/step - loss: 0.0845 - val_loss: 0.0834
Epoch 38/100
60000/60000 [==============================] - 70s 1ms/step - loss: 0.0843 - val_loss: 0.0833
Epoch 39/100
60000/60000 [==============================] - 2570s 43ms/step - loss: 0.0842 - val_loss: 0.0832
Epoch 40/100
60000/60000 [==============================] - 80s 1ms/step - loss: 0.0841 - val_loss: 0.0830
Epoch 41/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0840 - val_loss: 0.0831
Epoch 42/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0840 - val_loss: 0.0832
Epoch 43/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0838 - val_loss: 0.0828
Epoch 44/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0837 - val_loss: 0.0826
Epoch 45/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0836 - val_loss: 0.0826
Epoch 46/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0836 - val_loss: 0.0825
Epoch 47/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0835 - val_loss: 0.0825
Epoch 48/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0834 - val_loss: 0.0824
Epoch 49/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0833 - val_loss: 0.0831
Epoch 50/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0832 - val_loss: 0.0825
Epoch 51/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0832 - val_loss: 0.0828
Epoch 52/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0831 - val_loss: 0.0824
Epoch 53/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0830 - val_loss: 0.0821
Epoch 54/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0829 - val_loss: 0.0819
Epoch 55/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0829 - val_loss: 0.0821
Epoch 56/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0828 - val_loss: 0.0819
Epoch 57/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0828 - val_loss: 0.0821
Epoch 58/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0826 - val_loss: 0.0818
Epoch 59/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0826 - val_loss: 0.0817
Epoch 60/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0825 - val_loss: 0.0816
Epoch 61/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0825 - val_loss: 0.0816
Epoch 62/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0824 - val_loss: 0.0817
Epoch 63/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0824 - val_loss: 0.0817
Epoch 64/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0823 - val_loss: 0.0814
Epoch 65/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0822 - val_loss: 0.0813
Epoch 66/100
60000/60000 [==============================] - 65s 1ms/step - loss: 0.0822 - val_loss: 0.0812
Epoch 67/100
60000/60000 [==============================] - 64s 1ms/step - loss: 0.0821 - val_loss: 0.0812
Epoch 68/100
60000/60000 [==============================] - 64s 1ms/step - loss: 0.0821 - val_loss: 0.0811
Epoch 69/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0820 - val_loss: 0.0811
Epoch 70/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0820 - val_loss: 0.0811
Epoch 71/100
60000/60000 [==============================] - 71s 1ms/step - loss: 0.0819 - val_loss: 0.0809
Epoch 72/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0819 - val_loss: 0.0813
Epoch 73/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0819 - val_loss: 0.0809
Epoch 74/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0818 - val_loss: 0.0808
Epoch 75/100
60000/60000 [==============================] - 70s 1ms/step - loss: 0.0818 - val_loss: 0.0809
Epoch 76/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0817 - val_loss: 0.0807
Epoch 77/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0817 - val_loss: 0.0807
Epoch 78/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0816 - val_loss: 0.0807
Epoch 79/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0816 - val_loss: 0.0809
Epoch 80/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0815 - val_loss: 0.0809
Epoch 81/100
60000/60000 [==============================] - 70s 1ms/step - loss: 0.0815 - val_loss: 0.0805
Epoch 82/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0814 - val_loss: 0.0806
Epoch 83/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0814 - val_loss: 0.0805
Epoch 84/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0813 - val_loss: 0.0805
Epoch 85/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0813 - val_loss: 0.0803
Epoch 86/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0812 - val_loss: 0.0802
Epoch 87/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0812 - val_loss: 0.0803
Epoch 88/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0811 - val_loss: 0.0802
Epoch 89/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0811 - val_loss: 0.0801
Epoch 90/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0811 - val_loss: 0.0801
Epoch 91/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0810 - val_loss: 0.0802
Epoch 92/100
60000/60000 [==============================] - 66s 1ms/step - loss: 0.0810 - val_loss: 0.0805
Epoch 93/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0809 - val_loss: 0.0800
Epoch 94/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0809 - val_loss: 0.0799
Epoch 95/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0809 - val_loss: 0.0803
Epoch 96/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0808 - val_loss: 0.0799
Epoch 97/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0808 - val_loss: 0.0801
Epoch 98/100
60000/60000 [==============================] - 67s 1ms/step - loss: 0.0808 - val_loss: 0.0799
Epoch 99/100
60000/60000 [==============================] - 69s 1ms/step - loss: 0.0807 - val_loss: 0.0798
Epoch 100/100
60000/60000 [==============================] - 68s 1ms/step - loss: 0.0807 - val_loss: 0.0798
Out[16]:
<keras.callbacks.History at 0x182d375320>

The reconstructed digits look even better than before. This is no surprise given an even lower validation loss. Other than slight improved reconstruction, check out how the encoded image has changed. What's even cooler is that the encoded images of the 9 look similar as do those of the 8's. This similarity was far less pronounced for the simple and deep autoencoders.


In [17]:
num_images = 10
np.random.seed(42)
random_test_images = np.random.randint(x_test.shape[0], size=num_images)

encoded_imgs = encoder.predict(x_test)
decoded_imgs = autoencoder.predict(x_test)

plt.figure(figsize=(18, 4))

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(3, num_images, i + 1)
    plt.imshow(x_test[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    
    # plot encoded image
    ax = plt.subplot(3, num_images, num_images + i + 1)
    plt.imshow(encoded_imgs[image_idx].reshape(16, 8))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # plot reconstructed image
    ax = plt.subplot(3, num_images, 2*num_images + i + 1)
    plt.imshow(decoded_imgs[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


Denoising Images with the Convolutional Autoencoder

Earlier, I mentioned that autoencoders are useful for denoising data including images. When I learned about this concept in grad school, my mind was blown. This simple task helped me realize data can be manipulated in very useful ways and that the dirty data we often inherit can be cleansed using more advanced techniques.

With that in mind, let's add bit of noise to the test images and see how good the convolutional autoencoder is at removing the noise.


In [18]:
x_train_noisy = x_train + np.random.normal(loc=0.0, scale=0.5, size=x_train.shape)
x_train_noisy = np.clip(x_train_noisy, 0., 1.)

x_test_noisy = x_test + np.random.normal(loc=0.0, scale=0.5, size=x_test.shape)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

num_images = 10
np.random.seed(42)
random_test_images = np.random.randint(x_test.shape[0], size=num_images)

# Denoise test images
x_test_denoised = autoencoder.predict(x_test_noisy)

plt.figure(figsize=(18, 4))

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(2, num_images, i + 1)
    plt.imshow(x_test_noisy[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    
    # plot reconstructed image
    ax = plt.subplot(2, num_images, num_images + i + 1)
    plt.imshow(x_test_denoised[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


Convolutional Autoencoder - Take 2

Well, those images are terrible. They remind me of the mask from the movie Scream.

Okay, so let's try that again. This time we're going to build a ConvNet with a lot more parameters and forego visualizing the encoding layer. The network will be a bit larger and slower to train, but the results are definitely worth the effort.

One more thing: this time, let's use (x_train_noisy, x_train) as training data and (x_test_noisy, x_test) as validation data.


In [21]:
autoencoder = Sequential()

# Encoder Layers
autoencoder.add(Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=x_train.shape[1:]))
autoencoder.add(MaxPooling2D((2, 2), padding='same'))
autoencoder.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
autoencoder.add(MaxPooling2D((2, 2), padding='same'))

# Decoder Layers
autoencoder.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
autoencoder.add(UpSampling2D((2, 2)))
autoencoder.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
autoencoder.add(UpSampling2D((2, 2)))
autoencoder.add(Conv2D(1, (3, 3), activation='sigmoid', padding='same'))

autoencoder.summary()


_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_13 (Conv2D)           (None, 28, 28, 32)        320       
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 14, 14, 32)        9248      
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 7, 7, 32)          0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 7, 7, 32)          9248      
_________________________________________________________________
up_sampling2d_6 (UpSampling2 (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 14, 14, 32)        9248      
_________________________________________________________________
up_sampling2d_7 (UpSampling2 (None, 28, 28, 32)        0         
_________________________________________________________________
conv2d_17 (Conv2D)           (None, 28, 28, 1)         289       
=================================================================
Total params: 28,353
Trainable params: 28,353
Non-trainable params: 0
_________________________________________________________________

In [22]:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x_train_noisy, x_train,
                epochs=100,
                batch_size=128,
                validation_data=(x_test_noisy, x_test))


Train on 60000 samples, validate on 10000 samples
Epoch 1/100
60000/60000 [==============================] - 140s 2ms/step - loss: 0.1661 - val_loss: 0.1143
Epoch 2/100
60000/60000 [==============================] - 132s 2ms/step - loss: 0.1113 - val_loss: 0.1068
Epoch 3/100
60000/60000 [==============================] - 133s 2ms/step - loss: 0.1062 - val_loss: 0.1038
Epoch 4/100
60000/60000 [==============================] - 131s 2ms/step - loss: 0.1036 - val_loss: 0.1017
Epoch 5/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.1018 - val_loss: 0.1001
Epoch 6/100
60000/60000 [==============================] - 121s 2ms/step - loss: 0.1005 - val_loss: 0.0992
Epoch 7/100
60000/60000 [==============================] - 121s 2ms/step - loss: 0.0996 - val_loss: 0.0984
Epoch 8/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0989 - val_loss: 0.0980
Epoch 9/100
60000/60000 [==============================] - 128s 2ms/step - loss: 0.0983 - val_loss: 0.0972
Epoch 10/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0978 - val_loss: 0.0971
Epoch 11/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0974 - val_loss: 0.0965
Epoch 12/100
60000/60000 [==============================] - 120s 2ms/step - loss: 0.0970 - val_loss: 0.0962
Epoch 13/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0967 - val_loss: 0.0959
Epoch 14/100
60000/60000 [==============================] - 121s 2ms/step - loss: 0.0964 - val_loss: 0.0958
Epoch 15/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0962 - val_loss: 0.0964
Epoch 16/100
60000/60000 [==============================] - 122s 2ms/step - loss: 0.0960 - val_loss: 0.0952
Epoch 17/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0957 - val_loss: 0.0952
Epoch 18/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0956 - val_loss: 0.0953
Epoch 19/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0954 - val_loss: 0.0949
Epoch 20/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0953 - val_loss: 0.0946
Epoch 21/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0952 - val_loss: 0.0946
Epoch 22/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0951 - val_loss: 0.0944
Epoch 23/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0949 - val_loss: 0.0944
Epoch 24/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0948 - val_loss: 0.0945
Epoch 25/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0947 - val_loss: 0.0942
Epoch 26/100
60000/60000 [==============================] - 117s 2ms/step - loss: 0.0946 - val_loss: 0.0941
Epoch 27/100
60000/60000 [==============================] - 115s 2ms/step - loss: 0.0946 - val_loss: 0.0941
Epoch 28/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0945 - val_loss: 0.0940
Epoch 29/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0944 - val_loss: 0.0942
Epoch 30/100
60000/60000 [==============================] - 116s 2ms/step - loss: 0.0943 - val_loss: 0.0939
Epoch 31/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0943 - val_loss: 0.0938
Epoch 32/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0942 - val_loss: 0.0938
Epoch 33/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0942 - val_loss: 0.0937
Epoch 34/100
60000/60000 [==============================] - 120s 2ms/step - loss: 0.0941 - val_loss: 0.0936
Epoch 35/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0941 - val_loss: 0.0936
Epoch 36/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0940 - val_loss: 0.0937
Epoch 37/100
60000/60000 [==============================] - 132s 2ms/step - loss: 0.0940 - val_loss: 0.0937
Epoch 38/100
60000/60000 [==============================] - 126s 2ms/step - loss: 0.0940 - val_loss: 0.0936
Epoch 39/100
60000/60000 [==============================] - 131s 2ms/step - loss: 0.0939 - val_loss: 0.0936
Epoch 40/100
60000/60000 [==============================] - 129s 2ms/step - loss: 0.0939 - val_loss: 0.0934
Epoch 41/100
60000/60000 [==============================] - 137s 2ms/step - loss: 0.0938 - val_loss: 0.0934
Epoch 42/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0938 - val_loss: 0.0936
Epoch 43/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0938 - val_loss: 0.0934
Epoch 44/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0937 - val_loss: 0.0936
Epoch 45/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0937 - val_loss: 0.0934
Epoch 46/100
60000/60000 [==============================] - 123s 2ms/step - loss: 0.0937 - val_loss: 0.0935
Epoch 47/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0937 - val_loss: 0.0932
Epoch 48/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0937 - val_loss: 0.0936
Epoch 49/100
60000/60000 [==============================] - 125s 2ms/step - loss: 0.0936 - val_loss: 0.0934
Epoch 50/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0936 - val_loss: 0.0932
Epoch 51/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0936 - val_loss: 0.0932
Epoch 52/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0935 - val_loss: 0.0932
Epoch 53/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0935 - val_loss: 0.0932
Epoch 54/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0935 - val_loss: 0.0932
Epoch 55/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0935 - val_loss: 0.0940
Epoch 56/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0935 - val_loss: 0.0932
Epoch 57/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0934 - val_loss: 0.0931
Epoch 58/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0934 - val_loss: 0.0932
Epoch 59/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0934 - val_loss: 0.0932
Epoch 60/100
60000/60000 [==============================] - 117s 2ms/step - loss: 0.0934 - val_loss: 0.0938
Epoch 61/100
60000/60000 [==============================] - 117s 2ms/step - loss: 0.0934 - val_loss: 0.0930
Epoch 62/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0933 - val_loss: 0.0930
Epoch 63/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0934 - val_loss: 0.0932
Epoch 64/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0933 - val_loss: 0.0933
Epoch 65/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0933 - val_loss: 0.0930
Epoch 66/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0933 - val_loss: 0.0931
Epoch 67/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0933 - val_loss: 0.0933
Epoch 68/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0933 - val_loss: 0.0933
Epoch 69/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0932 - val_loss: 0.0932
Epoch 70/100
60000/60000 [==============================] - 114s 2ms/step - loss: 0.0932 - val_loss: 0.0930
Epoch 71/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0931
Epoch 72/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0938
Epoch 73/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0937
Epoch 74/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0932 - val_loss: 0.0932
Epoch 75/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0929
Epoch 76/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0929
Epoch 77/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 78/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0934
Epoch 79/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0932 - val_loss: 0.0929
Epoch 80/100
60000/60000 [==============================] - 117s 2ms/step - loss: 0.0932 - val_loss: 0.0930
Epoch 81/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 82/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 83/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0931 - val_loss: 0.0931
Epoch 84/100
60000/60000 [==============================] - 119s 2ms/step - loss: 0.0931 - val_loss: 0.0930
Epoch 85/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 86/100
60000/60000 [==============================] - 118s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 87/100
60000/60000 [==============================] - 120s 2ms/step - loss: 0.0930 - val_loss: 0.0929
Epoch 88/100
60000/60000 [==============================] - 128s 2ms/step - loss: 0.0931 - val_loss: 0.0929
Epoch 89/100
60000/60000 [==============================] - 122s 2ms/step - loss: 0.0930 - val_loss: 0.0930
Epoch 90/100
60000/60000 [==============================] - 125s 2ms/step - loss: 0.0930 - val_loss: 0.0931
Epoch 91/100
60000/60000 [==============================] - 128s 2ms/step - loss: 0.0930 - val_loss: 0.0929
Epoch 92/100
60000/60000 [==============================] - 122s 2ms/step - loss: 0.0930 - val_loss: 0.0930
Epoch 93/100
60000/60000 [==============================] - 130s 2ms/step - loss: 0.0930 - val_loss: 0.0928
Epoch 94/100
60000/60000 [==============================] - 125s 2ms/step - loss: 0.0930 - val_loss: 0.0929
Epoch 95/100
60000/60000 [==============================] - 132s 2ms/step - loss: 0.0930 - val_loss: 0.0928
Epoch 96/100
60000/60000 [==============================] - 127s 2ms/step - loss: 0.0930 - val_loss: 0.0928
Epoch 97/100
60000/60000 [==============================] - 126s 2ms/step - loss: 0.0930 - val_loss: 0.0927
Epoch 98/100
60000/60000 [==============================] - 126s 2ms/step - loss: 0.0930 - val_loss: 0.0927
Epoch 99/100
60000/60000 [==============================] - 126s 2ms/step - loss: 0.0930 - val_loss: 0.0928
Epoch 100/100
60000/60000 [==============================] - 124s 2ms/step - loss: 0.0929 - val_loss: 0.0932
Out[22]:
<keras.callbacks.History at 0x182fb17160>

In [23]:
# Denoise test images
x_test_denoised = autoencoder.predict(x_test_noisy)

plt.figure(figsize=(18, 4))

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(2, num_images, i + 1)
    plt.imshow(x_test_noisy[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    
    # plot reconstructed image
    ax = plt.subplot(2, num_images, num_images + i + 1)
    plt.imshow(x_test_denoised[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


Fantastic, those images almost look like the originals.


In [ ]: