Exercise: Convolution on Images using TensorFlow for Deep Neural Networks (DNN)

This exercise is about learning to apply convolutions on an image using Tensorflow. Idea is to create a weight matrix and apply the function conv2d with 'same' and 'Valid' padding to check the effects on output image.

To give an overview of how the output changes based on convolution parameters, this exercise is designed to build Layer one of Convolution Neural Network (CNN) along with maxpooling and relu functions and Visualize the outputs given an input image.

First, let's import the TensorFlow library and Python dependencies


In [1]:
#Importing
import numpy as np
from scipy import signal
from scipy import misc
import matplotlib.pyplot as plt
from PIL import Image

import tensorflow as tf

In [2]:
import scipy
scipy.__version__


Out[2]:
'0.19.0'

In [3]:
import PIL
PIL.__version__


Out[3]:
'4.1.0'

Read and display image by coverting it to a gray scale.

Read the input image as float data type as Tensorflow accepts images in float format.

First download a samole image:


In [2]:
!wget --output-document ../../data/lena.png https://ibm.box.com/shared/static/o4x2cvlqfre9lax05ihbn47cxpnzwlvb.png


--2017-05-23 10:39:30--  https://ibm.box.com/shared/static/o4x2cvlqfre9lax05ihbn47cxpnzwlvb.png
Resolving ibm.box.com (ibm.box.com)... 107.152.27.197, 107.152.26.197
Connecting to ibm.box.com (ibm.box.com)|107.152.27.197|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://ibm.ent.box.com/shared/static/o4x2cvlqfre9lax05ihbn47cxpnzwlvb.png [following]
--2017-05-23 10:39:30--  https://ibm.ent.box.com/shared/static/o4x2cvlqfre9lax05ihbn47cxpnzwlvb.png
Resolving ibm.ent.box.com (ibm.ent.box.com)... 107.152.26.211
Connecting to ibm.ent.box.com (ibm.ent.box.com)|107.152.26.211|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://public.boxcloud.com/d/1/LvG3HePSaPNDMo7a3c6TSlUtbbsfHbzV-8SeIP31OxJoNYHeXWioZfzepd_pgCcBoj_Xyqn3bhCU_OEgWmGf81VzN9fbVpiu3iSOQXfS448yYV6lEAOOarTroHzUc6GNyuX7B7okf9_azm5gQ4qluoNSprqJGye6CYdpXayfsf7JF5tLyV-W8ZssH0Xhkza8EwhEH1Uj3PruAdU_q1eqzc-to4xjHX9MR7vdneHPBgwk2DKnvjpT3ejYIYd9ywSAOh7zLEXq-HI4MrJ82sXJc02prNOgu5qIZyR1Dt7DJYXGu32ytifnfV_U_hvH7gaCzRZJzD880dI-_Dcx1MN3pb2i8q_jU8Jfa3MCiwKxWNIwbg-maCj4iqH1hV5Z4ySOCnSLj1errreeG0Eo3LiquST3O3qHxJ7IWUFr9zq-AVPzeBYrw2owN3HuWsEkqmgQmlHu6qxvMi5ryHg5i5WHVyTBgAGMDURBhoUxtk5coAUyO_DNRu9Ys38eYibu179bRknLhVF1vju70B4cEuLFvSpUDNZ0QJQwwlalrt-qxGMYwzPqlu5hMLVVUFBjCu7FoAG-lH7e6Mni3OaY1W2gLENl78_KoMeSkxoMzuzH3fsGNL2NhvcZG4UhITwmS3jC8pLPpePrQLM2sYvGLPZjSf--i_h0V_gVAZTqHmV_crFCv7RDSgPsDCR1UcECYQYMRyHq6Dq6wkVoWn3XJJ2tI-uHGJD1AecohUvRhaEeeB5k9dy60Dso7b7AEM-yT6CCcwcDwdBmaV7d4c2MM9RkBJYhYnDSLVU5MEgKkVNFDnLJOqg3I2xBxfV0J_v-Akm45ajnK7ArQAYi62rJN-4ZJuAFBHe8fP9s4g2MK8lr2KIPxRuz4800EIxYYEB3UsBM7JpLg-kWEUmVTruAH8oWgzXBvfLQC17hn65l-jCfDF_YoS6n_NmVHtiP9meuU-YWg25ymJHXVLH512NqAHnpUPxwnpvQlEBpfT_r3HNjVHrBuixYrz78r26R4jfKCcZvwHml4D3iwlc3cIl8q5AGkMkbqldPdtGqTn03t0xyWcLks_i1hsoRLfL08djBGP0XwsA19hNMr03odgEo0-lHQ6LXrcdBimKcX04wcD951b4riKQSziq_cfypQw3AvGj9W7Rp6ipFJyuW9UZht_44vNFQipZ7CYChUhNIi3Tkqc34ue96-yfw/download [following]
--2017-05-23 10:39:31--  https://public.boxcloud.com/d/1/LvG3HePSaPNDMo7a3c6TSlUtbbsfHbzV-8SeIP31OxJoNYHeXWioZfzepd_pgCcBoj_Xyqn3bhCU_OEgWmGf81VzN9fbVpiu3iSOQXfS448yYV6lEAOOarTroHzUc6GNyuX7B7okf9_azm5gQ4qluoNSprqJGye6CYdpXayfsf7JF5tLyV-W8ZssH0Xhkza8EwhEH1Uj3PruAdU_q1eqzc-to4xjHX9MR7vdneHPBgwk2DKnvjpT3ejYIYd9ywSAOh7zLEXq-HI4MrJ82sXJc02prNOgu5qIZyR1Dt7DJYXGu32ytifnfV_U_hvH7gaCzRZJzD880dI-_Dcx1MN3pb2i8q_jU8Jfa3MCiwKxWNIwbg-maCj4iqH1hV5Z4ySOCnSLj1errreeG0Eo3LiquST3O3qHxJ7IWUFr9zq-AVPzeBYrw2owN3HuWsEkqmgQmlHu6qxvMi5ryHg5i5WHVyTBgAGMDURBhoUxtk5coAUyO_DNRu9Ys38eYibu179bRknLhVF1vju70B4cEuLFvSpUDNZ0QJQwwlalrt-qxGMYwzPqlu5hMLVVUFBjCu7FoAG-lH7e6Mni3OaY1W2gLENl78_KoMeSkxoMzuzH3fsGNL2NhvcZG4UhITwmS3jC8pLPpePrQLM2sYvGLPZjSf--i_h0V_gVAZTqHmV_crFCv7RDSgPsDCR1UcECYQYMRyHq6Dq6wkVoWn3XJJ2tI-uHGJD1AecohUvRhaEeeB5k9dy60Dso7b7AEM-yT6CCcwcDwdBmaV7d4c2MM9RkBJYhYnDSLVU5MEgKkVNFDnLJOqg3I2xBxfV0J_v-Akm45ajnK7ArQAYi62rJN-4ZJuAFBHe8fP9s4g2MK8lr2KIPxRuz4800EIxYYEB3UsBM7JpLg-kWEUmVTruAH8oWgzXBvfLQC17hn65l-jCfDF_YoS6n_NmVHtiP9meuU-YWg25ymJHXVLH512NqAHnpUPxwnpvQlEBpfT_r3HNjVHrBuixYrz78r26R4jfKCcZvwHml4D3iwlc3cIl8q5AGkMkbqldPdtGqTn03t0xyWcLks_i1hsoRLfL08djBGP0XwsA19hNMr03odgEo0-lHQ6LXrcdBimKcX04wcD951b4riKQSziq_cfypQw3AvGj9W7Rp6ipFJyuW9UZht_44vNFQipZ7CYChUhNIi3Tkqc34ue96-yfw/download
Resolving public.boxcloud.com (public.boxcloud.com)... 107.152.26.200, 107.152.27.200
Connecting to public.boxcloud.com (public.boxcloud.com)|107.152.26.200|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 512673 (501K) [image/png]
Saving to: ‘../../data/lena.png’

../../data/lena.png 100%[===================>] 500,66K   619KB/s    in 0,8s    

2017-05-23 10:39:33 (619 KB/s) - ‘../../data/lena.png’ saved [512673/512673]

Run this Cell to experiment with Lena Image. RGB with Size (512, 512)


In [4]:
#read the image as Float data type
im=misc.imread("../../data/lena.png").astype(np.float)

#im=misc.imread("one.png").astype(np.float)

#Convert image to gray scale
grayim=np.dot(im[...,:3], [0.299, 0.587, 0.114])


#Plot the images
%matplotlib inline

plt.subplot(1, 2, 1)
plt.imshow(im)
plt.xlabel(" Float Image ")

plt.subplot(1, 2, 2)
plt.imshow(grayim, cmap=plt.get_cmap("gray"))
plt.xlabel(" Gray Scale Image ")


Out[4]:
<matplotlib.text.Text at 0x7f5584df9ef0>

Run this cell to experiemnt with MNIST image of Number 1. Gray Scale SIze (28, 28)

Print the shape of Gray Scale Image


In [5]:
# Your Code Goes Here
grayim.shape


Out[5]:
(512, 512)
``` print grayim.shape ```

Extend the Dimensions of the Gray Scale Image

For convolution, TensorFlow accepts Images of dimensions:

[num of images, width, height, channels].

In this case we are looking for dimensions of [1,512,512,1] from (512,512).


In [6]:
Image = np.expand_dims(np.expand_dims(grayim, 0), -1)
Image.shape


Out[6]:
(1, 512, 512, 1)

Create Place holder for an input image and Print the Shape

The placeholder takes input in float format and same size of input image


In [7]:
# Your Code Goes Here
in_img = tf.placeholder(dtype=tf.float32, shape=[None, 512, 512, 1])
in_img.get_shape().as_list()


Out[7]:
[None, 512, 512, 1]
``` img= tf.placeholder(tf.float32, [None,512,512,1]) print img.get_shape().as_list() ```

Create a Variable for Weight Matrix and Print the Shape

The shape of weight matrix is of the form: [ Height, widht, Input , Output].

In this case lets create weight matrix of size 5 X 5 and keeping number of inputs and output to just 1. So, the shape is of form [5,5,1,1].


In [8]:
# Your Code Goes Here
weights = tf.Variable(tf.constant(1., shape=[5, 5, 1, 1], dtype=tf.float32))
weights.get_shape().as_list()


Out[8]:
[5, 5, 1, 1]
``` shape=[5,5,1,1] weights =tf.Variable(tf.truncated_normal(shape, stddev=0.05)) print weights.get_shape().as_list() ```

Create Two Convolution Graphs in Tensorflow

lets use the functon tf.nn.conv2d to create a graph for convolution operation with padding 'same' and Padding 'Valid'.


In [9]:
# Your Code Goes Here

# For convolution output 1
ConOut = tf.nn.conv2d(input=in_img, filter=weights, strides=[1, 1, 1, 1], padding='SAME')

# For convolution output 2
ConOut2 = tf.nn.conv2d(input=in_img, filter=weights, strides=[1, 1, 1, 1], padding='VALID')
``` ConOut = tf.nn.conv2d(input=img, filter=weights, strides=[1, 1, 1, 1], padding='SAME') ConOut2 = tf.nn.conv2d(input=img, filter=weights, strides=[1, 1, 1, 1], padding='VALID') ```

Initialize all variables and Run the Sessions


In [10]:
init = tf.global_variables_initializer()
sess= tf.Session()
sess.run(init)

Run the sesions to get the results for two convolution operations


In [11]:
# Your Code Goes Here

# Session for Result 1
result = sess.run(ConOut, feed_dict={in_img: Image})


# Session for Result 2
result2 = sess.run(ConOut2, feed_dict={in_img: Image})
``` result = sess.run(ConOut,feed_dict={img:Image}) result2 = sess.run(ConOut2,feed_dict={img:Image}) ```

Display the output images

The result of convolution with 'same' padding is of the form [1,512,512,1] and for 'valid' padding image is of the shape [1,508,508,1]. To display the images, our job is to reshape the dimensions in the form (512,512) and (508,508) respectively.


In [12]:
# for the result with 'SAME' Padding 

#reduce the dimension
vec = np.reshape(result, (1, -1));
# Reshape the image
image= np.reshape(vec,(512,512))

print(image.shape)


# for the result with 'VALID' Padding 

#reduce the dimension
vec2 = np.reshape(result2, (1, -1));
# Reshape the image
image2= np.reshape(vec2,(508,508))

print(image2.shape)


(512, 512)
(508, 508)

Display the images


In [13]:
#Plot the images
%matplotlib inline

plt.subplot(1, 2, 1)
plt.imshow(image,cmap=plt.get_cmap("gray"))
plt.xlabel(" SAME Padding ")

plt.subplot(1, 2, 2)
plt.imshow(image2, cmap=plt.get_cmap("gray"))
plt.xlabel(" VALID Padding ")


Out[13]:
<matplotlib.text.Text at 0x7f5582ca88d0>

Feel free to change the weight matrix and experiment with different Paddings to see the changes in output images.

Create First Convolution Neural Network Layer

using above conv2d function lets build our first conv Layer. Usually most general CNN architecture Layer 1 comprises of Convolution, Relu and MaxPooling. Lets create these functions to check the effects on "Lena" Image. Depending on the architecture these functions may change. For this exercise lets assume our Layer 1 has just three functions Convolution, Relu and Maxpooling.

It is most often repetation of these layers stacked on top of each other to create Deep CNN


In [14]:
#lets create functions for convolution and Maxpooling

def conv2d(X,W):
    return tf.nn.conv2d(X, W, strides=[1, 1, 1, 1], padding='SAME')

def MaxPool(X):
    return tf.nn.max_pool(X, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
``` return tf.nn.conv2d(input=X,filter=W,strides=[1, 1, 1, 1],padding='SAME') return tf.nn.max_pool(X, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') ```

Create Weights and Biases for Convolution

The weights are of the shape [Height, Width , Input , Output]. Here lets create the weights of size 5X5 which has 1 input and 32 Outputs.


In [15]:
weights = {
        # 5 x 5 convolution, 1 input image, 32 outputs
        'W_conv1': tf.Variable(tf.random_normal([5, 5, 1, 32]))
        
       
    }

biases = {
        #bias should be of the size of number of Outputs
       'b_conv1': tf.Variable(tf.random_normal([32]))
        
    }
``` 'W_conv1': tf.Variable(tf.random_normal([5, 5, 1, 32])) 'b_conv1': tf.Variable(tf.random_normal([32])) ```

Define a TensorFlow Graph for Relu, Convolution and Maxpooling

The output of Conv2d is passed through Relu Layer and finally, the output of Relu is given as input for Maxpooling layer. Let's define the graph and print the shapes.The size of Image is reduced after passing through Maxpool Layer. You can change the size and strides in Maxpool layer to check how the image size varies


In [16]:
conv1 = tf.nn.relu(conv2d(in_img, weights['W_conv1']) + biases['b_conv1'])

Mxpool = MaxPool(conv1)

    
print(conv1.get_shape().as_list())

print(Mxpool.get_shape().as_list())


[None, 512, 512, 32]
[None, 256, 256, 32]
``` tf.nn.relu(conv2d(img, weights['W_conv1']) + biases['b_conv1']) MaxPool (conv1) ```

Initialize all TensorFlow Variables and Run the Session


In [17]:
init = tf.global_variables_initializer()
sess= tf.Session()
sess.run(init)

Run session to get the output of Layer 1

The session is run on MxPool which will be the final output


In [18]:
Layer1 = sess.run(Mxpool, feed_dict={in_img: Image})
``` sess.run(Mxpool,feed_dict={img:Image}) ```

Visualize the Output of Convolution layer1


In [19]:
print(Layer1.shape)

vec = np.reshape(Layer1, (256,256,32));
print(vec.shape)

for i in range (32):
    
    image=vec[:,:,i]
    #print image
    #image *= 255.0/image.max() 
    #print image
    plt.imshow(image,cmap=plt.get_cmap("gray"))
    plt.xlabel( i , fontsize=20, color='red')
    plt.show()
    plt.close()


(1, 256, 256, 32)
(256, 256, 32)

Please feel free to experiemnt with different values of: Padding, Kernel Size to see how the output varies

The Idea behind this exercise is to gain understanding on how to apply convolutions and other functions on Images. We are NOT training Neural Network here, however just checking the effects of changing parameters of the above functions which are basic building blocks of any Deep Convolution neural Networks.

Thank You Completing This Exercise

Created by Shashibushan Yenkanchi </h4>


Copyright © 2016 Big Data University. This notebook and its source code are released under the terms of the MIT License.