Sparse Autoencoder demo

You should start ipython notebook server with "--pylab inline" flag.
This notebook is based on the matlab exercise of section "Sparse Autoencoder" on http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial.



In [1]:

    
import sys
sys.path.append('../src')
from scipy.io import loadmat, savemat
from numpy.random import uniform, randint, seed
from matplotlib.cm import binary_r
from FeedforwardNeuNet import sigmoid, NnLayer, FeedforwardNeuNet
from CostFunc import sparse_CostFunc, sparse_CostFuncGrad

Loading, generating, and preprocessing random sample patches from data:

From "sampleIMAGES.m" in "sparseae_exercise.zip" on http://ufldl.stanford.edu/wiki/index.php/Exercise:Sparse_Autoencoder:

IMAGES.mat: A 3D array containing 10 images. For instance, IMAGES(:,:,6) is a 512 by 512 iamge containing the 6th image. Additionally, these images have been preprocessed using "whitening." Our task here is to generate 10000 8 by 8 subimages from the original data.



In [2]:

    
whitenImageData=loadmat('../demoDataSet/IMAGES.mat')['IMAGES']
imshow(whitenImageData[:,:,0], binary_r)
inputs=empty((1,64))
for patchNum in randint(0, 64**2*10, 10000):
    whichPic=patchNum//(64**2)
    patchNum-=64**2*whichPic
    imgRow=patchNum//64
    pxlRow=8*imgRow
    patchNum-=64*imgRow
    pxlCol=8*patchNum
    inputs=append(inputs, asmatrix(ravel(whitenImageData[pxlRow:pxlRow+8, pxlCol:pxlCol+8, whichPic])), 0)
inputs=inputs[1:] # discard empty((1, 64)) at the beginning
inputs-=mean(inputs) # gonna scale inputs to [0, 1] using the same procedure as in normalizeData function in sampleIMAGE.m
pstd= 3*std(inputs)
inputs=maximum(minimum(inputs, pstd), -pstd)/pstd
inputs=(inputs+1)*0.4+0.1
inputs = asarray(inputs)

Show how I split an image:



In [3]:

    
fig=zeros((8*64+65,8*64+65))
for row in xrange(0,64):
    for col in xrange(0,64):
        fig[8*row+row+1:8*(row+1)+row+1,8*col+col+1:8*(col+1)+col+1]=reshape(inputs[64*row+col], (8, 8))
imshow(fig, binary_r)









    Out[3]:





<matplotlib.image.AxesImage at 0x7ff9083bfcd0>

Create sparse neural network

Hyper Parameters of this sparse NN are the same as those defined at the beginning of "train.m" in "sparseae_exercise.zip" on
http://ufldl.stanford.edu/wiki/index.php/Exercise:Sparse_Autoencoder



In [4]:

    
nn = FeedforwardNeuNet((NnLayer(sigmoid, 64, 1, 25), NnLayer(sigmoid, 25, 1, 64)), 0.0001, 0.01, 3)
initRange=(6.0/(64+1+25+1))**0.5
nn.layersExOutputLy[0].updateForwardWeight(uniform(-initRange, initRange, (25,64+1)))
nn.layersExOutputLy[1].updateForwardWeight(uniform(-initRange, initRange, (64,25+1)))



In [5]:

    
nn.train(inputs, inputs, sparse_CostFunc, sparse_CostFuncGrad, 400)









    



Warning: Maximum number of iterations has been exceeded.
         Current function value: 0.497016
         Iterations: 400
         Function evaluations: 8236
         Gradient evaluations: 6908

Save optimized weights



In [44]:

    
save('./SparseAutoencoder_optWeight0.npy', nn.layersExOutputLy[0].forwardWeight)
save('./SparseAutoencoder_optWeight1.npy', nn.layersExOutputLy[1].forwardWeight)
savemat('./W1.mat', {'W1':nn.layersExOutputLy[0].forwardWeight[:-1]})

If you already have optimized weights saved, you can just load it



In [38]:

    
# in case you need to reload the weights
nn.layersExOutputLy[0].forwardWeight=load('./SparseAutoencoder_optWeight0.npy')
nn.layersExOutputLy[1].forwardWeight=load('./SparseAutoencoder_optWeight1.npy')

Visualization of the 1st hidden layer



In [41]:

    
# the code snippet in this cell mimic the behavior of directly passing weights into display_network function in display_network.m
fig=-ones((8*5+6,8*5+6))
weights=nn.layersExOutputLy[0].forwardWeight[:-1]
weights-=mean(weights) # to make the result identical to display_network.m
for row in xrange(0,5):
    for col in xrange(0,5):
        arr=weights[:,5*row+col].T # I use transpose here just to make the result identical to display_network.m
        arr=arr/amax(absolute(arr)) # to make the result identical to display_network.m
        fig[8*row+row+1:8*(row+1)+row+1,8*col+col+1:8*(col+1)+col+1]=reshape(arr, (8, 8)).T
print((amin(fig), amax(fig), mean(fig)))
imshow(fig, binary_r)









    



(-1.0, 1.0, -0.24394141880522874)






    Out[41]:





<matplotlib.image.AxesImage at 0x7ff909ee8f50>

If you have Matlab installed, you can use "W1.mat" saved above in "display_network.m" in "sparseae_exercise.zip" on
http://ufldl.stanford.edu/wiki/index.php/Exercise:Sparse_Autoencoder, modify the display_network function a little to save the resulting array, and load it up here to compare the 2 matrices.



In [42]:

    
pic=loadmat('./array.mat')['array']
print((amin(pic), amax(pic), mean(pic)))
imshow(pic, binary_r)









    



(-1.0, 1.0, -0.24394141880522846)






    Out[42]:





<matplotlib.image.AxesImage at 0x7ff909f10390>



In [43]:

    
# to show that they are almost identical
from numpy.testing import assert_array_almost_equal
assert_array_almost_equal(fig, pic)