Loss & Modules


In [32]:
from NNPy import NetworkModule


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-32-78223ea258d5> in <module>()
----> 1 from NNPy import NetworkModule

ImportError: cannot import name NetworkModule

Nous allons nous intéresser au développement d'un classifieur sur les données MNSIT (nous ferons des visages par la suite)

Le chargement des données MNIST s'effectue facilement grâce à la commande suivante:


In [25]:
from sklearn.datasets import fetch_mldata
mnist=fetch_mldata('MNIST original')
mnist


Out[25]:
{'COL_NAMES': ['label', 'data'],
 'DESCR': 'mldata.org dataset: mnist-original',
 'data': array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ..., 
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
 'target': array([ 0.,  0.,  0., ...,  9.,  9.,  9.])}

Visualisation des données


In [26]:
import matplotlib.pyplot as plt
import numpy as np

plt.rc("image",cmap="binary")
plt.subplot(10,10,1)

for i in range(100):
    plt.subplot(10,10,i)
    idx=np.random.randint(len(mnist.data))
    plt.imshow(mnist.data[idx].reshape(28,28))


Afin de réduire la taille des expérimentations, nous allons nous restreindre aux chiffres dont les étiquettes sont 6 et 8


In [27]:
import numpy as np

#Creation des vecteurs d'entrée
mnist_6=mnist.data[mnist.target==6]
nb_6=len(mnist_6)
mnist_8=mnist.data[mnist.target==8]
nb_8=len(mnist_8)
mnist_6_8=np.vstack((mnist_6,mnist_8))
print "%d 6s and %d 8s" % (nb_6,nb_8)

#Creation des vecteurs de sortie
target_6_8=np.array([[1]])
for i in range(nb_6-1):
    target_6_8=np.vstack((target_6_8,[1]))
for i in range(nb_8):
    target_6_8=np.vstack((target_6_8,[-1]))
    
print "%d/%d vecteurs d'apprentissage" % (len(target_6_8),len(mnist_6_8))


6876 6s and 6825 8s
13701/13701 vecteurs d'apprentissage

Sous-échantillonage des données


In [28]:
# Sous-Echantillonage des deux matrices (pour test)
randomvec=np.random.rand(len(target_6_8))
randomvec=randomvec>0.8
target_6_8=target_6_8[randomvec]
mnist_6_8=mnist_6_8[randomvec]
print "%d/%d vecteurs d'apprentissage apres echantillonage" % (len(target_6_8),len(mnist_6_8))


2704/2704 vecteurs d'apprentissage apres echantillonage

Création d'un ensemble de Train et de Test


In [29]:
#Creatin de train et test
randomvec=np.random.rand(len(target_6_8))
randomvec=randomvec>0.5

train_data=mnist_6_8[randomvec]
train_label=target_6_8[randomvec]
test_data=mnist_6_8[np.logical_not(randomvec)]
test_label=target_6_8[np.logical_not(randomvec)]
print "%d training examples and %d testing examples " % (len(train_data),len(test_data))
plt.imshow(train_data[np.random.randint(len(train_data))].reshape(28,28))


1341 training examples and 1363 testing examples 
Out[29]:
<matplotlib.image.AxesImage at 0x7f37f6d87b50>

Implémentation de classifieurs

Commencez par tester plusieurs classifieurs (architecture) sur ce problème.

Perceptron



In [30]:
perceptron = NetworkModule([LinearModule(28*28,1)],HingeLoss())

NBITER = 10
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
    perceptron.stochasticIter(train_data, train_label, verbose=False)
    
    predicted = perceptron.forwardAll(train_data)
    ok=0
    ko=0
    for pred,exp in zip(predicted,train_label):
        if pred[0]*exp[0] > 0:
            ok+=1
        else:
            ko+=1
    print "Iteration n°%d: %d correct (%f%%), %d incorrect (%f%%) " % (i,ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)


    
print "=======TEST ERROR======="  
predicted = perceptron.forwardAll(test_data)

ok=0
ko=0
for pred,exp in zip(predicted,test_label):
    if pred[0]*exp[0] > 0:
        ok+=1
    else:
        ko+=1
        
print "%d correct (%f%%), %d incorrect (%f%%) " % (ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-30-54a14c72a290> in <module>()
----> 1 perceptron = NetworkModule([LinearModule(28*28,1)],HingeLoss())
      2 
      3 NBITER = 10
      4 print "=======TRAIN ERROR======="
      5 for i in xrange(0,NBITER):

NameError: name 'NetworkModule' is not defined

Multi-Layer Perceptron



In [22]:
mod2 = LinearModule(50,1)
mod1 = LinearModule(28*28,50)

MLP = NetworkModule([mod1,TanhModule(20,20),mod2],SquareLoss())

#print mod1.parameters
NBITER = 50
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
    MLP.stochasticIter(train_data, train_label,gradient_step=0.0001, verbose=False)
    
    predicted = MLP.forwardAll(train_data)
    ok=0
    ko=0
    for pred,exp in zip(predicted,train_label):
        if pred[0]*exp[0] > 0:
            ok+=1
        else:
            ko+=1
    print "Iteration n°%d: %d correct (%f%%), %d incorrect (%f%%) " % (i,ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)

#print mod1.parameters
    
print "=======TEST ERROR======="  
predicted = MLP.forwardAll(test_data)

ok=0
ko=0
for pred,exp in zip(predicted,test_label):
    if pred[0]*exp[0] > 0:
        ok+=1
    else:
        ko+=1
        
print "%d correct (%f%%), %d incorrect (%f%%) " % (ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)


=======TRAIN ERROR=======
Iteration n°0: 778 correct (55.930985%), 613 incorrect (44.069015%) 
Iteration n°1: 826 correct (59.381740%), 565 incorrect (40.618260%) 
Iteration n°2: 805 correct (57.872035%), 586 incorrect (42.127965%) 
Iteration n°3: 800 correct (57.512581%), 591 incorrect (42.487419%) 
Iteration n°4: 821 correct (59.022286%), 570 incorrect (40.977714%) 
Iteration n°5: 846 correct (60.819554%), 545 incorrect (39.180446%) 
Iteration n°6: 869 correct (62.473041%), 522 incorrect (37.526959%) 
Iteration n°7: 892 correct (64.126528%), 499 incorrect (35.873472%) 
Iteration n°8: 924 correct (66.427031%), 467 incorrect (33.572969%) 
Iteration n°9: 948 correct (68.152408%), 443 incorrect (31.847592%) 
Iteration n°10: 968 correct (69.590223%), 423 incorrect (30.409777%) 
Iteration n°11: 985 correct (70.812365%), 406 incorrect (29.187635%) 
Iteration n°12: 1004 correct (72.178289%), 387 incorrect (27.821711%) 
Iteration n°13: 1014 correct (72.897196%), 377 incorrect (27.102804%) 
Iteration n°14: 1031 correct (74.119339%), 360 incorrect (25.880661%) 
Iteration n°15: 1055 correct (75.844716%), 336 incorrect (24.155284%) 
Iteration n°16: 1063 correct (76.419842%), 328 incorrect (23.580158%) 
Iteration n°17: 1072 correct (77.066858%), 319 incorrect (22.933142%) 
Iteration n°18: 1084 correct (77.929547%), 307 incorrect (22.070453%) 
Iteration n°19: 1097 correct (78.864127%), 294 incorrect (21.135873%) 
Iteration n°20: 1106 correct (79.511143%), 285 incorrect (20.488857%) 
Iteration n°21: 1114 correct (80.086269%), 277 incorrect (19.913731%) 
Iteration n°22: 1120 correct (80.517613%), 271 incorrect (19.482387%) 
Iteration n°23: 1128 correct (81.092739%), 263 incorrect (18.907261%) 
Iteration n°24: 1135 correct (81.595974%), 256 incorrect (18.404026%) 
Iteration n°25: 1144 correct (82.242991%), 247 incorrect (17.757009%) 
Iteration n°26: 1156 correct (83.105679%), 235 incorrect (16.894321%) 
Iteration n°27: 1162 correct (83.537024%), 229 incorrect (16.462976%) 
Iteration n°28: 1162 correct (83.537024%), 229 incorrect (16.462976%) 
Iteration n°29: 1162 correct (83.537024%), 229 incorrect (16.462976%) 
Iteration n°30: 1164 correct (83.680805%), 227 incorrect (16.319195%) 
Iteration n°31: 1166 correct (83.824587%), 225 incorrect (16.175413%) 
Iteration n°32: 1176 correct (84.543494%), 215 incorrect (15.456506%) 
Iteration n°33: 1178 correct (84.687275%), 213 incorrect (15.312725%) 
Iteration n°34: 1186 correct (85.262401%), 205 incorrect (14.737599%) 
Iteration n°35: 1186 correct (85.262401%), 205 incorrect (14.737599%) 
Iteration n°36: 1190 correct (85.549964%), 201 incorrect (14.450036%) 
Iteration n°37: 1191 correct (85.621855%), 200 incorrect (14.378145%) 
Iteration n°38: 1195 correct (85.909418%), 196 incorrect (14.090582%) 
Iteration n°39: 1200 correct (86.268871%), 191 incorrect (13.731129%) 
Iteration n°40: 1200 correct (86.268871%), 191 incorrect (13.731129%) 
Iteration n°41: 1203 correct (86.484543%), 188 incorrect (13.515457%) 
Iteration n°42: 1205 correct (86.628325%), 186 incorrect (13.371675%) 
Iteration n°43: 1205 correct (86.628325%), 186 incorrect (13.371675%) 
Iteration n°44: 1205 correct (86.628325%), 186 incorrect (13.371675%) 
Iteration n°45: 1211 correct (87.059669%), 180 incorrect (12.940331%) 
Iteration n°46: 1211 correct (87.059669%), 180 incorrect (12.940331%) 
Iteration n°47: 1212 correct (87.131560%), 179 incorrect (12.868440%) 
Iteration n°48: 1213 correct (87.203451%), 178 incorrect (12.796549%) 
Iteration n°49: 1213 correct (87.203451%), 178 incorrect (12.796549%) 
=======TEST ERROR=======
1203 correct (85.077793%), 211 incorrect (14.922207%) 

In [10]:
mod2 = LinearModule(50,1)
mod1 = LinearModule(28*28,50)

MLP = NetworkModule([mod1,TanhModule(20,20),mod2],SquareLoss())

#print mod1.parameters
NBITER = 50
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
    MLP.batchIter(train_data, train_label,gradient_step=0.000001, verbose=False)
    predicted = MLP.forwardAll(train_data)
    ok=0
    ko=0
    for pred,exp in zip(predicted,train_label):
        if pred[0]*exp[0] > 0:
            ok+=1
        else:
            ko+=1
    print "Iteration n°%d: %d correct (%f%%), %d incorrect (%f%%) " % (i,ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)

#print mod1.parameters
    
print "=======TEST ERROR======="  
predicted = MLP.forwardAll(test_data)

ok=0
ko=0
for pred,exp in zip(predicted,test_label):
    if pred[0]*exp[0] > 0:
        ok+=1
    else:
        ko+=1
        
print "%d correct (%f%%), %d incorrect (%f%%) " % (ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)


=======TRAIN ERROR=======
Iteration n°0: 807 correct (59.381898%), 552 incorrect (40.618102%) 
Iteration n°1: 812 correct (59.749816%), 547 incorrect (40.250184%) 
Iteration n°2: 805 correct (59.234731%), 554 incorrect (40.765269%) 
Iteration n°3: 807 correct (59.381898%), 552 incorrect (40.618102%) 
Iteration n°4: 802 correct (59.013981%), 557 incorrect (40.986019%) 
Iteration n°5: 800 correct (58.866814%), 559 incorrect (41.133186%) 
Iteration n°6: 797 correct (58.646063%), 562 incorrect (41.353937%) 
Iteration n°7: 801 correct (58.940397%), 558 incorrect (41.059603%) 
Iteration n°8: 799 correct (58.793230%), 560 incorrect (41.206770%) 
Iteration n°9: 798 correct (58.719647%), 561 incorrect (41.280353%) 
Iteration n°10: 793 correct (58.351729%), 566 incorrect (41.648271%) 
Iteration n°11: 799 correct (58.793230%), 560 incorrect (41.206770%) 
Iteration n°12: 796 correct (58.572480%), 563 incorrect (41.427520%) 
Iteration n°13: 794 correct (58.425313%), 565 incorrect (41.574687%) 
Iteration n°14: 796 correct (58.572480%), 563 incorrect (41.427520%) 
Iteration n°15: 796 correct (58.572480%), 563 incorrect (41.427520%) 
Iteration n°16: 794 correct (58.425313%), 565 incorrect (41.574687%) 
Iteration n°17: 796 correct (58.572480%), 563 incorrect (41.427520%) 
Iteration n°18: 796 correct (58.572480%), 563 incorrect (41.427520%) 
Iteration n°19: 792 correct (58.278146%), 567 incorrect (41.721854%) 
Iteration n°20: 790 correct (58.130979%), 569 incorrect (41.869021%) 
Iteration n°21: 789 correct (58.057395%), 570 incorrect (41.942605%) 
Iteration n°22: 786 correct (57.836645%), 573 incorrect (42.163355%) 
Iteration n°23: 778 correct (57.247976%), 581 incorrect (42.752024%) 
Iteration n°24: 778 correct (57.247976%), 581 incorrect (42.752024%) 
Iteration n°25: 777 correct (57.174393%), 582 incorrect (42.825607%) 
Iteration n°26: 777 correct (57.174393%), 582 incorrect (42.825607%) 
Iteration n°27: 779 correct (57.321560%), 580 incorrect (42.678440%) 
Iteration n°28: 780 correct (57.395143%), 579 incorrect (42.604857%) 
Iteration n°29: 780 correct (57.395143%), 579 incorrect (42.604857%) 
Iteration n°30: 779 correct (57.321560%), 580 incorrect (42.678440%) 
Iteration n°31: 774 correct (56.953642%), 585 incorrect (43.046358%) 
Iteration n°32: 774 correct (56.953642%), 585 incorrect (43.046358%) 
Iteration n°33: 774 correct (56.953642%), 585 incorrect (43.046358%) 
Iteration n°34: 779 correct (57.321560%), 580 incorrect (42.678440%) 
Iteration n°35: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°36: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°37: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°38: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°39: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°40: 774 correct (56.953642%), 585 incorrect (43.046358%) 
Iteration n°41: 775 correct (57.027226%), 584 incorrect (42.972774%) 
Iteration n°42: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°43: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°44: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°45: 776 correct (57.100809%), 583 incorrect (42.899191%) 
Iteration n°46: 775 correct (57.027226%), 584 incorrect (42.972774%) 
Iteration n°47: 774 correct (56.953642%), 585 incorrect (43.046358%) 
Iteration n°48: 773 correct (56.880059%), 586 incorrect (43.119941%) 
Iteration n°49: 772 correct (56.806475%), 587 incorrect (43.193525%) 
=======TEST ERROR=======
765 correct (56.582840%), 587 incorrect (43.417160%) 

In [21]:
mod2 = LinearModule(50,1)
mod1 = LinearModule(28*28,50)

MLP = NetworkModule([mod1,TanhModule(20,20),mod2],SquareLoss())

#print mod1.parameters
NBITER = 50
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
    MLP.miniBatchIter(train_data, train_label,batch_size=20,gradient_step=0.000001, verbose=False)
    predicted = MLP.forwardAll(train_data)
    ok=0
    ko=0
    for pred,exp in zip(predicted,train_label):
        if pred[0]*exp[0] > 0:
            ok+=1
        else:
            ko+=1
    print "Iteration n°%d: %d correct (%f%%), %d incorrect (%f%%) " % (i,ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)

#print mod1.parameters
    
print "=======TEST ERROR======="  
predicted = MLP.forwardAll(test_data)

ok=0
ko=0
for pred,exp in zip(predicted,test_label):
    if pred[0]*exp[0] > 0:
        ok+=1
    else:
        ko+=1
        
print "%d correct (%f%%), %d incorrect (%f%%) " % (ok,ok/(ok+ko+0.0)*100,ko,ko/(ok+ko+0.0)*100)


=======TRAIN ERROR=======
Iteration n°0: 464 correct (33.357297%), 927 incorrect (66.642703%) 
Iteration n°1: 470 correct (33.788641%), 921 incorrect (66.211359%) 
Iteration n°2: 473 correct (34.004313%), 918 incorrect (65.995687%) 
Iteration n°3: 474 correct (34.076204%), 917 incorrect (65.923796%) 
Iteration n°4: 477 correct (34.291876%), 914 incorrect (65.708124%) 
Iteration n°5: 480 correct (34.507549%), 911 incorrect (65.492451%) 
Iteration n°6: 480 correct (34.507549%), 911 incorrect (65.492451%) 
Iteration n°7: 482 correct (34.651330%), 909 incorrect (65.348670%) 
Iteration n°8: 484 correct (34.795111%), 907 incorrect (65.204889%) 
Iteration n°9: 488 correct (35.082674%), 903 incorrect (64.917326%) 
Iteration n°10: 492 correct (35.370237%), 899 incorrect (64.629763%) 
Iteration n°11: 495 correct (35.585909%), 896 incorrect (64.414091%) 
Iteration n°12: 496 correct (35.657800%), 895 incorrect (64.342200%) 
Iteration n°13: 496 correct (35.657800%), 895 incorrect (64.342200%) 
Iteration n°14: 498 correct (35.801582%), 893 incorrect (64.198418%) 
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-21-994b808df487> in <module>()
      8 print "=======TRAIN ERROR======="
      9 for i in xrange(0,NBITER):
---> 10     MLP.miniBatchIter(train_data, train_label,batch_size=20,gradient_step=0.000001, verbose=False)
     11     predicted = MLP.forwardAll(train_data)
     12     ok=0

/home/charles/AS/NNModule/NNPy.py in miniBatchIter(self, examples, labels, batch_size, gradient_step, verbose)
    282         for i, (example, label) in enumerate(zip(examples,labels)):
    283             pred = self.forward(example)
--> 284             loss = self.backward(pred,label,batch=True,gradient_step=gradient_step)
    285 
    286             if verbose:

/home/charles/AS/NNModule/NNPy.py in backward(self, predicted, wanted, batch, gradient_step)
    245     #Permet le calcul du gradient des cellules d'entrée
    246     def backward(self,predicted,wanted,batch=False,gradient_step=0.001):
--> 247         loss_delta = self.loss.backward(predicted,wanted)
    248         for module,input in zip(reversed(self.modules),reversed(self.inputs)):
    249             loss_delta = module.backward(input,loss_delta)

/home/charles/AS/NNModule/NNPy.py in backward(self, predicted_output, desired_output)
     68 
     69     def backward(self, predicted_output,desired_output):
---> 70         return 2*(predicted_output-desired_output)
     71 
     72 #HingeLoss

KeyboardInterrupt: 

Implémentation des Auto-encodeurs

Implémentez des auto-encodeurs et visualisez les patches appris


In [25]:
mod2 = LinearModule(100,28*28)
mod1 = LinearModule(28*28,100)

AE = NetworkModule([mod1,mod2],SquareLoss())

sq = SquareLoss()

#print sq.getLossValue(train_data[0],train_data[0])


#print mod1.parameters
NBITER = 200
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
   loss = AE.miniBatchIter(train_data, train_data,batch_size=150,gradient_step=0.000000000001, verbose=False)
   predicted = AE.forwardAll(train_data)
   err=0
    
   for pred,exp in zip(predicted,train_data):
       err += np.sum(np.power(pred-exp,2))
    
        
   #print loss
   print "Iteration n°%d - MSE: %f" % (i,err/len(predicted))


=======TRAIN ERROR=======
Iteration n°0 - MSE: 166021727987.599457
Iteration n°1 - MSE: 122143492625.978531
Iteration n°2 - MSE: 100487735920.768707
Iteration n°3 - MSE: 87037646272.574783
Iteration n°4 - MSE: 77558704349.099045
Iteration n°5 - MSE: 70363386247.994370
Iteration n°6 - MSE: 64639728215.449783
Iteration n°7 - MSE: 59940079989.730042
Iteration n°8 - MSE: 55991747489.922989
Iteration n°9 - MSE: 52616015250.980911
Iteration n°10 - MSE: 49689270742.220665
Iteration n°11 - MSE: 47122481207.087051
Iteration n°12 - MSE: 44849485078.385956
Iteration n°13 - MSE: 42819880338.848785
Iteration n°14 - MSE: 40994482457.468361
Iteration n°15 - MSE: 39342303999.184975
Iteration n°16 - MSE: 37838479298.894402
Iteration n°17 - MSE: 36462799551.772491
Iteration n°18 - MSE: 35198655165.821060
Iteration n°19 - MSE: 34032257299.433807
Iteration n°20 - MSE: 32952055230.172421
Iteration n°21 - MSE: 31948293826.524292
Iteration n°22 - MSE: 31012672995.422871
Iteration n°23 - MSE: 30138082494.111435
Iteration n°24 - MSE: 29318393204.185619
Iteration n°25 - MSE: 28548291230.939281
Iteration n°26 - MSE: 27823144851.277409
Iteration n°27 - MSE: 27138896918.163563
Iteration n°28 - MSE: 26491977180.888428
Iteration n°29 - MSE: 25879230323.672646
Iteration n°30 - MSE: 25297856511.272076
Iteration n°31 - MSE: 24745361962.183674
Iteration n°32 - MSE: 24219517618.795673
Iteration n°33 - MSE: 23718324399.124313
Iteration n°34 - MSE: 23239983831.844009
Iteration n°35 - MSE: 22782873120.395584
Iteration n°36 - MSE: 22345523871.294090
Iteration n°37 - MSE: 21926603869.713676
Iteration n°38 - MSE: 21524901401.838371
Iteration n°39 - MSE: 21139311715.650909
Iteration n°40 - MSE: 20768825285.291603
Iteration n°41 - MSE: 20412517602.982338
Iteration n°42 - MSE: 20069540269.954449
Iteration n°43 - MSE: 19739113196.252003
Iteration n°44 - MSE: 19420517750.569386
Iteration n°45 - MSE: 19113090726.880451
Iteration n°46 - MSE: 18816219015.648720
Iteration n°47 - MSE: 18529334884.768856
Iteration n°48 - MSE: 18251911789.774120
Iteration n°49 - MSE: 17983460644.816200
Iteration n°50 - MSE: 17723526495.918652
Iteration n°51 - MSE: 17471685546.386417
Iteration n°52 - MSE: 17227542491.300793
Iteration n°53 - MSE: 16990728123.979799
Iteration n°54 - MSE: 16760897182.318800
Iteration n°55 - MSE: 16537726407.205822
Iteration n°56 - MSE: 16320912788.850670
Iteration n°57 - MSE: 16110171979.978199
Iteration n°58 - MSE: 15905236857.507631
Iteration n°59 - MSE: 15705856216.627752
Iteration n°60 - MSE: 15511793583.154043
Iteration n°61 - MSE: 15322826131.757193
Iteration n°62 - MSE: 15138743699.127686
Iteration n°63 - MSE: 14959347882.421242
Iteration n°64 - MSE: 14784451214.442955
Iteration n°65 - MSE: 14613876407.998301
Iteration n°66 - MSE: 14447455662.685720
Iteration n°67 - MSE: 14285030028.147526
Iteration n°68 - MSE: 14126448818.447140
Iteration n°69 - MSE: 13971569072.810106
Iteration n°70 - MSE: 13820255058.471783
Iteration n°71 - MSE: 13672377811.816746
Iteration n°72 - MSE: 13527814714.387915
Iteration n°73 - MSE: 13386449100.690958
Iteration n°74 - MSE: 13248169895.026217
Iteration n°75 - MSE: 13112871274.854998
Iteration n°76 - MSE: 12980452358.449783
Iteration n°77 - MSE: 12850816914.794363
Iteration n°78 - MSE: 12723873093.894339
Iteration n°79 - MSE: 12599533175.830059
Iteration n°80 - MSE: 12477713337.039734
Iteration n°81 - MSE: 12358333432.458597
Iteration n°82 - MSE: 12241316792.264837
Iteration n°83 - MSE: 12126590032.094595
Iteration n°84 - MSE: 12014082875.689030
Iteration n°85 - MSE: 11903727989.027454
Iteration n°86 - MSE: 11795460825.081841
Iteration n°87 - MSE: 11689219478.402750
Iteration n°88 - MSE: 11584944548.812378
Iteration n°89 - MSE: 11482579013.542507
Iteration n°90 - MSE: 11382068107.208397
Iteration n°91 - MSE: 11283359209.060743
Iteration n°92 - MSE: 11186401737.002373
Iteration n°93 - MSE: 11091147047.897726
Iteration n°94 - MSE: 10997548343.740122
Iteration n°95 - MSE: 10905560583.277399
Iteration n°96 - MSE: 10815140398.725342
Iteration n°97 - MSE: 10726246017.228867
Iteration n°98 - MSE: 10638837186.755936
Iteration n°99 - MSE: 10552875106.132612
Iteration n°100 - MSE: 10468322358.950336
Iteration n°101 - MSE: 10385142851.095686
Iteration n°102 - MSE: 10303301751.671700
Iteration n°103 - MSE: 10222765437.095903
Iteration n°104 - MSE: 10143501438.176598
Iteration n°105 - MSE: 10065478389.982399
Iteration n°106 - MSE: 9988665984.333010
Iteration n°107 - MSE: 9913034924.752279
Iteration n°108 - MSE: 9838556883.733807
Iteration n°109 - MSE: 9765204462.181814
Iteration n°110 - MSE: 9692951150.897346
Iteration n°111 - MSE: 9621771293.990242
Iteration n°112 - MSE: 9551640054.104057
Iteration n°113 - MSE: 9482533379.349617
Iteration n°114 - MSE: 9414427971.849129
Iteration n°115 - MSE: 9347301257.799267
Iteration n°116 - MSE: 9281131358.967815
Iteration n°117 - MSE: 9215897065.543802
Iteration n°118 - MSE: 9151577810.266054
Iteration n°119 - MSE: 9088153643.759808
Iteration n°120 - MSE: 9025605211.015936
Iteration n°121 - MSE: 8963913728.950415
Iteration n°122 - MSE: 8903060964.986826
Iteration n°123 - MSE: 8843029216.606791
Iteration n°124 - MSE: 8783801291.817606
Iteration n°125 - MSE: 8725360490.489063
Iteration n°126 - MSE: 8667690586.514042
Iteration n°127 - MSE: 8610775810.750631
Iteration n°128 - MSE: 8554600834.705772
Iteration n°129 - MSE: 8499150754.922436
Iteration n°130 - MSE: 8444411078.035473
Iteration n°131 - MSE: 8390367706.462133
Iteration n°132 - MSE: 8337006924.695945
Iteration n°133 - MSE: 8284315386.174359
Iteration n°134 - MSE: 8232280100.691877
Iteration n°135 - MSE: 8180888422.332195
Iteration n°136 - MSE: 8130128037.894485
Iteration n°137 - MSE: 8079986955.790070
Iteration n°138 - MSE: 8030453495.387019
Iteration n°139 - MSE: 7981516276.781703
Iteration n°140 - MSE: 7933164210.977278
Iteration n°141 - MSE: 7885386490.449964
Iteration n°142 - MSE: 7838172580.085497
Iteration n°143 - MSE: 7791512208.468554
Iteration n°144 - MSE: 7745395359.509061
Iteration n°145 - MSE: 7699812264.390318
Iteration n°146 - MSE: 7654753393.824235
Iteration n°147 - MSE: 7610209450.600162
Iteration n°148 - MSE: 7566171362.414093
Iteration n°149 - MSE: 7522630274.966072
Iteration n°150 - MSE: 7479577545.314068
Iteration n°151 - MSE: 7437004735.472830
Iteration n°152 - MSE: 7394903606.247619
Iteration n°153 - MSE: 7353266111.292450
Iteration n°154 - MSE: 7312084391.383169
Iteration n°155 - MSE: 7271350768.896633
Iteration n°156 - MSE: 7231057742.486901
Iteration n°157 - MSE: 7191197981.950549
Iteration n°158 - MSE: 7151764323.272882
Iteration n°159 - MSE: 7112749763.847914
Iteration n°160 - MSE: 7074147457.864800
Iteration n°161 - MSE: 7035950711.853900
Iteration n°162 - MSE: 6998152980.386081
Iteration n°163 - MSE: 6960747861.919088
Iteration n°164 - MSE: 6923729094.784943
Iteration n°165 - MSE: 6887090553.313059
Iteration n°166 - MSE: 6850826244.083296
Iteration n°167 - MSE: 6814930302.304121
Iteration n°168 - MSE: 6779396988.311089
Iteration n°169 - MSE: 6744220684.180367
Iteration n°170 - MSE: 6709395890.453726
Iteration n°171 - MSE: 6674917222.970027
Iteration n°172 - MSE: 6640779409.799262
Iteration n°173 - MSE: 6606977288.275688
Iteration n°174 - MSE: 6573505802.125623
Iteration n°175 - MSE: 6540359998.686929
Iteration n°176 - MSE: 6507535026.216434
Iteration n°177 - MSE: 6475026131.282176
Iteration n°178 - MSE: 6442828656.237186
Iteration n°179 - MSE: 6410938036.772110
Iteration n°180 - MSE: 6379349799.543394
Iteration n°181 - MSE: 6348059559.874510
Iteration n°182 - MSE: 6317063019.527762
Iteration n°183 - MSE: 6286355964.543645
Iteration n°184 - MSE: 6255934263.145871
Iteration n°185 - MSE: 6225793863.709394
Iteration n°186 - MSE: 6195930792.789299
Iteration n°187 - MSE: 6166341153.208449
Iteration n°188 - MSE: 6137021122.201773
Iteration n°189 - MSE: 6107966949.615359
Iteration n°190 - MSE: 6079174956.158216
Iteration n°191 - MSE: 6050641531.705123
Iteration n°192 - MSE: 6022363133.648727
Iteration n°193 - MSE: 5994336285.299201
Iteration n°194 - MSE: 5966557574.329814
Iteration n°195 - MSE: 5939023651.266987
Iteration n°196 - MSE: 5911731228.023202
Iteration n°197 - MSE: 5884677076.471476
Iteration n°198 - MSE: 5857858027.059848
Iteration n°199 - MSE: 5831270967.464776

In [27]:
mod2 = LinearModule(100,28*28)
mod1 = LinearModule(28*28,100)

AE = NetworkModule([mod1,TanhModule(1,1),mod2],SquareLoss())

sq = SquareLoss()

#print sq.getLossValue(train_data[0],train_data[0])


#print mod1.parameters
NBITER = 50
print "=======TRAIN ERROR======="
for i in xrange(0,NBITER):
   loss = AE.miniBatchIter(train_data, train_data,batch_size=150,gradient_step=0.000001, verbose=False)
   predicted = AE.forwardAll(train_data)
   err=0
    
   for pred,exp in zip(predicted,train_data):
       err += np.sum(np.power(pred-exp,2))
    
        
   #print loss
   print "Iteration n°%d - MSE: %f" % (i,err/len(predicted))


=======TRAIN ERROR=======
Iteration n°0 - MSE: 5474544.805493
Iteration n°1 - MSE: 4691296.058353
Iteration n°2 - MSE: 4139160.515617
Iteration n°3 - MSE: 3779764.220062
Iteration n°4 - MSE: 3553208.713446
Iteration n°5 - MSE: 3412589.999868
Iteration n°6 - MSE: 3323332.653554
Iteration n°7 - MSE: 3267631.267727
Iteration n°8 - MSE: 3235355.539840
Iteration n°9 - MSE: 3212449.550038
Iteration n°10 - MSE: 3196961.497173
Iteration n°11 - MSE: 3185971.564809
Iteration n°12 - MSE: 3177670.031438
Iteration n°13 - MSE: 3173222.553728
Iteration n°14 - MSE: 3167941.138180
Iteration n°15 - MSE: 3163190.743178
Iteration n°16 - MSE: 3158768.362022
Iteration n°17 - MSE: 3158572.783054
Iteration n°18 - MSE: 3152991.882212
Iteration n°19 - MSE: 3147975.929005
Iteration n°20 - MSE: 3143291.917147
Iteration n°21 - MSE: 3144513.853090
Iteration n°22 - MSE: 3139318.945214
Iteration n°23 - MSE: 3134607.763933
Iteration n°24 - MSE: 3130238.137083
Iteration n°25 - MSE: 3126065.335902
Iteration n°26 - MSE: 3122018.614280
Iteration n°27 - MSE: 3118055.787697
Iteration n°28 - MSE: 3114151.787742
Iteration n°29 - MSE: 3110291.777949
Iteration n°30 - MSE: 3106467.018811
Iteration n°31 - MSE: 3102672.388471
Iteration n°32 - MSE: 3098904.899574
Iteration n°33 - MSE: 3095162.814727
Iteration n°34 - MSE: 3091445.121047
Iteration n°35 - MSE: 3087751.219676
Iteration n°36 - MSE: 3084080.743626
Iteration n°37 - MSE: 3080433.451930
Iteration n°38 - MSE: 3076809.168974
Iteration n°39 - MSE: 3073207.750371
Iteration n°40 - MSE: 3069629.064322
Iteration n°41 - MSE: 3066072.981878
Iteration n°42 - MSE: 3062539.372242
Iteration n°43 - MSE: 3059028.100855
Iteration n°44 - MSE: 3055539.028939
Iteration n°45 - MSE: 3052072.013786
Iteration n°46 - MSE: 3048626.909346
Iteration n°47 - MSE: 3045203.566924
Iteration n°48 - MSE: 3041801.835846
Iteration n°49 - MSE: 3038421.564064

In [32]:
init = mod1.forward(train_data[10])
b = TanhModule(1,1)
comp = b.forward(init)
patch = np.zeros(100)
patch[10] = 1
#plt.imshow(train_data[0].reshape(28,28))
plt.imshow(recomp.reshape(28,28))
plt.show()

In [9]:
input = np.array([[1, 2, 3],[1,2,3]])
np.sum(input,axis=0)


Out[9]:
array([2, 4, 6])