In [1]:
from lasagne.layers import *
from lasagne.nonlinearities import *
from lasagne import init

In [2]:
nn = InputLayer([None,3,100,100])

nn = Conv2DLayer(nn,num_filters=512, filter_size=(3,3),
                 W = init.Constant(0))

nn = Conv2DLayer(nn,num_filters=128,filter_size=(3,3),
                 W = init.Constant(0))

nn = Conv2DLayer(nn,num_filters=32,filter_size=(3,3),
                 W = init.Constant(0))

nn = Pool2DLayer(nn,pool_size=(6,6),mode='max')

nn = Conv2DLayer(nn,num_filters=8,filter_size=(10,10),
                W = init.Normal(std=0.01))

nn = Conv2DLayer(nn,num_filters=8,filter_size=(10,10),
                W = init.Normal(std=0.01))

nn = Pool2DLayer(nn,pool_size=(3,3),mode='max')

nn = DenseLayer(nn,512,nonlinearity=softmax)

nn = DropoutLayer(nn,p=0.5)

nn = DenseLayer(nn,512,nonlinearity=softmax)

nn = DenseLayer(nn,10,nonlinearity=sigmoid)

nn = DropoutLayer(nn,p=0.5)















Book of grudges

  • zero init for weights will cause symmetry effect
  • Too many filters for first 3x3 convolution - will lead to enormous matrix while there's just not enough relevant combinations of 3x3 images (overkill).
  • Usually the further you go, the more filters you need.
  • large filters (10x10 is generally a bad pactice, and you definitely need more than 10 of them
  • the second of 10x10 convolution gets 8x6x6 image as input, so it's technically unable to perform such convolution.
  • Softmax nonlinearity effectively makes only 1 or a few neurons from the entire layer to "fire", rendering 512-neuron layer almost useless. Softmax at the output layer is okay though
  • Dropout after probability prediciton is just lame. A few random classes get probability of 0, so your probabilities no longer sum to 1 and crossentropy goes -inf.

In [ ]: