Activation Maximization on MNIST

Lets build the mnist model and train it for 5 epochs. It should get to about ~99% test accuracy.



In [20]:

    
from __future__ import print_function

import numpy as np
import keras

from keras.datasets import mnist
from keras.models import Sequential, Model
from keras.layers import Dense, Dropout, Flatten, Activation, Input
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K

batch_size = 128
num_classes = 10
epochs = 5

# input image dimensions
img_rows, img_cols = 28, 28

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax', name='preds'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adam(),
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])









    



x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/5
60000/60000 [==============================] - 6s - loss: 0.2346 - acc: 0.9279 - val_loss: 0.0552 - val_acc: 0.9817
Epoch 2/5
60000/60000 [==============================] - 5s - loss: 0.0845 - acc: 0.9756 - val_loss: 0.0363 - val_acc: 0.9881
Epoch 3/5
60000/60000 [==============================] - 5s - loss: 0.0620 - acc: 0.9819 - val_loss: 0.0336 - val_acc: 0.9897
Epoch 4/5
60000/60000 [==============================] - 5s - loss: 0.0533 - acc: 0.9839 - val_loss: 0.0317 - val_acc: 0.9895
Epoch 5/5
60000/60000 [==============================] - 5s - loss: 0.0436 - acc: 0.9865 - val_loss: 0.0295 - val_acc: 0.9910
Test loss: 0.0295144059571
Test accuracy: 0.991

Dense Layer Visualizations

To visualize activation over final dense layer outputs, we need to switch the softmax activation out for linear since gradient of output node will depend on all the other node activations. Doing this in keras is tricky, so we provide utils.apply_modifications to modify network parameters and rebuild the graph.

If this swapping is not done, the results might be suboptimal. We will start by swapping out 'softmax' for 'linear' and compare what happens if we dont do this at the end.

Lets start by visualizing input that maximizes the output of node 0. Hopefully this looks like a 0.



In [21]:

    
from vis.visualization import visualize_activation
from vis.utils import utils
from keras import activations

from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (18, 6)

# Utility to search for layer index by name. 
# Alternatively we can specify this as -1 since it corresponds to the last layer.
layer_idx = utils.find_layer_idx(model, 'preds')

# Swap softmax with linear
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)

# This is the output node we want to maximize.
filter_idx = 0
img = visualize_activation(model, layer_idx, filter_indices=filter_idx)
plt.imshow(img[..., 0])









    Out[21]:





<matplotlib.image.AxesImage at 0x7fd935d14a50>

Hmm, it sort of looks like a 0, but not as clear as we hoped for. Activation maximization is notorious because regularization parameters needs to be tuned depending on the problem. Lets enumerate all the possible reasons why this didn't work very well.

The input to network is preprocessed to range (0, 1). We should specify input_range = (0., 1.) to constrain the input to this range.
The regularization parameter default weights might be dominating activation maximization loss weight. One way to debug this is to use verbose=True and examine individual loss values.

Lets do these step by step and see if we can improve it.

Debugging step 1: Specifying input_range



In [22]:

    
img = visualize_activation(model, layer_idx, filter_indices=filter_idx, input_range=(0., 1.))
plt.imshow(img[..., 0])









    Out[22]:





<matplotlib.image.AxesImage at 0x7fd93360dfd0>

Much better but still seems noisy. Lets examining the losses with verbose=True and tuning the weights.

Debugging step 2: Tuning regularization weights

One of the issues with activation maximization is that the input can go out of the training distribution space. Total variation and L-p norm are used to provide some hardcoded image priors for natural images. For example, Total variation ensures that images are blobber and not scattered. Unfotunately, sometimes these losses can dominate the main ActivationMaximization loss.

Lets see what individual losses are, with verbose=True



In [23]:

    
img = visualize_activation(model, layer_idx, filter_indices=filter_idx, input_range=(0., 1.), verbose=True)
plt.imshow(img[..., 0])









    



Iteration: 1, named_losses: [('ActivationMax Loss', 0.085303515),
 ('L-6.0 Norm Loss', 0.019828862),
 ('TV(2.0) Loss', 0.095951974)], overall loss: 0.201084345579
Iteration: 2, named_losses: [('ActivationMax Loss', -2.3290093),
 ('L-6.0 Norm Loss', 0.17779008),
 ('TV(2.0) Loss', 507.10727)], overall loss: 504.956054688
Iteration: 3, named_losses: [('ActivationMax Loss', -128.49548),
 ('L-6.0 Norm Loss', 0.19667891),
 ('TV(2.0) Loss', 164.45709)], overall loss: 36.1582946777
Iteration: 4, named_losses: [('ActivationMax Loss', -250.48241),
 ('L-6.0 Norm Loss', 0.19896419),
 ('TV(2.0) Loss', 178.49908)], overall loss: -71.784362793
Iteration: 5, named_losses: [('ActivationMax Loss', -355.06989),
 ('L-6.0 Norm Loss', 0.23281187),
 ('TV(2.0) Loss', 168.41599)], overall loss: -186.421081543
Iteration: 6, named_losses: [('ActivationMax Loss', -434.47665),
 ('L-6.0 Norm Loss', 0.27569067),
 ('TV(2.0) Loss', 198.10806)], overall loss: -236.092895508
Iteration: 7, named_losses: [('ActivationMax Loss', -523.30029),
 ('L-6.0 Norm Loss', 0.30946341),
 ('TV(2.0) Loss', 210.7379)], overall loss: -312.252929688
Iteration: 8, named_losses: [('ActivationMax Loss', -589.32751),
 ('L-6.0 Norm Loss', 0.34706685),
 ('TV(2.0) Loss', 243.71239)], overall loss: -345.268066406
Iteration: 9, named_losses: [('ActivationMax Loss', -649.27594),
 ('L-6.0 Norm Loss', 0.37611905),
 ('TV(2.0) Loss', 256.13412)], overall loss: -392.765716553
Iteration: 10, named_losses: [('ActivationMax Loss', -709.78107),
 ('L-6.0 Norm Loss', 0.41110787),
 ('TV(2.0) Loss', 284.26044)], overall loss: -425.10949707
Iteration: 11, named_losses: [('ActivationMax Loss', -761.53046),
 ('L-6.0 Norm Loss', 0.43992683),
 ('TV(2.0) Loss', 306.72531)], overall loss: -454.365203857
Iteration: 12, named_losses: [('ActivationMax Loss', -803.29578),
 ('L-6.0 Norm Loss', 0.46880898),
 ('TV(2.0) Loss', 327.67728)], overall loss: -475.149688721
Iteration: 13, named_losses: [('ActivationMax Loss', -846.75684),
 ('L-6.0 Norm Loss', 0.49682248),
 ('TV(2.0) Loss', 344.90573)], overall loss: -501.354278564
Iteration: 14, named_losses: [('ActivationMax Loss', -883.35626),
 ('L-6.0 Norm Loss', 0.52348113),
 ('TV(2.0) Loss', 363.5777)], overall loss: -519.255065918
Iteration: 15, named_losses: [('ActivationMax Loss', -919.09418),
 ('L-6.0 Norm Loss', 0.5485183),
 ('TV(2.0) Loss', 378.69067)], overall loss: -539.854980469
Iteration: 16, named_losses: [('ActivationMax Loss', -948.51349),
 ('L-6.0 Norm Loss', 0.56927013),
 ('TV(2.0) Loss', 390.0394)], overall loss: -557.904785156
Iteration: 17, named_losses: [('ActivationMax Loss', -978.81732),
 ('L-6.0 Norm Loss', 0.59408391),
 ('TV(2.0) Loss', 406.41376)], overall loss: -571.809509277
Iteration: 18, named_losses: [('ActivationMax Loss', -1010.2624),
 ('L-6.0 Norm Loss', 0.61176264),
 ('TV(2.0) Loss', 424.86453)], overall loss: -584.786132812
Iteration: 19, named_losses: [('ActivationMax Loss', -1032.5134),
 ('L-6.0 Norm Loss', 0.63450885),
 ('TV(2.0) Loss', 435.95856)], overall loss: -595.920349121
Iteration: 20, named_losses: [('ActivationMax Loss', -1055.4211),
 ('L-6.0 Norm Loss', 0.64917922),
 ('TV(2.0) Loss', 447.13791)], overall loss: -607.634033203
Iteration: 21, named_losses: [('ActivationMax Loss', -1078.2205),
 ('L-6.0 Norm Loss', 0.67029834),
 ('TV(2.0) Loss', 460.73138)], overall loss: -616.818786621
Iteration: 22, named_losses: [('ActivationMax Loss', -1096.8605),
 ('L-6.0 Norm Loss', 0.68686134),
 ('TV(2.0) Loss', 468.72369)], overall loss: -627.449890137
Iteration: 23, named_losses: [('ActivationMax Loss', -1115.5505),
 ('L-6.0 Norm Loss', 0.70358962),
 ('TV(2.0) Loss', 477.83783)], overall loss: -637.009094238
Iteration: 24, named_losses: [('ActivationMax Loss', -1135.0172),
 ('L-6.0 Norm Loss', 0.72281784),
 ('TV(2.0) Loss', 488.59711)], overall loss: -645.69732666
Iteration: 25, named_losses: [('ActivationMax Loss', -1152.8055),
 ('L-6.0 Norm Loss', 0.73808521),
 ('TV(2.0) Loss', 493.71191)], overall loss: -658.35559082
Iteration: 26, named_losses: [('ActivationMax Loss', -1169.7035),
 ('L-6.0 Norm Loss', 0.75386697),
 ('TV(2.0) Loss', 499.82452)], overall loss: -669.125061035
Iteration: 27, named_losses: [('ActivationMax Loss', -1189.5547),
 ('L-6.0 Norm Loss', 0.77207923),
 ('TV(2.0) Loss', 516.74023)], overall loss: -672.042358398
Iteration: 28, named_losses: [('ActivationMax Loss', -1206.5782),
 ('L-6.0 Norm Loss', 0.7868582),
 ('TV(2.0) Loss', 521.85266)], overall loss: -683.938720703
Iteration: 29, named_losses: [('ActivationMax Loss', -1226.3325),
 ('L-6.0 Norm Loss', 0.80321062),
 ('TV(2.0) Loss', 536.8913)], overall loss: -688.638000488
Iteration: 30, named_losses: [('ActivationMax Loss', -1243.972),
 ('L-6.0 Norm Loss', 0.81793976),
 ('TV(2.0) Loss', 539.8042)], overall loss: -703.349853516
Iteration: 31, named_losses: [('ActivationMax Loss', -1262.2374),
 ('L-6.0 Norm Loss', 0.83369249),
 ('TV(2.0) Loss', 555.77899)], overall loss: -705.624694824
Iteration: 32, named_losses: [('ActivationMax Loss', -1277.142),
 ('L-6.0 Norm Loss', 0.84892559),
 ('TV(2.0) Loss', 559.75134)], overall loss: -716.541748047
Iteration: 33, named_losses: [('ActivationMax Loss', -1291.3621),
 ('L-6.0 Norm Loss', 0.86305672),
 ('TV(2.0) Loss', 571.33044)], overall loss: -719.168579102
Iteration: 34, named_losses: [('ActivationMax Loss', -1304.9154),
 ('L-6.0 Norm Loss', 0.87694514),
 ('TV(2.0) Loss', 578.43042)], overall loss: -725.608032227
Iteration: 35, named_losses: [('ActivationMax Loss', -1318.2659),
 ('L-6.0 Norm Loss', 0.89181656),
 ('TV(2.0) Loss', 591.61877)], overall loss: -725.755249023
Iteration: 36, named_losses: [('ActivationMax Loss', -1329.4354),
 ('L-6.0 Norm Loss', 0.90528834),
 ('TV(2.0) Loss', 595.05951)], overall loss: -733.47064209
Iteration: 37, named_losses: [('ActivationMax Loss', -1341.7084),
 ('L-6.0 Norm Loss', 0.91941857),
 ('TV(2.0) Loss', 605.28784)], overall loss: -735.501098633
Iteration: 38, named_losses: [('ActivationMax Loss', -1354.1248),
 ('L-6.0 Norm Loss', 0.93217278),
 ('TV(2.0) Loss', 612.47931)], overall loss: -740.713317871
Iteration: 39, named_losses: [('ActivationMax Loss', -1364.4468),
 ('L-6.0 Norm Loss', 0.94575441),
 ('TV(2.0) Loss', 619.2569)], overall loss: -744.24407959
Iteration: 40, named_losses: [('ActivationMax Loss', -1374.3206),
 ('L-6.0 Norm Loss', 0.9577564),
 ('TV(2.0) Loss', 625.97729)], overall loss: -747.385498047
Iteration: 41, named_losses: [('ActivationMax Loss', -1384.3914),
 ('L-6.0 Norm Loss', 0.96838021),
 ('TV(2.0) Loss', 633.25616)], overall loss: -750.166809082
Iteration: 42, named_losses: [('ActivationMax Loss', -1393.1029),
 ('L-6.0 Norm Loss', 0.98039478),
 ('TV(2.0) Loss', 639.09497)], overall loss: -753.027587891
Iteration: 43, named_losses: [('ActivationMax Loss', -1402.049),
 ('L-6.0 Norm Loss', 0.99183488),
 ('TV(2.0) Loss', 644.44427)], overall loss: -756.612854004
Iteration: 44, named_losses: [('ActivationMax Loss', -1410.5962),
 ('L-6.0 Norm Loss', 1.0034488),
 ('TV(2.0) Loss', 651.58691)], overall loss: -758.005859375
Iteration: 45, named_losses: [('ActivationMax Loss', -1419.7144),
 ('L-6.0 Norm Loss', 1.0143936),
 ('TV(2.0) Loss', 658.54822)], overall loss: -760.151733398
Iteration: 46, named_losses: [('ActivationMax Loss', -1427.2026),
 ('L-6.0 Norm Loss', 1.0241909),
 ('TV(2.0) Loss', 662.922)], overall loss: -763.256469727
Iteration: 47, named_losses: [('ActivationMax Loss', -1435.6279),
 ('L-6.0 Norm Loss', 1.0342484),
 ('TV(2.0) Loss', 667.56097)], overall loss: -767.032653809
Iteration: 48, named_losses: [('ActivationMax Loss', -1440.479),
 ('L-6.0 Norm Loss', 1.0420389),
 ('TV(2.0) Loss', 672.19409)], overall loss: -767.242919922
Iteration: 49, named_losses: [('ActivationMax Loss', -1453.1812),
 ('L-6.0 Norm Loss', 1.0515953),
 ('TV(2.0) Loss', 680.70709)], overall loss: -771.422424316
Iteration: 50, named_losses: [('ActivationMax Loss', -1454.2151),
 ('L-6.0 Norm Loss', 1.0597379),
 ('TV(2.0) Loss', 682.62335)], overall loss: -770.532043457
Iteration: 51, named_losses: [('ActivationMax Loss', -1468.5298),
 ('L-6.0 Norm Loss', 1.0692028),
 ('TV(2.0) Loss', 692.5365)], overall loss: -774.924072266
Iteration: 52, named_losses: [('ActivationMax Loss', -1468.5466),
 ('L-6.0 Norm Loss', 1.0780802),
 ('TV(2.0) Loss', 694.88562)], overall loss: -772.582885742
Iteration: 53, named_losses: [('ActivationMax Loss', -1478.1926),
 ('L-6.0 Norm Loss', 1.0857446),
 ('TV(2.0) Loss', 699.60638)], overall loss: -777.500549316
Iteration: 54, named_losses: [('ActivationMax Loss', -1483.5609),
 ('L-6.0 Norm Loss', 1.0934379),
 ('TV(2.0) Loss', 706.73547)], overall loss: -775.732055664
Iteration: 55, named_losses: [('ActivationMax Loss', -1491.7773),
 ('L-6.0 Norm Loss', 1.1018604),
 ('TV(2.0) Loss', 710.49097)], overall loss: -780.184570312
Iteration: 56, named_losses: [('ActivationMax Loss', -1496.1121),
 ('L-6.0 Norm Loss', 1.1094059),
 ('TV(2.0) Loss', 716.18457)], overall loss: -778.818115234
Iteration: 57, named_losses: [('ActivationMax Loss', -1503.3049),
 ('L-6.0 Norm Loss', 1.1176937),
 ('TV(2.0) Loss', 719.49158)], overall loss: -782.695678711
Iteration: 58, named_losses: [('ActivationMax Loss', -1507.1075),
 ('L-6.0 Norm Loss', 1.1252179),
 ('TV(2.0) Loss', 723.2171)], overall loss: -782.765197754
Iteration: 59, named_losses: [('ActivationMax Loss', -1508.6431),
 ('L-6.0 Norm Loss', 1.1337999),
 ('TV(2.0) Loss', 722.63678)], overall loss: -784.872497559
Iteration: 60, named_losses: [('ActivationMax Loss', -1514.7773),
 ('L-6.0 Norm Loss', 1.1402849),
 ('TV(2.0) Loss', 728.61115)], overall loss: -785.025939941
Iteration: 61, named_losses: [('ActivationMax Loss', -1514.7411),
 ('L-6.0 Norm Loss', 1.1487488),
 ('TV(2.0) Loss', 727.47565)], overall loss: -786.116638184
Iteration: 62, named_losses: [('ActivationMax Loss', -1521.7354),
 ('L-6.0 Norm Loss', 1.1555879),
 ('TV(2.0) Loss', 731.41626)], overall loss: -789.163452148
Iteration: 63, named_losses: [('ActivationMax Loss', -1522.7675),
 ('L-6.0 Norm Loss', 1.1635603),
 ('TV(2.0) Loss', 733.62488)], overall loss: -787.979003906
Iteration: 64, named_losses: [('ActivationMax Loss', -1529.7563),
 ('L-6.0 Norm Loss', 1.1674472),
 ('TV(2.0) Loss', 736.68823)], overall loss: -791.900634766
Iteration: 65, named_losses: [('ActivationMax Loss', -1531.0713),
 ('L-6.0 Norm Loss', 1.1751125),
 ('TV(2.0) Loss', 739.92639)], overall loss: -789.969726562
Iteration: 66, named_losses: [('ActivationMax Loss', -1534.9249),
 ('L-6.0 Norm Loss', 1.1804783),
 ('TV(2.0) Loss', 740.112)], overall loss: -793.632507324
Iteration: 67, named_losses: [('ActivationMax Loss', -1535.5088),
 ('L-6.0 Norm Loss', 1.1873333),
 ('TV(2.0) Loss', 741.09711)], overall loss: -793.224304199
Iteration: 68, named_losses: [('ActivationMax Loss', -1539.807),
 ('L-6.0 Norm Loss', 1.1929805),
 ('TV(2.0) Loss', 742.85437)], overall loss: -795.759643555
Iteration: 69, named_losses: [('ActivationMax Loss', -1537.9623),
 ('L-6.0 Norm Loss', 1.1999267),
 ('TV(2.0) Loss', 742.83746)], overall loss: -793.924865723
Iteration: 70, named_losses: [('ActivationMax Loss', -1549.0408),
 ('L-6.0 Norm Loss', 1.2060388),
 ('TV(2.0) Loss', 749.69873)], overall loss: -798.135986328
Iteration: 71, named_losses: [('ActivationMax Loss', -1549.3425),
 ('L-6.0 Norm Loss', 1.2124674),
 ('TV(2.0) Loss', 750.94574)], overall loss: -797.184265137
Iteration: 72, named_losses: [('ActivationMax Loss', -1553.8987),
 ('L-6.0 Norm Loss', 1.21632),
 ('TV(2.0) Loss', 752.06598)], overall loss: -800.616394043
Iteration: 73, named_losses: [('ActivationMax Loss', -1554.4263),
 ('L-6.0 Norm Loss', 1.2229993),
 ('TV(2.0) Loss', 754.32281)], overall loss: -798.880432129
Iteration: 74, named_losses: [('ActivationMax Loss', -1559.6265),
 ('L-6.0 Norm Loss', 1.2293643),
 ('TV(2.0) Loss', 758.35571)], overall loss: -800.041381836
Iteration: 75, named_losses: [('ActivationMax Loss', -1561.2106),
 ('L-6.0 Norm Loss', 1.2329044),
 ('TV(2.0) Loss', 760.01758)], overall loss: -799.960083008
Iteration: 76, named_losses: [('ActivationMax Loss', -1562.5452),
 ('L-6.0 Norm Loss', 1.2385361),
 ('TV(2.0) Loss', 759.76331)], overall loss: -801.543334961
Iteration: 77, named_losses: [('ActivationMax Loss', -1564.2513),
 ('L-6.0 Norm Loss', 1.2413583),
 ('TV(2.0) Loss', 760.64935)], overall loss: -802.360656738
Iteration: 78, named_losses: [('ActivationMax Loss', -1566.9346),
 ('L-6.0 Norm Loss', 1.2466581),
 ('TV(2.0) Loss', 764.04144)], overall loss: -801.64642334
Iteration: 79, named_losses: [('ActivationMax Loss', -1570.8954),
 ('L-6.0 Norm Loss', 1.2509888),
 ('TV(2.0) Loss', 766.50098)], overall loss: -803.143432617
Iteration: 80, named_losses: [('ActivationMax Loss', -1572.2623),
 ('L-6.0 Norm Loss', 1.2554729),
 ('TV(2.0) Loss', 766.65845)], overall loss: -804.348388672
Iteration: 81, named_losses: [('ActivationMax Loss', -1574.7688),
 ('L-6.0 Norm Loss', 1.2608629),
 ('TV(2.0) Loss', 768.88452)], overall loss: -804.623413086
Iteration: 82, named_losses: [('ActivationMax Loss', -1574.8457),
 ('L-6.0 Norm Loss', 1.264998),
 ('TV(2.0) Loss', 767.55579)], overall loss: -806.024902344
Iteration: 83, named_losses: [('ActivationMax Loss', -1578.4631),
 ('L-6.0 Norm Loss', 1.2702675),
 ('TV(2.0) Loss', 770.93079)], overall loss: -806.262084961
Iteration: 84, named_losses: [('ActivationMax Loss', -1579.1116),
 ('L-6.0 Norm Loss', 1.2734473),
 ('TV(2.0) Loss', 770.62146)], overall loss: -807.216674805
Iteration: 85, named_losses: [('ActivationMax Loss', -1581.9132),
 ('L-6.0 Norm Loss', 1.2792308),
 ('TV(2.0) Loss', 772.18756)], overall loss: -808.446472168
Iteration: 86, named_losses: [('ActivationMax Loss', -1579.4325),
 ('L-6.0 Norm Loss', 1.2830144),
 ('TV(2.0) Loss', 766.79077)], overall loss: -811.358764648
Iteration: 87, named_losses: [('ActivationMax Loss', -1585.8596),
 ('L-6.0 Norm Loss', 1.2859445),
 ('TV(2.0) Loss', 775.53821)], overall loss: -809.035522461
Iteration: 88, named_losses: [('ActivationMax Loss', -1585.7672),
 ('L-6.0 Norm Loss', 1.2900306),
 ('TV(2.0) Loss', 774.06342)], overall loss: -810.413757324
Iteration: 89, named_losses: [('ActivationMax Loss', -1591.1372),
 ('L-6.0 Norm Loss', 1.2948806),
 ('TV(2.0) Loss', 778.63391)], overall loss: -811.208374023
Iteration: 90, named_losses: [('ActivationMax Loss', -1590.0701),
 ('L-6.0 Norm Loss', 1.2955327),
 ('TV(2.0) Loss', 777.77234)], overall loss: -811.002197266
Iteration: 91, named_losses: [('ActivationMax Loss', -1590.9371),
 ('L-6.0 Norm Loss', 1.3006396),
 ('TV(2.0) Loss', 777.98926)], overall loss: -811.647216797
Iteration: 92, named_losses: [('ActivationMax Loss', -1597.2491),
 ('L-6.0 Norm Loss', 1.3033614),
 ('TV(2.0) Loss', 783.2879)], overall loss: -812.657897949
Iteration: 93, named_losses: [('ActivationMax Loss', -1593.3665),
 ('L-6.0 Norm Loss', 1.3075004),
 ('TV(2.0) Loss', 779.44653)], overall loss: -812.612426758
Iteration: 94, named_losses: [('ActivationMax Loss', -1597.2029),
 ('L-6.0 Norm Loss', 1.3099935),
 ('TV(2.0) Loss', 781.82526)], overall loss: -814.067687988
Iteration: 95, named_losses: [('ActivationMax Loss', -1597.9543),
 ('L-6.0 Norm Loss', 1.3138223),
 ('TV(2.0) Loss', 783.27612)], overall loss: -813.364379883
Iteration: 96, named_losses: [('ActivationMax Loss', -1600.9514),
 ('L-6.0 Norm Loss', 1.3189675),
 ('TV(2.0) Loss', 784.29456)], overall loss: -815.337890625
Iteration: 97, named_losses: [('ActivationMax Loss', -1603.2629),
 ('L-6.0 Norm Loss', 1.3227088),
 ('TV(2.0) Loss', 787.86951)], overall loss: -814.070678711
Iteration: 98, named_losses: [('ActivationMax Loss', -1603.4062),
 ('L-6.0 Norm Loss', 1.3246108),
 ('TV(2.0) Loss', 786.18884)], overall loss: -815.892822266
Iteration: 99, named_losses: [('ActivationMax Loss', -1602.4215),
 ('L-6.0 Norm Loss', 1.3267232),
 ('TV(2.0) Loss', 784.94574)], overall loss: -816.148986816
Iteration: 100, named_losses: [('ActivationMax Loss', -1606.3367),
 ('L-6.0 Norm Loss', 1.3313514),
 ('TV(2.0) Loss', 788.65411)], overall loss: -816.351257324
Iteration: 101, named_losses: [('ActivationMax Loss', -1605.1202),
 ('L-6.0 Norm Loss', 1.3336935),
 ('TV(2.0) Loss', 787.86621)], overall loss: -815.920288086
Iteration: 102, named_losses: [('ActivationMax Loss', -1609.257),
 ('L-6.0 Norm Loss', 1.3371928),
 ('TV(2.0) Loss', 790.25415)], overall loss: -817.665649414
Iteration: 103, named_losses: [('ActivationMax Loss', -1610.245),
 ('L-6.0 Norm Loss', 1.3413918),
 ('TV(2.0) Loss', 791.96606)], overall loss: -816.9375
Iteration: 104, named_losses: [('ActivationMax Loss', -1610.9143),
 ('L-6.0 Norm Loss', 1.3415542),
 ('TV(2.0) Loss', 789.74146)], overall loss: -819.831298828
Iteration: 105, named_losses: [('ActivationMax Loss', -1609.8514),
 ('L-6.0 Norm Loss', 1.3457475),
 ('TV(2.0) Loss', 789.99353)], overall loss: -818.512207031
Iteration: 106, named_losses: [('ActivationMax Loss', -1612.2958),
 ('L-6.0 Norm Loss', 1.3498583),
 ('TV(2.0) Loss', 789.27185)], overall loss: -821.674072266
Iteration: 107, named_losses: [('ActivationMax Loss', -1613.5688),
 ('L-6.0 Norm Loss', 1.3565289),
 ('TV(2.0) Loss', 793.34521)], overall loss: -818.86706543
Iteration: 108, named_losses: [('ActivationMax Loss', -1617.8153),
 ('L-6.0 Norm Loss', 1.358252),
 ('TV(2.0) Loss', 794.26929)], overall loss: -822.187744141
Iteration: 109, named_losses: [('ActivationMax Loss', -1617.1315),
 ('L-6.0 Norm Loss', 1.3626406),
 ('TV(2.0) Loss', 794.89624)], overall loss: -820.872558594
Iteration: 110, named_losses: [('ActivationMax Loss', -1619.2756),
 ('L-6.0 Norm Loss', 1.3654678),
 ('TV(2.0) Loss', 794.74115)], overall loss: -823.169006348
Iteration: 111, named_losses: [('ActivationMax Loss', -1620.2999),
 ('L-6.0 Norm Loss', 1.368789),
 ('TV(2.0) Loss', 797.3844)], overall loss: -821.54675293
Iteration: 112, named_losses: [('ActivationMax Loss', -1619.2432),
 ('L-6.0 Norm Loss', 1.3725339),
 ('TV(2.0) Loss', 793.66309)], overall loss: -824.207519531
Iteration: 113, named_losses: [('ActivationMax Loss', -1625.923),
 ('L-6.0 Norm Loss', 1.3758172),
 ('TV(2.0) Loss', 800.75488)], overall loss: -823.792236328
Iteration: 114, named_losses: [('ActivationMax Loss', -1622.631),
 ('L-6.0 Norm Loss', 1.3799908),
 ('TV(2.0) Loss', 797.35974)], overall loss: -823.891235352
Iteration: 115, named_losses: [('ActivationMax Loss', -1630.325),
 ('L-6.0 Norm Loss', 1.384307),
 ('TV(2.0) Loss', 806.6814)], overall loss: -822.259277344
Iteration: 116, named_losses: [('ActivationMax Loss', -1627.0356),
 ('L-6.0 Norm Loss', 1.3886331),
 ('TV(2.0) Loss', 802.25623)], overall loss: -823.39074707
Iteration: 117, named_losses: [('ActivationMax Loss', -1632.6683),
 ('L-6.0 Norm Loss', 1.3907378),
 ('TV(2.0) Loss', 809.20844)], overall loss: -822.069152832
Iteration: 118, named_losses: [('ActivationMax Loss', -1632.0132),
 ('L-6.0 Norm Loss', 1.3947749),
 ('TV(2.0) Loss', 806.38666)], overall loss: -824.231750488
Iteration: 119, named_losses: [('ActivationMax Loss', -1633.7933),
 ('L-6.0 Norm Loss', 1.3970679),
 ('TV(2.0) Loss', 809.47327)], overall loss: -822.922973633
Iteration: 120, named_losses: [('ActivationMax Loss', -1631.33),
 ('L-6.0 Norm Loss', 1.4000779),
 ('TV(2.0) Loss', 804.59418)], overall loss: -825.335754395
Iteration: 121, named_losses: [('ActivationMax Loss', -1638.6511),
 ('L-6.0 Norm Loss', 1.4029664),
 ('TV(2.0) Loss', 812.93854)], overall loss: -824.309631348
Iteration: 122, named_losses: [('ActivationMax Loss', -1631.8582),
 ('L-6.0 Norm Loss', 1.4053441),
 ('TV(2.0) Loss', 804.75452)], overall loss: -825.698242188
Iteration: 123, named_losses: [('ActivationMax Loss', -1638.329),
 ('L-6.0 Norm Loss', 1.4088998),
 ('TV(2.0) Loss', 811.19293)], overall loss: -825.727111816
Iteration: 124, named_losses: [('ActivationMax Loss', -1632.4836),
 ('L-6.0 Norm Loss', 1.4110554),
 ('TV(2.0) Loss', 804.57092)], overall loss: -826.501708984
Iteration: 125, named_losses: [('ActivationMax Loss', -1639.8102),
 ('L-6.0 Norm Loss', 1.4141355),
 ('TV(2.0) Loss', 812.0838)], overall loss: -826.312194824
Iteration: 126, named_losses: [('ActivationMax Loss', -1630.1312),
 ('L-6.0 Norm Loss', 1.4141332),
 ('TV(2.0) Loss', 800.64581)], overall loss: -828.071228027
Iteration: 127, named_losses: [('ActivationMax Loss', -1639.1195),
 ('L-6.0 Norm Loss', 1.4192623),
 ('TV(2.0) Loss', 809.91083)], overall loss: -827.789367676
Iteration: 128, named_losses: [('ActivationMax Loss', -1630.7975),
 ('L-6.0 Norm Loss', 1.4200757),
 ('TV(2.0) Loss', 800.08356)], overall loss: -829.293884277
Iteration: 129, named_losses: [('ActivationMax Loss', -1638.1284),
 ('L-6.0 Norm Loss', 1.4252733),
 ('TV(2.0) Loss', 809.07318)], overall loss: -827.629943848
Iteration: 130, named_losses: [('ActivationMax Loss', -1636.0492),
 ('L-6.0 Norm Loss', 1.4285793),
 ('TV(2.0) Loss', 803.84949)], overall loss: -830.771118164
Iteration: 131, named_losses: [('ActivationMax Loss', -1641.9642),
 ('L-6.0 Norm Loss', 1.433162),
 ('TV(2.0) Loss', 811.50299)], overall loss: -829.028137207
Iteration: 132, named_losses: [('ActivationMax Loss', -1639.944),
 ('L-6.0 Norm Loss', 1.4367878),
 ('TV(2.0) Loss', 807.71741)], overall loss: -830.789794922
Iteration: 133, named_losses: [('ActivationMax Loss', -1642.401),
 ('L-6.0 Norm Loss', 1.4381592),
 ('TV(2.0) Loss', 811.63275)], overall loss: -829.33013916
Iteration: 134, named_losses: [('ActivationMax Loss', -1643.0408),
 ('L-6.0 Norm Loss', 1.442253),
 ('TV(2.0) Loss', 810.08728)], overall loss: -831.511230469
Iteration: 135, named_losses: [('ActivationMax Loss', -1647.0085),
 ('L-6.0 Norm Loss', 1.444943),
 ('TV(2.0) Loss', 815.15967)], overall loss: -830.403930664
Iteration: 136, named_losses: [('ActivationMax Loss', -1644.9188),
 ('L-6.0 Norm Loss', 1.4477811),
 ('TV(2.0) Loss', 811.31506)], overall loss: -832.156005859
Iteration: 137, named_losses: [('ActivationMax Loss', -1653.4288),
 ('L-6.0 Norm Loss', 1.4509344),
 ('TV(2.0) Loss', 821.15198)], overall loss: -830.825927734
Iteration: 138, named_losses: [('ActivationMax Loss', -1648.1528),
 ('L-6.0 Norm Loss', 1.4527802),
 ('TV(2.0) Loss', 814.87396)], overall loss: -831.82611084
Iteration: 139, named_losses: [('ActivationMax Loss', -1649.7593),
 ('L-6.0 Norm Loss', 1.4559247),
 ('TV(2.0) Loss', 817.54858)], overall loss: -830.754760742
Iteration: 140, named_losses: [('ActivationMax Loss', -1651.0073),
 ('L-6.0 Norm Loss', 1.4594543),
 ('TV(2.0) Loss', 816.99554)], overall loss: -832.552307129
Iteration: 141, named_losses: [('ActivationMax Loss', -1652.6647),
 ('L-6.0 Norm Loss', 1.4621137),
 ('TV(2.0) Loss', 819.40613)], overall loss: -831.796386719
Iteration: 142, named_losses: [('ActivationMax Loss', -1654.1587),
 ('L-6.0 Norm Loss', 1.4648318),
 ('TV(2.0) Loss', 820.21576)], overall loss: -832.478088379
Iteration: 143, named_losses: [('ActivationMax Loss', -1651.4489),
 ('L-6.0 Norm Loss', 1.465472),
 ('TV(2.0) Loss', 812.49847)], overall loss: -837.484924316
Iteration: 144, named_losses: [('ActivationMax Loss', -1655.8921),
 ('L-6.0 Norm Loss', 1.4706283),
 ('TV(2.0) Loss', 819.5379)], overall loss: -834.883605957
Iteration: 145, named_losses: [('ActivationMax Loss', -1651.3011),
 ('L-6.0 Norm Loss', 1.4713544),
 ('TV(2.0) Loss', 814.84924)], overall loss: -834.98059082
Iteration: 146, named_losses: [('ActivationMax Loss', -1656.283),
 ('L-6.0 Norm Loss', 1.4738773),
 ('TV(2.0) Loss', 823.04236)], overall loss: -831.766723633
Iteration: 147, named_losses: [('ActivationMax Loss', -1660.2869),
 ('L-6.0 Norm Loss', 1.4773164),
 ('TV(2.0) Loss', 824.23651)], overall loss: -834.573059082
Iteration: 148, named_losses: [('ActivationMax Loss', -1662.1096),
 ('L-6.0 Norm Loss', 1.4793681),
 ('TV(2.0) Loss', 828.23236)], overall loss: -832.397888184
Iteration: 149, named_losses: [('ActivationMax Loss', -1661.8433),
 ('L-6.0 Norm Loss', 1.4821515),
 ('TV(2.0) Loss', 825.2392)], overall loss: -835.121887207
Iteration: 150, named_losses: [('ActivationMax Loss', -1665.4824),
 ('L-6.0 Norm Loss', 1.4861856),
 ('TV(2.0) Loss', 829.73688)], overall loss: -834.259338379
Iteration: 151, named_losses: [('ActivationMax Loss', -1664.7596),
 ('L-6.0 Norm Loss', 1.4876471),
 ('TV(2.0) Loss', 826.73108)], overall loss: -836.540893555
Iteration: 152, named_losses: [('ActivationMax Loss', -1666.2804),
 ('L-6.0 Norm Loss', 1.4919786),
 ('TV(2.0) Loss', 830.87958)], overall loss: -833.908874512
Iteration: 153, named_losses: [('ActivationMax Loss', -1668.1177),
 ('L-6.0 Norm Loss', 1.4925706),
 ('TV(2.0) Loss', 830.95911)], overall loss: -835.666015625
Iteration: 154, named_losses: [('ActivationMax Loss', -1668.7156),
 ('L-6.0 Norm Loss', 1.4961685),
 ('TV(2.0) Loss', 831.15045)], overall loss: -836.068908691
Iteration: 155, named_losses: [('ActivationMax Loss', -1669.6495),
 ('L-6.0 Norm Loss', 1.4984686),
 ('TV(2.0) Loss', 831.70325)], overall loss: -836.447875977
Iteration: 156, named_losses: [('ActivationMax Loss', -1673.0961),
 ('L-6.0 Norm Loss', 1.5010495),
 ('TV(2.0) Loss', 835.07239)], overall loss: -836.522583008
Iteration: 157, named_losses: [('ActivationMax Loss', -1672.6519),
 ('L-6.0 Norm Loss', 1.5066335),
 ('TV(2.0) Loss', 835.20679)], overall loss: -835.938476562
Iteration: 158, named_losses: [('ActivationMax Loss', -1677.0101),
 ('L-6.0 Norm Loss', 1.5054392),
 ('TV(2.0) Loss', 837.4469)], overall loss: -838.057739258
Iteration: 159, named_losses: [('ActivationMax Loss', -1675.916),
 ('L-6.0 Norm Loss', 1.5089097),
 ('TV(2.0) Loss', 836.17157)], overall loss: -838.235534668
Iteration: 160, named_losses: [('ActivationMax Loss', -1674.426),
 ('L-6.0 Norm Loss', 1.5099328),
 ('TV(2.0) Loss', 834.57648)], overall loss: -838.339660645
Iteration: 161, named_losses: [('ActivationMax Loss', -1676.6719),
 ('L-6.0 Norm Loss', 1.5128522),
 ('TV(2.0) Loss', 836.09076)], overall loss: -839.06829834
Iteration: 162, named_losses: [('ActivationMax Loss', -1676.5416),
 ('L-6.0 Norm Loss', 1.5148972),
 ('TV(2.0) Loss', 835.69391)], overall loss: -839.332824707
Iteration: 163, named_losses: [('ActivationMax Loss', -1676.1887),
 ('L-6.0 Norm Loss', 1.516475),
 ('TV(2.0) Loss', 834.33459)], overall loss: -840.337646484
Iteration: 164, named_losses: [('ActivationMax Loss', -1677.428),
 ('L-6.0 Norm Loss', 1.5198225),
 ('TV(2.0) Loss', 835.4469)], overall loss: -840.461303711
Iteration: 165, named_losses: [('ActivationMax Loss', -1677.0521),
 ('L-6.0 Norm Loss', 1.5217334),
 ('TV(2.0) Loss', 833.99994)], overall loss: -841.530456543
Iteration: 166, named_losses: [('ActivationMax Loss', -1680.4186),
 ('L-6.0 Norm Loss', 1.527281),
 ('TV(2.0) Loss', 839.37067)], overall loss: -839.520690918
Iteration: 167, named_losses: [('ActivationMax Loss', -1682.5245),
 ('L-6.0 Norm Loss', 1.5286567),
 ('TV(2.0) Loss', 839.15784)], overall loss: -841.838012695
Iteration: 168, named_losses: [('ActivationMax Loss', -1679.6903),
 ('L-6.0 Norm Loss', 1.5303226),
 ('TV(2.0) Loss', 837.42731)], overall loss: -840.732727051
Iteration: 169, named_losses: [('ActivationMax Loss', -1684.0806),
 ('L-6.0 Norm Loss', 1.5321445),
 ('TV(2.0) Loss', 840.0152)], overall loss: -842.53326416
Iteration: 170, named_losses: [('ActivationMax Loss', -1681.1489),
 ('L-6.0 Norm Loss', 1.5356567),
 ('TV(2.0) Loss', 837.11945)], overall loss: -842.493835449
Iteration: 171, named_losses: [('ActivationMax Loss', -1683.8822),
 ('L-6.0 Norm Loss', 1.5385299),
 ('TV(2.0) Loss', 839.58362)], overall loss: -842.760009766
Iteration: 172, named_losses: [('ActivationMax Loss', -1680.66),
 ('L-6.0 Norm Loss', 1.5389018),
 ('TV(2.0) Loss', 833.68958)], overall loss: -845.431518555
Iteration: 173, named_losses: [('ActivationMax Loss', -1684.0814),
 ('L-6.0 Norm Loss', 1.5411844),
 ('TV(2.0) Loss', 839.33081)], overall loss: -843.209472656
Iteration: 174, named_losses: [('ActivationMax Loss', -1681.9932),
 ('L-6.0 Norm Loss', 1.5433018),
 ('TV(2.0) Loss', 837.27722)], overall loss: -843.172607422
Iteration: 175, named_losses: [('ActivationMax Loss', -1686.0037),
 ('L-6.0 Norm Loss', 1.5452924),
 ('TV(2.0) Loss', 841.27289)], overall loss: -843.18548584
Iteration: 176, named_losses: [('ActivationMax Loss', -1687.5237),
 ('L-6.0 Norm Loss', 1.5497506),
 ('TV(2.0) Loss', 845.2395)], overall loss: -840.734375
Iteration: 177, named_losses: [('ActivationMax Loss', -1692.7676),
 ('L-6.0 Norm Loss', 1.5519809),
 ('TV(2.0) Loss', 847.61469)], overall loss: -843.600891113
Iteration: 178, named_losses: [('ActivationMax Loss', -1691.171),
 ('L-6.0 Norm Loss', 1.5546336),
 ('TV(2.0) Loss', 848.039)], overall loss: -841.577331543
Iteration: 179, named_losses: [('ActivationMax Loss', -1692.5399),
 ('L-6.0 Norm Loss', 1.5569656),
 ('TV(2.0) Loss', 846.35913)], overall loss: -844.623779297
Iteration: 180, named_losses: [('ActivationMax Loss', -1690.7684),
 ('L-6.0 Norm Loss', 1.558495),
 ('TV(2.0) Loss', 846.54626)], overall loss: -842.663696289
Iteration: 181, named_losses: [('ActivationMax Loss', -1694.9828),
 ('L-6.0 Norm Loss', 1.558827),
 ('TV(2.0) Loss', 848.42737)], overall loss: -844.996582031
Iteration: 182, named_losses: [('ActivationMax Loss', -1693.8423),
 ('L-6.0 Norm Loss', 1.5600398),
 ('TV(2.0) Loss', 847.75787)], overall loss: -844.524353027
Iteration: 183, named_losses: [('ActivationMax Loss', -1692.3008),
 ('L-6.0 Norm Loss', 1.5624027),
 ('TV(2.0) Loss', 846.34583)], overall loss: -844.392578125
Iteration: 184, named_losses: [('ActivationMax Loss', -1690.6182),
 ('L-6.0 Norm Loss', 1.5639281),
 ('TV(2.0) Loss', 845.07623)], overall loss: -843.977966309
Iteration: 185, named_losses: [('ActivationMax Loss', -1694.6195),
 ('L-6.0 Norm Loss', 1.5679384),
 ('TV(2.0) Loss', 848.89496)], overall loss: -844.156555176
Iteration: 186, named_losses: [('ActivationMax Loss', -1694.5382),
 ('L-6.0 Norm Loss', 1.5688758),
 ('TV(2.0) Loss', 849.05872)], overall loss: -843.910644531
Iteration: 187, named_losses: [('ActivationMax Loss', -1700.241),
 ('L-6.0 Norm Loss', 1.5718434),
 ('TV(2.0) Loss', 853.30139)], overall loss: -845.367675781
Iteration: 188, named_losses: [('ActivationMax Loss', -1694.207),
 ('L-6.0 Norm Loss', 1.5731733),
 ('TV(2.0) Loss', 848.63531)], overall loss: -843.998596191
Iteration: 189, named_losses: [('ActivationMax Loss', -1702.188),
 ('L-6.0 Norm Loss', 1.5767503),
 ('TV(2.0) Loss', 855.16302)], overall loss: -845.448181152
Iteration: 190, named_losses: [('ActivationMax Loss', -1696.559),
 ('L-6.0 Norm Loss', 1.5781298),
 ('TV(2.0) Loss', 848.68616)], overall loss: -846.294677734
Iteration: 191, named_losses: [('ActivationMax Loss', -1698.0985),
 ('L-6.0 Norm Loss', 1.5782369),
 ('TV(2.0) Loss', 849.00452)], overall loss: -847.51574707
Iteration: 192, named_losses: [('ActivationMax Loss', -1695.0345),
 ('L-6.0 Norm Loss', 1.5810226),
 ('TV(2.0) Loss', 846.17896)], overall loss: -847.274536133
Iteration: 193, named_losses: [('ActivationMax Loss', -1698.2462),
 ('L-6.0 Norm Loss', 1.5834887),
 ('TV(2.0) Loss', 846.41547)], overall loss: -850.247253418
Iteration: 194, named_losses: [('ActivationMax Loss', -1698.2938),
 ('L-6.0 Norm Loss', 1.5867252),
 ('TV(2.0) Loss', 846.93781)], overall loss: -849.769348145
Iteration: 195, named_losses: [('ActivationMax Loss', -1700.386),
 ('L-6.0 Norm Loss', 1.5881807),
 ('TV(2.0) Loss', 848.39166)], overall loss: -850.406188965
Iteration: 196, named_losses: [('ActivationMax Loss', -1703.4745),
 ('L-6.0 Norm Loss', 1.5905124),
 ('TV(2.0) Loss', 854.04553)], overall loss: -847.838500977
Iteration: 197, named_losses: [('ActivationMax Loss', -1706.6498),
 ('L-6.0 Norm Loss', 1.5927349),
 ('TV(2.0) Loss', 856.89001)], overall loss: -848.166992188
Iteration: 198, named_losses: [('ActivationMax Loss', -1710.0554),
 ('L-6.0 Norm Loss', 1.5939628),
 ('TV(2.0) Loss', 859.12634)], overall loss: -849.335083008
Iteration: 199, named_losses: [('ActivationMax Loss', -1707.3005),
 ('L-6.0 Norm Loss', 1.5956845),
 ('TV(2.0) Loss', 855.88831)], overall loss: -849.81652832
Iteration: 200, named_losses: [('ActivationMax Loss', -1709.6353),
 ('L-6.0 Norm Loss', 1.5962927),
 ('TV(2.0) Loss', 858.9491)], overall loss: -849.08984375






    Out[23]:





<matplotlib.image.AxesImage at 0x7fd9332cbe90>

In this case, ActivationMax Loss is not bouncing bouncing around and converging? Perhaps we could get that loss to be lower by reducing weights of other losses that might be dominating the overall loss being minimized.

The simplest way to tune these weights is to first start with 0. weights for all regularization losses.



In [24]:

    
img = visualize_activation(model, layer_idx, filter_indices=filter_idx, input_range=(0., 1.), 
                           tv_weight=0., lp_norm_weight=0., verbose=True)
plt.imshow(img[..., 0])









    



Iteration: 1, named_losses: [('ActivationMax Loss', 0.2693989)], overall loss: 0.269398897886
Iteration: 2, named_losses: [('ActivationMax Loss', -6.6923928)], overall loss: -6.69239282608
Iteration: 3, named_losses: [('ActivationMax Loss', -225.87207)], overall loss: -225.872070312
Iteration: 4, named_losses: [('ActivationMax Loss', -462.48257)], overall loss: -462.482574463
Iteration: 5, named_losses: [('ActivationMax Loss', -689.16077)], overall loss: -689.160766602
Iteration: 6, named_losses: [('ActivationMax Loss', -889.75427)], overall loss: -889.754272461
Iteration: 7, named_losses: [('ActivationMax Loss', -1073.9482)], overall loss: -1073.94824219
Iteration: 8, named_losses: [('ActivationMax Loss', -1242.8525)], overall loss: -1242.85253906
Iteration: 9, named_losses: [('ActivationMax Loss', -1402.7773)], overall loss: -1402.77734375
Iteration: 10, named_losses: [('ActivationMax Loss', -1558.7063)], overall loss: -1558.70629883
Iteration: 11, named_losses: [('ActivationMax Loss', -1701.7627)], overall loss: -1701.76269531
Iteration: 12, named_losses: [('ActivationMax Loss', -1840.2097)], overall loss: -1840.2097168
Iteration: 13, named_losses: [('ActivationMax Loss', -1975.5297)], overall loss: -1975.52966309
Iteration: 14, named_losses: [('ActivationMax Loss', -2106.1282)], overall loss: -2106.12817383
Iteration: 15, named_losses: [('ActivationMax Loss', -2241.844)], overall loss: -2241.84399414
Iteration: 16, named_losses: [('ActivationMax Loss', -2373.1284)], overall loss: -2373.12841797
Iteration: 17, named_losses: [('ActivationMax Loss', -2499.5166)], overall loss: -2499.51660156
Iteration: 18, named_losses: [('ActivationMax Loss', -2621.3357)], overall loss: -2621.33569336
Iteration: 19, named_losses: [('ActivationMax Loss', -2742.8274)], overall loss: -2742.82739258
Iteration: 20, named_losses: [('ActivationMax Loss', -2862.6206)], overall loss: -2862.62060547
Iteration: 21, named_losses: [('ActivationMax Loss', -2978.5793)], overall loss: -2978.5793457
Iteration: 22, named_losses: [('ActivationMax Loss', -3092.2688)], overall loss: -3092.26879883
Iteration: 23, named_losses: [('ActivationMax Loss', -3205.6792)], overall loss: -3205.67919922
Iteration: 24, named_losses: [('ActivationMax Loss', -3318.1287)], overall loss: -3318.12866211
Iteration: 25, named_losses: [('ActivationMax Loss', -3428.925)], overall loss: -3428.92504883
Iteration: 26, named_losses: [('ActivationMax Loss', -3536.9773)], overall loss: -3536.97729492
Iteration: 27, named_losses: [('ActivationMax Loss', -3644.4365)], overall loss: -3644.43652344
Iteration: 28, named_losses: [('ActivationMax Loss', -3751.6382)], overall loss: -3751.63818359
Iteration: 29, named_losses: [('ActivationMax Loss', -3857.9834)], overall loss: -3857.98339844
Iteration: 30, named_losses: [('ActivationMax Loss', -3964.2317)], overall loss: -3964.23168945
Iteration: 31, named_losses: [('ActivationMax Loss', -4069.6455)], overall loss: -4069.64550781
Iteration: 32, named_losses: [('ActivationMax Loss', -4173.1313)], overall loss: -4173.13134766
Iteration: 33, named_losses: [('ActivationMax Loss', -4277.3506)], overall loss: -4277.35058594
Iteration: 34, named_losses: [('ActivationMax Loss', -4380.5352)], overall loss: -4380.53515625
Iteration: 35, named_losses: [('ActivationMax Loss', -4484.7056)], overall loss: -4484.70556641
Iteration: 36, named_losses: [('ActivationMax Loss', -4587.394)], overall loss: -4587.39404297
Iteration: 37, named_losses: [('ActivationMax Loss', -4690.5303)], overall loss: -4690.53027344
Iteration: 38, named_losses: [('ActivationMax Loss', -4792.5132)], overall loss: -4792.51318359
Iteration: 39, named_losses: [('ActivationMax Loss', -4893.3652)], overall loss: -4893.36523438
Iteration: 40, named_losses: [('ActivationMax Loss', -4993.0762)], overall loss: -4993.07617188
Iteration: 41, named_losses: [('ActivationMax Loss', -5093.8433)], overall loss: -5093.84326172
Iteration: 42, named_losses: [('ActivationMax Loss', -5193.5752)], overall loss: -5193.57519531
Iteration: 43, named_losses: [('ActivationMax Loss', -5294.8237)], overall loss: -5294.82373047
Iteration: 44, named_losses: [('ActivationMax Loss', -5394.9277)], overall loss: -5394.92773438
Iteration: 45, named_losses: [('ActivationMax Loss', -5494.9424)], overall loss: -5494.94238281
Iteration: 46, named_losses: [('ActivationMax Loss', -5593.3843)], overall loss: -5593.38427734
Iteration: 47, named_losses: [('ActivationMax Loss', -5694.7734)], overall loss: -5694.7734375
Iteration: 48, named_losses: [('ActivationMax Loss', -5793.5518)], overall loss: -5793.55175781
Iteration: 49, named_losses: [('ActivationMax Loss', -5892.7148)], overall loss: -5892.71484375
Iteration: 50, named_losses: [('ActivationMax Loss', -5992.6699)], overall loss: -5992.66992188
Iteration: 51, named_losses: [('ActivationMax Loss', -6092.2173)], overall loss: -6092.21728516
Iteration: 52, named_losses: [('ActivationMax Loss', -6190.8921)], overall loss: -6190.89208984
Iteration: 53, named_losses: [('ActivationMax Loss', -6290.4185)], overall loss: -6290.41845703
Iteration: 54, named_losses: [('ActivationMax Loss', -6388.0405)], overall loss: -6388.04052734
Iteration: 55, named_losses: [('ActivationMax Loss', -6486.2891)], overall loss: -6486.2890625
Iteration: 56, named_losses: [('ActivationMax Loss', -6585.1128)], overall loss: -6585.11279297
Iteration: 57, named_losses: [('ActivationMax Loss', -6681.5127)], overall loss: -6681.51269531
Iteration: 58, named_losses: [('ActivationMax Loss', -6779.0796)], overall loss: -6779.07958984
Iteration: 59, named_losses: [('ActivationMax Loss', -6876.9258)], overall loss: -6876.92578125
Iteration: 60, named_losses: [('ActivationMax Loss', -6974.938)], overall loss: -6974.93798828
Iteration: 61, named_losses: [('ActivationMax Loss', -7070.7168)], overall loss: -7070.71679688
Iteration: 62, named_losses: [('ActivationMax Loss', -7166.8511)], overall loss: -7166.85107422
Iteration: 63, named_losses: [('ActivationMax Loss', -7264.6636)], overall loss: -7264.66357422
Iteration: 64, named_losses: [('ActivationMax Loss', -7360.7495)], overall loss: -7360.74951172
Iteration: 65, named_losses: [('ActivationMax Loss', -7456.3501)], overall loss: -7456.35009766
Iteration: 66, named_losses: [('ActivationMax Loss', -7551.9409)], overall loss: -7551.94091797
Iteration: 67, named_losses: [('ActivationMax Loss', -7648.4165)], overall loss: -7648.41650391
Iteration: 68, named_losses: [('ActivationMax Loss', -7744.001)], overall loss: -7744.00097656
Iteration: 69, named_losses: [('ActivationMax Loss', -7838.5376)], overall loss: -7838.53759766
Iteration: 70, named_losses: [('ActivationMax Loss', -7934.4219)], overall loss: -7934.421875
Iteration: 71, named_losses: [('ActivationMax Loss', -8030.0952)], overall loss: -8030.09521484
Iteration: 72, named_losses: [('ActivationMax Loss', -8125.4727)], overall loss: -8125.47265625
Iteration: 73, named_losses: [('ActivationMax Loss', -8220.7197)], overall loss: -8220.71972656
Iteration: 74, named_losses: [('ActivationMax Loss', -8316.6143)], overall loss: -8316.61425781
Iteration: 75, named_losses: [('ActivationMax Loss', -8412.3506)], overall loss: -8412.35058594
Iteration: 76, named_losses: [('ActivationMax Loss', -8507.9229)], overall loss: -8507.92285156
Iteration: 77, named_losses: [('ActivationMax Loss', -8604.7178)], overall loss: -8604.71777344
Iteration: 78, named_losses: [('ActivationMax Loss', -8702.0508)], overall loss: -8702.05078125
Iteration: 79, named_losses: [('ActivationMax Loss', -8797.6445)], overall loss: -8797.64453125
Iteration: 80, named_losses: [('ActivationMax Loss', -8894.2295)], overall loss: -8894.22949219
Iteration: 81, named_losses: [('ActivationMax Loss', -8990.502)], overall loss: -8990.50195312
Iteration: 82, named_losses: [('ActivationMax Loss', -9087.7773)], overall loss: -9087.77734375
Iteration: 83, named_losses: [('ActivationMax Loss', -9185.1807)], overall loss: -9185.18066406
Iteration: 84, named_losses: [('ActivationMax Loss', -9282.834)], overall loss: -9282.83398438
Iteration: 85, named_losses: [('ActivationMax Loss', -9380.6523)], overall loss: -9380.65234375
Iteration: 86, named_losses: [('ActivationMax Loss', -9478.0107)], overall loss: -9478.01074219
Iteration: 87, named_losses: [('ActivationMax Loss', -9575.5498)], overall loss: -9575.54980469
Iteration: 88, named_losses: [('ActivationMax Loss', -9673.2188)], overall loss: -9673.21875
Iteration: 89, named_losses: [('ActivationMax Loss', -9770.9697)], overall loss: -9770.96972656
Iteration: 90, named_losses: [('ActivationMax Loss', -9868.7793)], overall loss: -9868.77929688
Iteration: 91, named_losses: [('ActivationMax Loss', -9964.5371)], overall loss: -9964.53710938
Iteration: 92, named_losses: [('ActivationMax Loss', -10062.412)], overall loss: -10062.4121094
Iteration: 93, named_losses: [('ActivationMax Loss', -10158.876)], overall loss: -10158.8759766
Iteration: 94, named_losses: [('ActivationMax Loss', -10256.677)], overall loss: -10256.6767578
Iteration: 95, named_losses: [('ActivationMax Loss', -10352.661)], overall loss: -10352.6611328
Iteration: 96, named_losses: [('ActivationMax Loss', -10447.359)], overall loss: -10447.359375
Iteration: 97, named_losses: [('ActivationMax Loss', -10544.511)], overall loss: -10544.5107422
Iteration: 98, named_losses: [('ActivationMax Loss', -10640.197)], overall loss: -10640.1972656
Iteration: 99, named_losses: [('ActivationMax Loss', -10737.199)], overall loss: -10737.1992188
Iteration: 100, named_losses: [('ActivationMax Loss', -10832.438)], overall loss: -10832.4375
Iteration: 101, named_losses: [('ActivationMax Loss', -10929.422)], overall loss: -10929.421875
Iteration: 102, named_losses: [('ActivationMax Loss', -11025.653)], overall loss: -11025.6533203
Iteration: 103, named_losses: [('ActivationMax Loss', -11121.746)], overall loss: -11121.7460938
Iteration: 104, named_losses: [('ActivationMax Loss', -11216.964)], overall loss: -11216.9638672
Iteration: 105, named_losses: [('ActivationMax Loss', -11313.283)], overall loss: -11313.2832031
Iteration: 106, named_losses: [('ActivationMax Loss', -11410.26)], overall loss: -11410.2597656
Iteration: 107, named_losses: [('ActivationMax Loss', -11506.09)], overall loss: -11506.0898438
Iteration: 108, named_losses: [('ActivationMax Loss', -11600.643)], overall loss: -11600.6425781
Iteration: 109, named_losses: [('ActivationMax Loss', -11697.956)], overall loss: -11697.9560547
Iteration: 110, named_losses: [('ActivationMax Loss', -11793.214)], overall loss: -11793.2138672
Iteration: 111, named_losses: [('ActivationMax Loss', -11889.193)], overall loss: -11889.1933594
Iteration: 112, named_losses: [('ActivationMax Loss', -11984.958)], overall loss: -11984.9580078
Iteration: 113, named_losses: [('ActivationMax Loss', -12081.564)], overall loss: -12081.5644531
Iteration: 114, named_losses: [('ActivationMax Loss', -12177.07)], overall loss: -12177.0703125
Iteration: 115, named_losses: [('ActivationMax Loss', -12273.154)], overall loss: -12273.1542969
Iteration: 116, named_losses: [('ActivationMax Loss', -12370.32)], overall loss: -12370.3203125
Iteration: 117, named_losses: [('ActivationMax Loss', -12466.484)], overall loss: -12466.484375
Iteration: 118, named_losses: [('ActivationMax Loss', -12562.509)], overall loss: -12562.5087891
Iteration: 119, named_losses: [('ActivationMax Loss', -12658.77)], overall loss: -12658.7695312
Iteration: 120, named_losses: [('ActivationMax Loss', -12755.0)], overall loss: -12755.0
Iteration: 121, named_losses: [('ActivationMax Loss', -12852.065)], overall loss: -12852.0654297
Iteration: 122, named_losses: [('ActivationMax Loss', -12948.583)], overall loss: -12948.5830078
Iteration: 123, named_losses: [('ActivationMax Loss', -13043.738)], overall loss: -13043.7382812
Iteration: 124, named_losses: [('ActivationMax Loss', -13139.02)], overall loss: -13139.0195312
Iteration: 125, named_losses: [('ActivationMax Loss', -13234.346)], overall loss: -13234.3457031
Iteration: 126, named_losses: [('ActivationMax Loss', -13331.491)], overall loss: -13331.4912109
Iteration: 127, named_losses: [('ActivationMax Loss', -13427.523)], overall loss: -13427.5234375
Iteration: 128, named_losses: [('ActivationMax Loss', -13522.241)], overall loss: -13522.2412109
Iteration: 129, named_losses: [('ActivationMax Loss', -13617.408)], overall loss: -13617.4082031
Iteration: 130, named_losses: [('ActivationMax Loss', -13714.182)], overall loss: -13714.1816406
Iteration: 131, named_losses: [('ActivationMax Loss', -13809.398)], overall loss: -13809.3984375
Iteration: 132, named_losses: [('ActivationMax Loss', -13904.697)], overall loss: -13904.6972656
Iteration: 133, named_losses: [('ActivationMax Loss', -14002.63)], overall loss: -14002.6298828
Iteration: 134, named_losses: [('ActivationMax Loss', -14099.208)], overall loss: -14099.2080078
Iteration: 135, named_losses: [('ActivationMax Loss', -14195.607)], overall loss: -14195.6074219
Iteration: 136, named_losses: [('ActivationMax Loss', -14291.809)], overall loss: -14291.8085938
Iteration: 137, named_losses: [('ActivationMax Loss', -14389.49)], overall loss: -14389.4902344
Iteration: 138, named_losses: [('ActivationMax Loss', -14486.239)], overall loss: -14486.2392578
Iteration: 139, named_losses: [('ActivationMax Loss', -14584.095)], overall loss: -14584.0947266
Iteration: 140, named_losses: [('ActivationMax Loss', -14682.867)], overall loss: -14682.8671875
Iteration: 141, named_losses: [('ActivationMax Loss', -14781.787)], overall loss: -14781.7871094
Iteration: 142, named_losses: [('ActivationMax Loss', -14879.436)], overall loss: -14879.4355469
Iteration: 143, named_losses: [('ActivationMax Loss', -14978.084)], overall loss: -14978.0839844
Iteration: 144, named_losses: [('ActivationMax Loss', -15077.616)], overall loss: -15077.6162109
Iteration: 145, named_losses: [('ActivationMax Loss', -15175.915)], overall loss: -15175.9150391
Iteration: 146, named_losses: [('ActivationMax Loss', -15274.069)], overall loss: -15274.0693359
Iteration: 147, named_losses: [('ActivationMax Loss', -15373.179)], overall loss: -15373.1787109
Iteration: 148, named_losses: [('ActivationMax Loss', -15471.81)], overall loss: -15471.8095703
Iteration: 149, named_losses: [('ActivationMax Loss', -15570.835)], overall loss: -15570.8349609
Iteration: 150, named_losses: [('ActivationMax Loss', -15669.877)], overall loss: -15669.8769531
Iteration: 151, named_losses: [('ActivationMax Loss', -15768.857)], overall loss: -15768.8574219
Iteration: 152, named_losses: [('ActivationMax Loss', -15867.168)], overall loss: -15867.1679688
Iteration: 153, named_losses: [('ActivationMax Loss', -15965.778)], overall loss: -15965.7783203
Iteration: 154, named_losses: [('ActivationMax Loss', -16064.601)], overall loss: -16064.6005859
Iteration: 155, named_losses: [('ActivationMax Loss', -16162.262)], overall loss: -16162.2617188
Iteration: 156, named_losses: [('ActivationMax Loss', -16262.289)], overall loss: -16262.2890625
Iteration: 157, named_losses: [('ActivationMax Loss', -16359.585)], overall loss: -16359.5849609
Iteration: 158, named_losses: [('ActivationMax Loss', -16459.596)], overall loss: -16459.5957031
Iteration: 159, named_losses: [('ActivationMax Loss', -16559.24)], overall loss: -16559.2402344
Iteration: 160, named_losses: [('ActivationMax Loss', -16659.156)], overall loss: -16659.15625
Iteration: 161, named_losses: [('ActivationMax Loss', -16759.33)], overall loss: -16759.3300781
Iteration: 162, named_losses: [('ActivationMax Loss', -16861.242)], overall loss: -16861.2421875
Iteration: 163, named_losses: [('ActivationMax Loss', -16959.898)], overall loss: -16959.8984375
Iteration: 164, named_losses: [('ActivationMax Loss', -17058.123)], overall loss: -17058.1230469
Iteration: 165, named_losses: [('ActivationMax Loss', -17158.26)], overall loss: -17158.2597656
Iteration: 166, named_losses: [('ActivationMax Loss', -17258.967)], overall loss: -17258.9667969
Iteration: 167, named_losses: [('ActivationMax Loss', -17358.326)], overall loss: -17358.3261719
Iteration: 168, named_losses: [('ActivationMax Loss', -17458.109)], overall loss: -17458.109375
Iteration: 169, named_losses: [('ActivationMax Loss', -17557.682)], overall loss: -17557.6816406
Iteration: 170, named_losses: [('ActivationMax Loss', -17655.744)], overall loss: -17655.7441406
Iteration: 171, named_losses: [('ActivationMax Loss', -17755.303)], overall loss: -17755.3027344
Iteration: 172, named_losses: [('ActivationMax Loss', -17855.494)], overall loss: -17855.4941406
Iteration: 173, named_losses: [('ActivationMax Loss', -17955.205)], overall loss: -17955.2050781
Iteration: 174, named_losses: [('ActivationMax Loss', -18053.369)], overall loss: -18053.3691406
Iteration: 175, named_losses: [('ActivationMax Loss', -18151.721)], overall loss: -18151.7207031
Iteration: 176, named_losses: [('ActivationMax Loss', -18251.09)], overall loss: -18251.0898438
Iteration: 177, named_losses: [('ActivationMax Loss', -18349.832)], overall loss: -18349.8320312
Iteration: 178, named_losses: [('ActivationMax Loss', -18448.842)], overall loss: -18448.8417969
Iteration: 179, named_losses: [('ActivationMax Loss', -18547.826)], overall loss: -18547.8261719
Iteration: 180, named_losses: [('ActivationMax Loss', -18646.492)], overall loss: -18646.4921875
Iteration: 181, named_losses: [('ActivationMax Loss', -18745.336)], overall loss: -18745.3359375
Iteration: 182, named_losses: [('ActivationMax Loss', -18844.691)], overall loss: -18844.6914062
Iteration: 183, named_losses: [('ActivationMax Loss', -18944.184)], overall loss: -18944.1835938
Iteration: 184, named_losses: [('ActivationMax Loss', -19042.6)], overall loss: -19042.5996094
Iteration: 185, named_losses: [('ActivationMax Loss', -19142.037)], overall loss: -19142.0371094
Iteration: 186, named_losses: [('ActivationMax Loss', -19241.221)], overall loss: -19241.2207031
Iteration: 187, named_losses: [('ActivationMax Loss', -19340.268)], overall loss: -19340.2675781
Iteration: 188, named_losses: [('ActivationMax Loss', -19439.75)], overall loss: -19439.75
Iteration: 189, named_losses: [('ActivationMax Loss', -19539.535)], overall loss: -19539.5351562
Iteration: 190, named_losses: [('ActivationMax Loss', -19639.137)], overall loss: -19639.1367188
Iteration: 191, named_losses: [('ActivationMax Loss', -19738.834)], overall loss: -19738.8339844
Iteration: 192, named_losses: [('ActivationMax Loss', -19838.582)], overall loss: -19838.5820312
Iteration: 193, named_losses: [('ActivationMax Loss', -19938.512)], overall loss: -19938.5117188
Iteration: 194, named_losses: [('ActivationMax Loss', -20038.475)], overall loss: -20038.4746094
Iteration: 195, named_losses: [('ActivationMax Loss', -20139.158)], overall loss: -20139.1582031
Iteration: 196, named_losses: [('ActivationMax Loss', -20238.393)], overall loss: -20238.3925781
Iteration: 197, named_losses: [('ActivationMax Loss', -20339.186)], overall loss: -20339.1855469
Iteration: 198, named_losses: [('ActivationMax Loss', -20438.551)], overall loss: -20438.5507812
Iteration: 199, named_losses: [('ActivationMax Loss', -20537.754)], overall loss: -20537.7539062
Iteration: 200, named_losses: [('ActivationMax Loss', -20637.371)], overall loss: -20637.3710938






    Out[24]:





<matplotlib.image.AxesImage at 0x7fd933115990>

It does indeed go to much lower values, but the image looks less natural. Let's try varous range of total variation weights to enforce naturalness.



In [25]:

    
for tv_weight in [1e-3, 1e-2, 1e-1, 1, 10]:
    # Lets turn off verbose output this time to avoid clutter and just see the output.
    img = visualize_activation(model, layer_idx, filter_indices=filter_idx, input_range=(0., 1.), 
                               tv_weight=tv_weight, lp_norm_weight=0.)
    plt.figure()
    plt.imshow(img[..., 0])

We can see how total variation loss is enforcing blobbiness. In this case the default value of tv_weight=10 seems to work very well. The point of this exercise was to show how weights can be tuned.

Lets visualize all other output categories and see what we get.



In [26]:

    
for output_idx in np.arange(10):
    # Lets turn off verbose output this time to avoid clutter and just see the output.
    img = visualize_activation(model, layer_idx, filter_indices=output_idx, input_range=(0., 1.))
    plt.figure()
    plt.title('Networks perception of {}'.format(output_idx))
    plt.imshow(img[..., 0])

Pretty cool. Its amazing that we can even generate an input image via backprop!

Obviously you can tune the visualizations to look better by experimenting with image_modifiers, lp-norm weight etc. Basically, a regularizer is needed to enforce image naturalness prior to limit the input image search space. By this point, GANs should come to your mind. We could easily take a GAN trained on mnist and use discriminator loss as a regularizer. For using custom loss, you can use visualize_activation_with_losses API.

Feel free to submit a PR if you try the GAN regularizer :)

Other fun stuff

The API to visualize_activation accepts filter_indices. This is generally meant for multi label classifiers, but nothing prevents us from having some fun.

By setting filter_indices=[1, 7], we can generate an input that the network thinks is both 1 and 7 simultaneously. Its like asking the network

Generate input image that you think is both 1 and a 7.



In [28]:

    
img = visualize_activation(model, layer_idx, filter_indices=[1, 7], input_range=(0., 1.))
plt.imshow(img[..., 0])









    Out[28]:





<matplotlib.image.AxesImage at 0x7fd92febccd0>

Compare this to the 1 generated above and you should be able to see the difference. Nifty indded!

Visualizations without swapping softmax

As alluded at the beginning of the tutorial, we want to compare and see what happens if we didnt swap out softmax for linear activation.



In [19]:

    
# Swap linear back with softmax
model.layers[layer_idx].activation = activations.softmax
model = utils.apply_modifications(model)

for output_idx in np.arange(10):
    # Lets turn off verbose output this time to avoid clutter and just see the output.
    img = visualize_activation(model, layer_idx, filter_indices=output_idx, input_range=(0., 1.))
    plt.figure()
    plt.title('Networks perception of {}'.format(output_idx))
    plt.imshow(img[..., 0])

It does not work! The reason is that maximizing an output node can be done by minimizing other outputs. Softmax is weird that way. It is the only activation that depends on other node output(s) in the layer.