To visualize activation over final dense layer outputs, we need to switch the softmax activation out for linear since gradient of output node will depend on all the other node activations. Doing this in keras is tricky, so we provide utils.apply_modifications to modify network parameters and rebuild the graph.
If this swapping is not done, the results might be suboptimal. We will start by swapping out 'softmax' for 'linear' and compare what happens if we dont do this at the end.
In [8]:
from keras.applications import VGG16
from vis.utils import utils
from keras import activations
# Build the VGG16 network with ImageNet weights
model = VGG16(weights='imagenet', include_top=True)
# Utility to search for layer index by name.
# Alternatively we can specify this as -1 since it corresponds to the last layer.
layer_idx = utils.find_layer_idx(model, 'predictions')
# Swap softmax with linear
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)
Lets try visualizing a specific output category. We will pick ouzel which corresponds to imagenet category 20
In [9]:
from vis.visualization import visualize_activation
from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (18, 6)
# 20 is the imagenet category for 'ouzel'
img = visualize_activation(model, layer_idx, filter_indices=20)
plt.imshow(img)
Out[9]:
<matplotlib.image.AxesImage at 0x7f54187c57d0>
Hmm, that sort of looks like a bird. Lets see if we can get better results with more iterations. This time, lets see the verbose output during the optimization process.
In [10]:
# 20 is the imagenet category for 'ouzel'
img = visualize_activation(model, layer_idx, filter_indices=20, max_iter=500, verbose=True)
plt.imshow(img)
Iteration: 1, named_losses: [('ActivationMax Loss', -0.23652837),
('L-6.0 Norm Loss', 0.063257359),
('TV(2.0) Loss', 6454.2188)], overall loss: 6454.04541016
Iteration: 2, named_losses: [('ActivationMax Loss', -0.38862073),
('L-6.0 Norm Loss', 0.062633291),
('TV(2.0) Loss', 3182.8979)], overall loss: 3182.57202148
Iteration: 3, named_losses: [('ActivationMax Loss', -0.80792999),
('L-6.0 Norm Loss', 0.062285963),
('TV(2.0) Loss', 1660.4106)], overall loss: 1659.66503906
Iteration: 4, named_losses: [('ActivationMax Loss', -1.3529515),
('L-6.0 Norm Loss', 0.062074915),
('TV(2.0) Loss', 848.22705)], overall loss: 846.936157227
Iteration: 5, named_losses: [('ActivationMax Loss', -1.9309249),
('L-6.0 Norm Loss', 0.061946813),
('TV(2.0) Loss', 414.09256)], overall loss: 412.223571777
Iteration: 6, named_losses: [('ActivationMax Loss', -2.6263728),
('L-6.0 Norm Loss', 0.0618737),
('TV(2.0) Loss', 215.12886)], overall loss: 212.564361572
Iteration: 7, named_losses: [('ActivationMax Loss', -2.5867615),
('L-6.0 Norm Loss', 0.061841115),
('TV(2.0) Loss', 155.9509)], overall loss: 153.425979614
Iteration: 8, named_losses: [('ActivationMax Loss', -3.5514884),
('L-6.0 Norm Loss', 0.061822094),
('TV(2.0) Loss', 109.82358)], overall loss: 106.333908081
Iteration: 9, named_losses: [('ActivationMax Loss', -4.945785),
('L-6.0 Norm Loss', 0.061810952),
('TV(2.0) Loss', 83.539429)], overall loss: 78.655456543
Iteration: 10, named_losses: [('ActivationMax Loss', -6.3652425),
('L-6.0 Norm Loss', 0.061804228),
('TV(2.0) Loss', 63.23769)], overall loss: 56.9342498779
Iteration: 11, named_losses: [('ActivationMax Loss', -6.484271),
('L-6.0 Norm Loss', 0.061801635),
('TV(2.0) Loss', 54.48938)], overall loss: 48.06690979
Iteration: 12, named_losses: [('ActivationMax Loss', -8.5590439),
('L-6.0 Norm Loss', 0.061799563),
('TV(2.0) Loss', 47.193943)], overall loss: 38.6967010498
Iteration: 13, named_losses: [('ActivationMax Loss', -9.3418932),
('L-6.0 Norm Loss', 0.061799757),
('TV(2.0) Loss', 42.500381)], overall loss: 33.220287323
Iteration: 14, named_losses: [('ActivationMax Loss', -11.457891),
('L-6.0 Norm Loss', 0.061800055),
('TV(2.0) Loss', 37.536629)], overall loss: 26.140537262
Iteration: 15, named_losses: [('ActivationMax Loss', -13.214234),
('L-6.0 Norm Loss', 0.06180035),
('TV(2.0) Loss', 41.195164)], overall loss: 28.0427284241
Iteration: 16, named_losses: [('ActivationMax Loss', -14.028315),
('L-6.0 Norm Loss', 0.061799593),
('TV(2.0) Loss', 35.364285)], overall loss: 21.397769928
Iteration: 17, named_losses: [('ActivationMax Loss', -15.863123),
('L-6.0 Norm Loss', 0.061801132),
('TV(2.0) Loss', 34.293465)], overall loss: 18.4921417236
Iteration: 18, named_losses: [('ActivationMax Loss', -18.475391),
('L-6.0 Norm Loss', 0.061800994),
('TV(2.0) Loss', 33.899643)], overall loss: 15.4860534668
Iteration: 19, named_losses: [('ActivationMax Loss', -18.495914),
('L-6.0 Norm Loss', 0.061802723),
('TV(2.0) Loss', 35.162006)], overall loss: 16.7278938293
Iteration: 20, named_losses: [('ActivationMax Loss', -18.313532),
('L-6.0 Norm Loss', 0.061802253),
('TV(2.0) Loss', 29.397903)], overall loss: 11.1461734772
Iteration: 21, named_losses: [('ActivationMax Loss', -19.300627),
('L-6.0 Norm Loss', 0.061804581),
('TV(2.0) Loss', 28.367479)], overall loss: 9.12865638733
Iteration: 22, named_losses: [('ActivationMax Loss', -20.045309),
('L-6.0 Norm Loss', 0.06180536),
('TV(2.0) Loss', 31.620438)], overall loss: 11.6369342804
Iteration: 23, named_losses: [('ActivationMax Loss', -20.119078),
('L-6.0 Norm Loss', 0.061806351),
('TV(2.0) Loss', 30.15749)], overall loss: 10.1002178192
Iteration: 24, named_losses: [('ActivationMax Loss', -22.055819),
('L-6.0 Norm Loss', 0.06180672),
('TV(2.0) Loss', 28.695919)], overall loss: 6.70190811157
Iteration: 25, named_losses: [('ActivationMax Loss', -21.252024),
('L-6.0 Norm Loss', 0.061808087),
('TV(2.0) Loss', 28.868387)], overall loss: 7.67817115784
Iteration: 26, named_losses: [('ActivationMax Loss', -24.149572),
('L-6.0 Norm Loss', 0.061808869),
('TV(2.0) Loss', 28.443205)], overall loss: 4.35544204712
Iteration: 27, named_losses: [('ActivationMax Loss', -21.984728),
('L-6.0 Norm Loss', 0.061811917),
('TV(2.0) Loss', 25.873085)], overall loss: 3.95016860962
Iteration: 28, named_losses: [('ActivationMax Loss', -25.596382),
('L-6.0 Norm Loss', 0.06181211),
('TV(2.0) Loss', 27.402933)], overall loss: 1.86836242676
Iteration: 29, named_losses: [('ActivationMax Loss', -25.598436),
('L-6.0 Norm Loss', 0.061814241),
('TV(2.0) Loss', 26.746065)], overall loss: 1.20944213867
Iteration: 30, named_losses: [('ActivationMax Loss', -26.960041),
('L-6.0 Norm Loss', 0.061815515),
('TV(2.0) Loss', 26.27165)], overall loss: -0.626575469971
Iteration: 31, named_losses: [('ActivationMax Loss', -27.99596),
('L-6.0 Norm Loss', 0.061817057),
('TV(2.0) Loss', 25.858105)], overall loss: -2.0760383606
Iteration: 32, named_losses: [('ActivationMax Loss', -30.061876),
('L-6.0 Norm Loss', 0.06181993),
('TV(2.0) Loss', 27.144829)], overall loss: -2.85522842407
Iteration: 33, named_losses: [('ActivationMax Loss', -30.428617),
('L-6.0 Norm Loss', 0.061821472),
('TV(2.0) Loss', 28.232914)], overall loss: -2.13388252258
Iteration: 34, named_losses: [('ActivationMax Loss', -31.3414),
('L-6.0 Norm Loss', 0.061822757),
('TV(2.0) Loss', 27.857368)], overall loss: -3.42220878601
Iteration: 35, named_losses: [('ActivationMax Loss', -32.909225),
('L-6.0 Norm Loss', 0.061823759),
('TV(2.0) Loss', 27.695848)], overall loss: -5.15155220032
Iteration: 36, named_losses: [('ActivationMax Loss', -34.16124),
('L-6.0 Norm Loss', 0.061824419),
('TV(2.0) Loss', 27.488815)], overall loss: -6.61059951782
Iteration: 37, named_losses: [('ActivationMax Loss', -33.892025),
('L-6.0 Norm Loss', 0.061826207),
('TV(2.0) Loss', 29.994162)], overall loss: -3.83603858948
Iteration: 38, named_losses: [('ActivationMax Loss', -37.093285),
('L-6.0 Norm Loss', 0.061826963),
('TV(2.0) Loss', 28.946991)], overall loss: -8.08446502686
Iteration: 39, named_losses: [('ActivationMax Loss', -35.601936),
('L-6.0 Norm Loss', 0.061829448),
('TV(2.0) Loss', 30.475609)], overall loss: -5.06449890137
Iteration: 40, named_losses: [('ActivationMax Loss', -36.167427),
('L-6.0 Norm Loss', 0.061830733),
('TV(2.0) Loss', 27.217939)], overall loss: -8.88765525818
Iteration: 41, named_losses: [('ActivationMax Loss', -36.437752),
('L-6.0 Norm Loss', 0.061833352),
('TV(2.0) Loss', 29.842951)], overall loss: -6.53296852112
Iteration: 42, named_losses: [('ActivationMax Loss', -38.717491),
('L-6.0 Norm Loss', 0.061834473),
('TV(2.0) Loss', 28.724369)], overall loss: -9.93128585815
Iteration: 43, named_losses: [('ActivationMax Loss', -40.111435),
('L-6.0 Norm Loss', 0.061836857),
('TV(2.0) Loss', 29.991962)], overall loss: -10.057636261
Iteration: 44, named_losses: [('ActivationMax Loss', -36.964535),
('L-6.0 Norm Loss', 0.061837479),
('TV(2.0) Loss', 28.889849)], overall loss: -8.01284980774
Iteration: 45, named_losses: [('ActivationMax Loss', -37.320229),
('L-6.0 Norm Loss', 0.061839785),
('TV(2.0) Loss', 29.115425)], overall loss: -8.14296340942
Iteration: 46, named_losses: [('ActivationMax Loss', -39.844109),
('L-6.0 Norm Loss', 0.061841976),
('TV(2.0) Loss', 27.454922)], overall loss: -12.3273429871
Iteration: 47, named_losses: [('ActivationMax Loss', -37.078773),
('L-6.0 Norm Loss', 0.06184388),
('TV(2.0) Loss', 28.987179)], overall loss: -8.02975082397
Iteration: 48, named_losses: [('ActivationMax Loss', -40.641052),
('L-6.0 Norm Loss', 0.061846986),
('TV(2.0) Loss', 28.054935)], overall loss: -12.524269104
Iteration: 49, named_losses: [('ActivationMax Loss', -37.442184),
('L-6.0 Norm Loss', 0.06184734),
('TV(2.0) Loss', 29.338017)], overall loss: -8.04232025146
Iteration: 50, named_losses: [('ActivationMax Loss', -42.176495),
('L-6.0 Norm Loss', 0.061851047),
('TV(2.0) Loss', 28.415512)], overall loss: -13.699131012
Iteration: 51, named_losses: [('ActivationMax Loss', -41.259262),
('L-6.0 Norm Loss', 0.061850555),
('TV(2.0) Loss', 29.640665)], overall loss: -11.5567455292
Iteration: 52, named_losses: [('ActivationMax Loss', -43.201393),
('L-6.0 Norm Loss', 0.061853886),
('TV(2.0) Loss', 29.517044)], overall loss: -13.6224937439
Iteration: 53, named_losses: [('ActivationMax Loss', -44.086586),
('L-6.0 Norm Loss', 0.061855465),
('TV(2.0) Loss', 29.958473)], overall loss: -14.0662574768
Iteration: 54, named_losses: [('ActivationMax Loss', -46.265251),
('L-6.0 Norm Loss', 0.061856702),
('TV(2.0) Loss', 30.644764)], overall loss: -15.558631897
Iteration: 55, named_losses: [('ActivationMax Loss', -46.109085),
('L-6.0 Norm Loss', 0.061856948),
('TV(2.0) Loss', 32.343224)], overall loss: -13.7040061951
Iteration: 56, named_losses: [('ActivationMax Loss', -46.154011),
('L-6.0 Norm Loss', 0.061859209),
('TV(2.0) Loss', 29.866631)], overall loss: -16.2255210876
Iteration: 57, named_losses: [('ActivationMax Loss', -48.1483),
('L-6.0 Norm Loss', 0.061861962),
('TV(2.0) Loss', 31.805061)], overall loss: -16.281375885
Iteration: 58, named_losses: [('ActivationMax Loss', -47.153053),
('L-6.0 Norm Loss', 0.061863355),
('TV(2.0) Loss', 32.083012)], overall loss: -15.0081787109
Iteration: 59, named_losses: [('ActivationMax Loss', -49.861752),
('L-6.0 Norm Loss', 0.061865266),
('TV(2.0) Loss', 31.922997)], overall loss: -17.8768882751
Iteration: 60, named_losses: [('ActivationMax Loss', -47.217636),
('L-6.0 Norm Loss', 0.061866514),
('TV(2.0) Loss', 32.404999)], overall loss: -14.7507705688
Iteration: 61, named_losses: [('ActivationMax Loss', -47.677673),
('L-6.0 Norm Loss', 0.061867926),
('TV(2.0) Loss', 31.500904)], overall loss: -16.1149024963
Iteration: 62, named_losses: [('ActivationMax Loss', -48.208061),
('L-6.0 Norm Loss', 0.061871447),
('TV(2.0) Loss', 30.149673)], overall loss: -17.9965171814
Iteration: 63, named_losses: [('ActivationMax Loss', -48.763683),
('L-6.0 Norm Loss', 0.061873682),
('TV(2.0) Loss', 30.57785)], overall loss: -18.1239585876
Iteration: 64, named_losses: [('ActivationMax Loss', -49.660126),
('L-6.0 Norm Loss', 0.061875287),
('TV(2.0) Loss', 31.21563)], overall loss: -18.3826217651
Iteration: 65, named_losses: [('ActivationMax Loss', -50.520203),
('L-6.0 Norm Loss', 0.061876103),
('TV(2.0) Loss', 32.312675)], overall loss: -18.145652771
Iteration: 66, named_losses: [('ActivationMax Loss', -52.544136),
('L-6.0 Norm Loss', 0.061878048),
('TV(2.0) Loss', 32.571915)], overall loss: -19.9103431702
Iteration: 67, named_losses: [('ActivationMax Loss', -53.523029),
('L-6.0 Norm Loss', 0.061879326),
('TV(2.0) Loss', 33.641754)], overall loss: -19.8193969727
Iteration: 68, named_losses: [('ActivationMax Loss', -52.716587),
('L-6.0 Norm Loss', 0.061881624),
('TV(2.0) Loss', 33.863369)], overall loss: -18.7913360596
Iteration: 69, named_losses: [('ActivationMax Loss', -55.23497),
('L-6.0 Norm Loss', 0.061883334),
('TV(2.0) Loss', 33.474625)], overall loss: -21.6984634399
Iteration: 70, named_losses: [('ActivationMax Loss', -52.227325),
('L-6.0 Norm Loss', 0.06188463),
('TV(2.0) Loss', 32.758717)], overall loss: -19.4067230225
Iteration: 71, named_losses: [('ActivationMax Loss', -55.021244),
('L-6.0 Norm Loss', 0.061888061),
('TV(2.0) Loss', 32.97226)], overall loss: -21.9870948792
Iteration: 72, named_losses: [('ActivationMax Loss', -51.993431),
('L-6.0 Norm Loss', 0.061890103),
('TV(2.0) Loss', 33.576019)], overall loss: -18.3555221558
Iteration: 73, named_losses: [('ActivationMax Loss', -51.739635),
('L-6.0 Norm Loss', 0.061892442),
('TV(2.0) Loss', 33.036316)], overall loss: -18.6414260864
Iteration: 74, named_losses: [('ActivationMax Loss', -53.485603),
('L-6.0 Norm Loss', 0.061893709),
('TV(2.0) Loss', 33.111656)], overall loss: -20.3120536804
Iteration: 75, named_losses: [('ActivationMax Loss', -54.898994),
('L-6.0 Norm Loss', 0.061895926),
('TV(2.0) Loss', 32.140568)], overall loss: -22.6965293884
Iteration: 76, named_losses: [('ActivationMax Loss', -54.934818),
('L-6.0 Norm Loss', 0.061899848),
('TV(2.0) Loss', 34.627632)], overall loss: -20.2452850342
Iteration: 77, named_losses: [('ActivationMax Loss', -56.31601),
('L-6.0 Norm Loss', 0.061902486),
('TV(2.0) Loss', 32.778198)], overall loss: -23.4759101868
Iteration: 78, named_losses: [('ActivationMax Loss', -53.962742),
('L-6.0 Norm Loss', 0.061902851),
('TV(2.0) Loss', 34.602619)], overall loss: -19.2982215881
Iteration: 79, named_losses: [('ActivationMax Loss', -57.711037),
('L-6.0 Norm Loss', 0.061906222),
('TV(2.0) Loss', 33.1007)], overall loss: -24.5484313965
Iteration: 80, named_losses: [('ActivationMax Loss', -57.343006),
('L-6.0 Norm Loss', 0.061907329),
('TV(2.0) Loss', 34.195686)], overall loss: -23.0854110718
Iteration: 81, named_losses: [('ActivationMax Loss', -57.987549),
('L-6.0 Norm Loss', 0.061911486),
('TV(2.0) Loss', 34.674801)], overall loss: -23.2508354187
Iteration: 82, named_losses: [('ActivationMax Loss', -58.243034),
('L-6.0 Norm Loss', 0.061911572),
('TV(2.0) Loss', 35.852852)], overall loss: -22.3282699585
Iteration: 83, named_losses: [('ActivationMax Loss', -58.164795),
('L-6.0 Norm Loss', 0.061914217),
('TV(2.0) Loss', 34.573292)], overall loss: -23.5295906067
Iteration: 84, named_losses: [('ActivationMax Loss', -56.125908),
('L-6.0 Norm Loss', 0.061915576),
('TV(2.0) Loss', 36.036762)], overall loss: -20.0272293091
Iteration: 85, named_losses: [('ActivationMax Loss', -58.724846),
('L-6.0 Norm Loss', 0.061918765),
('TV(2.0) Loss', 34.975479)], overall loss: -23.6874465942
Iteration: 86, named_losses: [('ActivationMax Loss', -59.214962),
('L-6.0 Norm Loss', 0.061919611),
('TV(2.0) Loss', 35.705399)], overall loss: -23.44764328
Iteration: 87, named_losses: [('ActivationMax Loss', -59.618362),
('L-6.0 Norm Loss', 0.061922338),
('TV(2.0) Loss', 35.50737)], overall loss: -24.0490684509
Iteration: 88, named_losses: [('ActivationMax Loss', -56.870541),
('L-6.0 Norm Loss', 0.061923005),
('TV(2.0) Loss', 36.305687)], overall loss: -20.5029296875
Iteration: 89, named_losses: [('ActivationMax Loss', -61.641098),
('L-6.0 Norm Loss', 0.061927117),
('TV(2.0) Loss', 35.943317)], overall loss: -25.6358528137
Iteration: 90, named_losses: [('ActivationMax Loss', -60.660625),
('L-6.0 Norm Loss', 0.061929706),
('TV(2.0) Loss', 37.644867)], overall loss: -22.9538269043
Iteration: 91, named_losses: [('ActivationMax Loss', -62.358536),
('L-6.0 Norm Loss', 0.061931904),
('TV(2.0) Loss', 37.847153)], overall loss: -24.4494514465
Iteration: 92, named_losses: [('ActivationMax Loss', -61.129471),
('L-6.0 Norm Loss', 0.061932709),
('TV(2.0) Loss', 37.4174)], overall loss: -23.650138855
Iteration: 93, named_losses: [('ActivationMax Loss', -65.039703),
('L-6.0 Norm Loss', 0.061934352),
('TV(2.0) Loss', 36.973907)], overall loss: -28.0038604736
Iteration: 94, named_losses: [('ActivationMax Loss', -59.182236),
('L-6.0 Norm Loss', 0.061936423),
('TV(2.0) Loss', 38.246948)], overall loss: -20.8733520508
Iteration: 95, named_losses: [('ActivationMax Loss', -64.539116),
('L-6.0 Norm Loss', 0.061939154),
('TV(2.0) Loss', 37.030712)], overall loss: -27.4464683533
Iteration: 96, named_losses: [('ActivationMax Loss', -63.779083),
('L-6.0 Norm Loss', 0.061940245),
('TV(2.0) Loss', 36.523453)], overall loss: -27.1936912537
Iteration: 97, named_losses: [('ActivationMax Loss', -62.718475),
('L-6.0 Norm Loss', 0.061943103),
('TV(2.0) Loss', 38.223194)], overall loss: -24.4333381653
Iteration: 98, named_losses: [('ActivationMax Loss', -64.328323),
('L-6.0 Norm Loss', 0.061946243),
('TV(2.0) Loss', 36.81934)], overall loss: -27.4470405579
Iteration: 99, named_losses: [('ActivationMax Loss', -61.688591),
('L-6.0 Norm Loss', 0.061947718),
('TV(2.0) Loss', 38.590553)], overall loss: -23.0360908508
Iteration: 100, named_losses: [('ActivationMax Loss', -63.715092),
('L-6.0 Norm Loss', 0.061950605),
('TV(2.0) Loss', 37.157112)], overall loss: -26.4960289001
Iteration: 101, named_losses: [('ActivationMax Loss', -63.079693),
('L-6.0 Norm Loss', 0.061953332),
('TV(2.0) Loss', 38.64534)], overall loss: -24.3723983765
Iteration: 102, named_losses: [('ActivationMax Loss', -67.596748),
('L-6.0 Norm Loss', 0.061954927),
('TV(2.0) Loss', 37.264496)], overall loss: -30.2702941895
Iteration: 103, named_losses: [('ActivationMax Loss', -64.665634),
('L-6.0 Norm Loss', 0.061956808),
('TV(2.0) Loss', 39.460297)], overall loss: -25.1433792114
Iteration: 104, named_losses: [('ActivationMax Loss', -68.975647),
('L-6.0 Norm Loss', 0.061960772),
('TV(2.0) Loss', 38.939651)], overall loss: -29.9740371704
Iteration: 105, named_losses: [('ActivationMax Loss', -64.330437),
('L-6.0 Norm Loss', 0.061962936),
('TV(2.0) Loss', 39.573547)], overall loss: -24.6949234009
Iteration: 106, named_losses: [('ActivationMax Loss', -67.531845),
('L-6.0 Norm Loss', 0.061964907),
('TV(2.0) Loss', 38.740841)], overall loss: -28.7290382385
Iteration: 107, named_losses: [('ActivationMax Loss', -65.714714),
('L-6.0 Norm Loss', 0.061967168),
('TV(2.0) Loss', 39.831196)], overall loss: -25.8215522766
Iteration: 108, named_losses: [('ActivationMax Loss', -69.740456),
('L-6.0 Norm Loss', 0.061969183),
('TV(2.0) Loss', 39.020229)], overall loss: -30.6582603455
Iteration: 109, named_losses: [('ActivationMax Loss', -67.502937),
('L-6.0 Norm Loss', 0.061970394),
('TV(2.0) Loss', 40.945988)], overall loss: -26.4949760437
Iteration: 110, named_losses: [('ActivationMax Loss', -66.85611),
('L-6.0 Norm Loss', 0.061974101),
('TV(2.0) Loss', 39.716278)], overall loss: -27.0778579712
Iteration: 111, named_losses: [('ActivationMax Loss', -68.598862),
('L-6.0 Norm Loss', 0.061975479),
('TV(2.0) Loss', 41.146049)], overall loss: -27.390838623
Iteration: 112, named_losses: [('ActivationMax Loss', -70.261124),
('L-6.0 Norm Loss', 0.061977986),
('TV(2.0) Loss', 39.597488)], overall loss: -30.6016540527
Iteration: 113, named_losses: [('ActivationMax Loss', -66.714874),
('L-6.0 Norm Loss', 0.061979216),
('TV(2.0) Loss', 41.231609)], overall loss: -25.4212837219
Iteration: 114, named_losses: [('ActivationMax Loss', -70.259338),
('L-6.0 Norm Loss', 0.061982453),
('TV(2.0) Loss', 40.249741)], overall loss: -29.9476165771
Iteration: 115, named_losses: [('ActivationMax Loss', -69.399033),
('L-6.0 Norm Loss', 0.061984587),
('TV(2.0) Loss', 40.335354)], overall loss: -29.0016975403
Iteration: 116, named_losses: [('ActivationMax Loss', -71.065346),
('L-6.0 Norm Loss', 0.061987635),
('TV(2.0) Loss', 41.799816)], overall loss: -29.203540802
Iteration: 117, named_losses: [('ActivationMax Loss', -72.835297),
('L-6.0 Norm Loss', 0.061989769),
('TV(2.0) Loss', 40.994858)], overall loss: -31.7784500122
Iteration: 118, named_losses: [('ActivationMax Loss', -71.935997),
('L-6.0 Norm Loss', 0.061991692),
('TV(2.0) Loss', 42.238888)], overall loss: -29.6351203918
Iteration: 119, named_losses: [('ActivationMax Loss', -69.850922),
('L-6.0 Norm Loss', 0.061993204),
('TV(2.0) Loss', 41.699505)], overall loss: -28.0894203186
Iteration: 120, named_losses: [('ActivationMax Loss', -75.908012),
('L-6.0 Norm Loss', 0.061996683),
('TV(2.0) Loss', 42.971413)], overall loss: -32.8746032715
Iteration: 121, named_losses: [('ActivationMax Loss', -72.654961),
('L-6.0 Norm Loss', 0.061998829),
('TV(2.0) Loss', 42.499371)], overall loss: -30.0935935974
Iteration: 122, named_losses: [('ActivationMax Loss', -75.435516),
('L-6.0 Norm Loss', 0.062002096),
('TV(2.0) Loss', 43.51696)], overall loss: -31.856552124
Iteration: 123, named_losses: [('ActivationMax Loss', -72.926193),
('L-6.0 Norm Loss', 0.062002286),
('TV(2.0) Loss', 43.371983)], overall loss: -29.4922065735
Iteration: 124, named_losses: [('ActivationMax Loss', -75.617188),
('L-6.0 Norm Loss', 0.062004235),
('TV(2.0) Loss', 42.084713)], overall loss: -33.4704704285
Iteration: 125, named_losses: [('ActivationMax Loss', -73.594917),
('L-6.0 Norm Loss', 0.062007267),
('TV(2.0) Loss', 41.617821)], overall loss: -31.9150924683
Iteration: 126, named_losses: [('ActivationMax Loss', -73.42215),
('L-6.0 Norm Loss', 0.062009893),
('TV(2.0) Loss', 42.667156)], overall loss: -30.69298172
Iteration: 127, named_losses: [('ActivationMax Loss', -75.425133),
('L-6.0 Norm Loss', 0.062012836),
('TV(2.0) Loss', 41.357712)], overall loss: -34.0054092407
Iteration: 128, named_losses: [('ActivationMax Loss', -74.973259),
('L-6.0 Norm Loss', 0.062015209),
('TV(2.0) Loss', 43.628525)], overall loss: -31.2827224731
Iteration: 129, named_losses: [('ActivationMax Loss', -74.783409),
('L-6.0 Norm Loss', 0.062016975),
('TV(2.0) Loss', 42.33416)], overall loss: -32.3872299194
Iteration: 130, named_losses: [('ActivationMax Loss', -74.920242),
('L-6.0 Norm Loss', 0.062019497),
('TV(2.0) Loss', 43.562069)], overall loss: -31.2961540222
Iteration: 131, named_losses: [('ActivationMax Loss', -74.740448),
('L-6.0 Norm Loss', 0.062022813),
('TV(2.0) Loss', 42.652668)], overall loss: -32.0257606506
Iteration: 132, named_losses: [('ActivationMax Loss', -75.096954),
('L-6.0 Norm Loss', 0.062026158),
('TV(2.0) Loss', 44.228703)], overall loss: -30.806224823
Iteration: 133, named_losses: [('ActivationMax Loss', -77.228172),
('L-6.0 Norm Loss', 0.062028356),
('TV(2.0) Loss', 42.727112)], overall loss: -34.4390335083
Iteration: 134, named_losses: [('ActivationMax Loss', -74.963516),
('L-6.0 Norm Loss', 0.0620302),
('TV(2.0) Loss', 44.56879)], overall loss: -30.332698822
Iteration: 135, named_losses: [('ActivationMax Loss', -79.792282),
('L-6.0 Norm Loss', 0.0620308),
('TV(2.0) Loss', 43.277798)], overall loss: -36.4524497986
Iteration: 136, named_losses: [('ActivationMax Loss', -79.992973),
('L-6.0 Norm Loss', 0.062034879),
('TV(2.0) Loss', 44.352303)], overall loss: -35.5786361694
Iteration: 137, named_losses: [('ActivationMax Loss', -79.821152),
('L-6.0 Norm Loss', 0.062036708),
('TV(2.0) Loss', 43.979073)], overall loss: -35.7800445557
Iteration: 138, named_losses: [('ActivationMax Loss', -78.841103),
('L-6.0 Norm Loss', 0.062039301),
('TV(2.0) Loss', 44.952309)], overall loss: -33.826751709
Iteration: 139, named_losses: [('ActivationMax Loss', -80.606247),
('L-6.0 Norm Loss', 0.06204002),
('TV(2.0) Loss', 44.716331)], overall loss: -35.82787323
Iteration: 140, named_losses: [('ActivationMax Loss', -76.75869),
('L-6.0 Norm Loss', 0.062042855),
('TV(2.0) Loss', 45.755634)], overall loss: -30.9410133362
Iteration: 141, named_losses: [('ActivationMax Loss', -83.319611),
('L-6.0 Norm Loss', 0.06204576),
('TV(2.0) Loss', 44.883099)], overall loss: -38.3744697571
Iteration: 142, named_losses: [('ActivationMax Loss', -79.541634),
('L-6.0 Norm Loss', 0.06204699),
('TV(2.0) Loss', 45.271133)], overall loss: -34.2084503174
Iteration: 143, named_losses: [('ActivationMax Loss', -80.731552),
('L-6.0 Norm Loss', 0.062049665),
('TV(2.0) Loss', 45.206139)], overall loss: -35.4633636475
Iteration: 144, named_losses: [('ActivationMax Loss', -79.980911),
('L-6.0 Norm Loss', 0.062052872),
('TV(2.0) Loss', 44.870697)], overall loss: -35.0481643677
Iteration: 145, named_losses: [('ActivationMax Loss', -81.328354),
('L-6.0 Norm Loss', 0.062056288),
('TV(2.0) Loss', 45.792099)], overall loss: -35.4741973877
Iteration: 146, named_losses: [('ActivationMax Loss', -81.544037),
('L-6.0 Norm Loss', 0.062058225),
('TV(2.0) Loss', 45.111961)], overall loss: -36.3700180054
Iteration: 147, named_losses: [('ActivationMax Loss', -81.074211),
('L-6.0 Norm Loss', 0.062061563),
('TV(2.0) Loss', 46.412197)], overall loss: -34.5999488831
Iteration: 148, named_losses: [('ActivationMax Loss', -80.086357),
('L-6.0 Norm Loss', 0.062064234),
('TV(2.0) Loss', 45.670845)], overall loss: -34.3534469604
Iteration: 149, named_losses: [('ActivationMax Loss', -82.843842),
('L-6.0 Norm Loss', 0.06206771),
('TV(2.0) Loss', 45.885696)], overall loss: -36.8960800171
Iteration: 150, named_losses: [('ActivationMax Loss', -78.639267),
('L-6.0 Norm Loss', 0.062069252),
('TV(2.0) Loss', 46.430859)], overall loss: -32.1463356018
Iteration: 151, named_losses: [('ActivationMax Loss', -80.162636),
('L-6.0 Norm Loss', 0.062073454),
('TV(2.0) Loss', 46.224102)], overall loss: -33.8764610291
Iteration: 152, named_losses: [('ActivationMax Loss', -83.380333),
('L-6.0 Norm Loss', 0.062075607),
('TV(2.0) Loss', 45.654049)], overall loss: -37.6642112732
Iteration: 153, named_losses: [('ActivationMax Loss', -79.667259),
('L-6.0 Norm Loss', 0.062078506),
('TV(2.0) Loss', 46.391674)], overall loss: -33.2135047913
Iteration: 154, named_losses: [('ActivationMax Loss', -85.381157),
('L-6.0 Norm Loss', 0.062079921),
('TV(2.0) Loss', 45.745384)], overall loss: -39.5736923218
Iteration: 155, named_losses: [('ActivationMax Loss', -83.609505),
('L-6.0 Norm Loss', 0.062081322),
('TV(2.0) Loss', 45.66774)], overall loss: -37.8796844482
Iteration: 156, named_losses: [('ActivationMax Loss', -82.389984),
('L-6.0 Norm Loss', 0.062085431),
('TV(2.0) Loss', 45.546261)], overall loss: -36.7816352844
Iteration: 157, named_losses: [('ActivationMax Loss', -85.263351),
('L-6.0 Norm Loss', 0.062087107),
('TV(2.0) Loss', 46.222122)], overall loss: -38.9791412354
Iteration: 158, named_losses: [('ActivationMax Loss', -84.586708),
('L-6.0 Norm Loss', 0.062091459),
('TV(2.0) Loss', 46.984322)], overall loss: -37.5402984619
Iteration: 159, named_losses: [('ActivationMax Loss', -84.094154),
('L-6.0 Norm Loss', 0.062092349),
('TV(2.0) Loss', 47.529602)], overall loss: -36.502456665
Iteration: 160, named_losses: [('ActivationMax Loss', -83.759598),
('L-6.0 Norm Loss', 0.062095713),
('TV(2.0) Loss', 47.066273)], overall loss: -36.6312294006
Iteration: 161, named_losses: [('ActivationMax Loss', -87.085777),
('L-6.0 Norm Loss', 0.062097646),
('TV(2.0) Loss', 47.431206)], overall loss: -39.5924758911
Iteration: 162, named_losses: [('ActivationMax Loss', -85.065201),
('L-6.0 Norm Loss', 0.062101152),
('TV(2.0) Loss', 47.21352)], overall loss: -37.7895774841
Iteration: 163, named_losses: [('ActivationMax Loss', -89.332809),
('L-6.0 Norm Loss', 0.062102411),
('TV(2.0) Loss', 47.742847)], overall loss: -41.5278587341
Iteration: 164, named_losses: [('ActivationMax Loss', -84.093758),
('L-6.0 Norm Loss', 0.062105823),
('TV(2.0) Loss', 48.421814)], overall loss: -35.6098403931
Iteration: 165, named_losses: [('ActivationMax Loss', -85.390877),
('L-6.0 Norm Loss', 0.062107254),
('TV(2.0) Loss', 48.512482)], overall loss: -36.8162841797
Iteration: 166, named_losses: [('ActivationMax Loss', -89.280769),
('L-6.0 Norm Loss', 0.062109586),
('TV(2.0) Loss', 47.733635)], overall loss: -41.4850234985
Iteration: 167, named_losses: [('ActivationMax Loss', -86.773773),
('L-6.0 Norm Loss', 0.062111802),
('TV(2.0) Loss', 48.405296)], overall loss: -38.3063659668
Iteration: 168, named_losses: [('ActivationMax Loss', -89.266304),
('L-6.0 Norm Loss', 0.062114853),
('TV(2.0) Loss', 47.790852)], overall loss: -41.4133338928
Iteration: 169, named_losses: [('ActivationMax Loss', -85.502991),
('L-6.0 Norm Loss', 0.062117644),
('TV(2.0) Loss', 49.425331)], overall loss: -36.0155410767
Iteration: 170, named_losses: [('ActivationMax Loss', -85.752541),
('L-6.0 Norm Loss', 0.062120721),
('TV(2.0) Loss', 48.492397)], overall loss: -37.1980247498
Iteration: 171, named_losses: [('ActivationMax Loss', -89.4711),
('L-6.0 Norm Loss', 0.06212277),
('TV(2.0) Loss', 48.197517)], overall loss: -41.2114562988
Iteration: 172, named_losses: [('ActivationMax Loss', -84.256325),
('L-6.0 Norm Loss', 0.062125843),
('TV(2.0) Loss', 48.419376)], overall loss: -35.7748222351
Iteration: 173, named_losses: [('ActivationMax Loss', -89.153397),
('L-6.0 Norm Loss', 0.062128186),
('TV(2.0) Loss', 48.649033)], overall loss: -40.442237854
Iteration: 174, named_losses: [('ActivationMax Loss', -89.42482),
('L-6.0 Norm Loss', 0.062131628),
('TV(2.0) Loss', 48.84708)], overall loss: -40.5156059265
Iteration: 175, named_losses: [('ActivationMax Loss', -89.780655),
('L-6.0 Norm Loss', 0.062133592),
('TV(2.0) Loss', 48.559395)], overall loss: -41.1591262817
Iteration: 176, named_losses: [('ActivationMax Loss', -87.505371),
('L-6.0 Norm Loss', 0.062137157),
('TV(2.0) Loss', 49.34486)], overall loss: -38.0983772278
Iteration: 177, named_losses: [('ActivationMax Loss', -90.80497),
('L-6.0 Norm Loss', 0.062139709),
('TV(2.0) Loss', 48.458553)], overall loss: -42.2842750549
Iteration: 178, named_losses: [('ActivationMax Loss', -90.880226),
('L-6.0 Norm Loss', 0.062140673),
('TV(2.0) Loss', 50.447983)], overall loss: -40.3701019287
Iteration: 179, named_losses: [('ActivationMax Loss', -88.743202),
('L-6.0 Norm Loss', 0.062143836),
('TV(2.0) Loss', 48.893677)], overall loss: -39.7873840332
Iteration: 180, named_losses: [('ActivationMax Loss', -92.292145),
('L-6.0 Norm Loss', 0.062145084),
('TV(2.0) Loss', 49.529144)], overall loss: -42.7008590698
Iteration: 181, named_losses: [('ActivationMax Loss', -90.334358),
('L-6.0 Norm Loss', 0.062147133),
('TV(2.0) Loss', 49.376839)], overall loss: -40.8953704834
Iteration: 182, named_losses: [('ActivationMax Loss', -93.505615),
('L-6.0 Norm Loss', 0.062149204),
('TV(2.0) Loss', 49.574623)], overall loss: -43.8688430786
Iteration: 183, named_losses: [('ActivationMax Loss', -89.368149),
('L-6.0 Norm Loss', 0.062151831),
('TV(2.0) Loss', 50.479847)], overall loss: -38.8261528015
Iteration: 184, named_losses: [('ActivationMax Loss', -91.670448),
('L-6.0 Norm Loss', 0.062155567),
('TV(2.0) Loss', 50.49128)], overall loss: -41.1170120239
Iteration: 185, named_losses: [('ActivationMax Loss', -91.845055),
('L-6.0 Norm Loss', 0.062158324),
('TV(2.0) Loss', 50.634842)], overall loss: -41.1480560303
Iteration: 186, named_losses: [('ActivationMax Loss', -93.885223),
('L-6.0 Norm Loss', 0.062160846),
('TV(2.0) Loss', 51.10173)], overall loss: -42.7213287354
Iteration: 187, named_losses: [('ActivationMax Loss', -87.125481),
('L-6.0 Norm Loss', 0.062164038),
('TV(2.0) Loss', 51.786007)], overall loss: -35.2773094177
Iteration: 188, named_losses: [('ActivationMax Loss', -94.432861),
('L-6.0 Norm Loss', 0.062165372),
('TV(2.0) Loss', 50.792919)], overall loss: -43.5777778625
Iteration: 189, named_losses: [('ActivationMax Loss', -92.998787),
('L-6.0 Norm Loss', 0.062168516),
('TV(2.0) Loss', 50.617378)], overall loss: -42.3192367554
Iteration: 190, named_losses: [('ActivationMax Loss', -94.037918),
('L-6.0 Norm Loss', 0.062169418),
('TV(2.0) Loss', 50.953403)], overall loss: -43.0223426819
Iteration: 191, named_losses: [('ActivationMax Loss', -92.43676),
('L-6.0 Norm Loss', 0.062173188),
('TV(2.0) Loss', 50.758865)], overall loss: -41.6157226562
Iteration: 192, named_losses: [('ActivationMax Loss', -94.432266),
('L-6.0 Norm Loss', 0.06217441),
('TV(2.0) Loss', 51.39883)], overall loss: -42.9712638855
Iteration: 193, named_losses: [('ActivationMax Loss', -94.284225),
('L-6.0 Norm Loss', 0.062178232),
('TV(2.0) Loss', 50.184383)], overall loss: -44.0376625061
Iteration: 194, named_losses: [('ActivationMax Loss', -95.05014),
('L-6.0 Norm Loss', 0.062180407),
('TV(2.0) Loss', 51.994118)], overall loss: -42.9938430786
Iteration: 195, named_losses: [('ActivationMax Loss', -92.610092),
('L-6.0 Norm Loss', 0.062183313),
('TV(2.0) Loss', 50.379807)], overall loss: -42.1681060791
Iteration: 196, named_losses: [('ActivationMax Loss', -93.360321),
('L-6.0 Norm Loss', 0.062185787),
('TV(2.0) Loss', 52.400833)], overall loss: -40.8973007202
Iteration: 197, named_losses: [('ActivationMax Loss', -95.436844),
('L-6.0 Norm Loss', 0.062187679),
('TV(2.0) Loss', 50.404907)], overall loss: -44.9697494507
Iteration: 198, named_losses: [('ActivationMax Loss', -95.442024),
('L-6.0 Norm Loss', 0.062191367),
('TV(2.0) Loss', 51.862446)], overall loss: -43.5173835754
Iteration: 199, named_losses: [('ActivationMax Loss', -94.860168),
('L-6.0 Norm Loss', 0.06219282),
('TV(2.0) Loss', 50.87299)], overall loss: -43.9249839783
Iteration: 200, named_losses: [('ActivationMax Loss', -95.843559),
('L-6.0 Norm Loss', 0.06219532),
('TV(2.0) Loss', 52.328056)], overall loss: -43.4533081055
Iteration: 201, named_losses: [('ActivationMax Loss', -97.020523),
('L-6.0 Norm Loss', 0.062198706),
('TV(2.0) Loss', 51.485229)], overall loss: -45.4730911255
Iteration: 202, named_losses: [('ActivationMax Loss', -98.151337),
('L-6.0 Norm Loss', 0.062202491),
('TV(2.0) Loss', 53.322594)], overall loss: -44.7665405273
Iteration: 203, named_losses: [('ActivationMax Loss', -95.426918),
('L-6.0 Norm Loss', 0.062204488),
('TV(2.0) Loss', 53.215027)], overall loss: -42.1496887207
Iteration: 204, named_losses: [('ActivationMax Loss', -98.068665),
('L-6.0 Norm Loss', 0.062205907),
('TV(2.0) Loss', 54.215515)], overall loss: -43.7909469604
Iteration: 205, named_losses: [('ActivationMax Loss', -95.863991),
('L-6.0 Norm Loss', 0.062209398),
('TV(2.0) Loss', 53.636875)], overall loss: -42.1649055481
Iteration: 206, named_losses: [('ActivationMax Loss', -98.729401),
('L-6.0 Norm Loss', 0.062210523),
('TV(2.0) Loss', 54.282337)], overall loss: -44.384853363
Iteration: 207, named_losses: [('ActivationMax Loss', -96.110382),
('L-6.0 Norm Loss', 0.06221268),
('TV(2.0) Loss', 53.255119)], overall loss: -42.7930526733
Iteration: 208, named_losses: [('ActivationMax Loss', -98.217163),
('L-6.0 Norm Loss', 0.062215261),
('TV(2.0) Loss', 53.936409)], overall loss: -44.218536377
Iteration: 209, named_losses: [('ActivationMax Loss', -101.32747),
('L-6.0 Norm Loss', 0.062217344),
('TV(2.0) Loss', 52.384407)], overall loss: -48.8808441162
Iteration: 210, named_losses: [('ActivationMax Loss', -102.2364),
('L-6.0 Norm Loss', 0.062220935),
('TV(2.0) Loss', 54.511322)], overall loss: -47.6628646851
Iteration: 211, named_losses: [('ActivationMax Loss', -99.871643),
('L-6.0 Norm Loss', 0.062223282),
('TV(2.0) Loss', 53.764351)], overall loss: -46.0450668335
Iteration: 212, named_losses: [('ActivationMax Loss', -99.592453),
('L-6.0 Norm Loss', 0.062226798),
('TV(2.0) Loss', 54.300797)], overall loss: -45.2294311523
Iteration: 213, named_losses: [('ActivationMax Loss', -101.54116),
('L-6.0 Norm Loss', 0.062228646),
('TV(2.0) Loss', 54.218273)], overall loss: -47.2606620789
Iteration: 214, named_losses: [('ActivationMax Loss', -99.221146),
('L-6.0 Norm Loss', 0.062231064),
('TV(2.0) Loss', 53.949696)], overall loss: -45.2092170715
Iteration: 215, named_losses: [('ActivationMax Loss', -101.65981),
('L-6.0 Norm Loss', 0.062233083),
('TV(2.0) Loss', 53.531853)], overall loss: -48.0657196045
Iteration: 216, named_losses: [('ActivationMax Loss', -100.08763),
('L-6.0 Norm Loss', 0.06223705),
('TV(2.0) Loss', 54.031837)], overall loss: -45.9935531616
Iteration: 217, named_losses: [('ActivationMax Loss', -100.44298),
('L-6.0 Norm Loss', 0.06223971),
('TV(2.0) Loss', 53.505089)], overall loss: -46.8756484985
Iteration: 218, named_losses: [('ActivationMax Loss', -99.539574),
('L-6.0 Norm Loss', 0.06224139),
('TV(2.0) Loss', 54.246307)], overall loss: -45.2310256958
Iteration: 219, named_losses: [('ActivationMax Loss', -100.79933),
('L-6.0 Norm Loss', 0.062244087),
('TV(2.0) Loss', 53.78986)], overall loss: -46.9472312927
Iteration: 220, named_losses: [('ActivationMax Loss', -102.77279),
('L-6.0 Norm Loss', 0.062245931),
('TV(2.0) Loss', 53.56958)], overall loss: -49.1409606934
Iteration: 221, named_losses: [('ActivationMax Loss', -99.615746),
('L-6.0 Norm Loss', 0.062250301),
('TV(2.0) Loss', 54.42424)], overall loss: -45.1292572021
Iteration: 222, named_losses: [('ActivationMax Loss', -103.13307),
('L-6.0 Norm Loss', 0.062251575),
('TV(2.0) Loss', 54.386383)], overall loss: -48.6844406128
Iteration: 223, named_losses: [('ActivationMax Loss', -96.229507),
('L-6.0 Norm Loss', 0.062256515),
('TV(2.0) Loss', 56.048912)], overall loss: -40.1183395386
Iteration: 224, named_losses: [('ActivationMax Loss', -104.89292),
('L-6.0 Norm Loss', 0.062258039),
('TV(2.0) Loss', 55.30854)], overall loss: -49.5221252441
Iteration: 225, named_losses: [('ActivationMax Loss', -97.592621),
('L-6.0 Norm Loss', 0.062260509),
('TV(2.0) Loss', 55.895943)], overall loss: -41.6344146729
Iteration: 226, named_losses: [('ActivationMax Loss', -107.39828),
('L-6.0 Norm Loss', 0.062263034),
('TV(2.0) Loss', 55.099182)], overall loss: -52.2368392944
Iteration: 227, named_losses: [('ActivationMax Loss', -102.12806),
('L-6.0 Norm Loss', 0.062265098),
('TV(2.0) Loss', 56.022156)], overall loss: -46.0436401367
Iteration: 228, named_losses: [('ActivationMax Loss', -101.97128),
('L-6.0 Norm Loss', 0.062269926),
('TV(2.0) Loss', 55.481373)], overall loss: -46.4276313782
Iteration: 229, named_losses: [('ActivationMax Loss', -99.648331),
('L-6.0 Norm Loss', 0.06227085),
('TV(2.0) Loss', 56.001991)], overall loss: -43.5840682983
Iteration: 230, named_losses: [('ActivationMax Loss', -105.40375),
('L-6.0 Norm Loss', 0.062272847),
('TV(2.0) Loss', 55.378609)], overall loss: -49.9628677368
Iteration: 231, named_losses: [('ActivationMax Loss', -99.835266),
('L-6.0 Norm Loss', 0.062275451),
('TV(2.0) Loss', 55.077816)], overall loss: -44.6951713562
Iteration: 232, named_losses: [('ActivationMax Loss', -102.91353),
('L-6.0 Norm Loss', 0.062278029),
('TV(2.0) Loss', 56.010662)], overall loss: -46.840587616
Iteration: 233, named_losses: [('ActivationMax Loss', -102.78975),
('L-6.0 Norm Loss', 0.062280152),
('TV(2.0) Loss', 55.402893)], overall loss: -47.3245773315
Iteration: 234, named_losses: [('ActivationMax Loss', -104.81158),
('L-6.0 Norm Loss', 0.062282369),
('TV(2.0) Loss', 57.238579)], overall loss: -47.5107269287
Iteration: 235, named_losses: [('ActivationMax Loss', -105.24297),
('L-6.0 Norm Loss', 0.062285244),
('TV(2.0) Loss', 55.808445)], overall loss: -49.3722343445
Iteration: 236, named_losses: [('ActivationMax Loss', -104.96114),
('L-6.0 Norm Loss', 0.062287703),
('TV(2.0) Loss', 57.81691)], overall loss: -47.0819396973
Iteration: 237, named_losses: [('ActivationMax Loss', -102.74883),
('L-6.0 Norm Loss', 0.06228995),
('TV(2.0) Loss', 56.162151)], overall loss: -46.5243873596
Iteration: 238, named_losses: [('ActivationMax Loss', -102.88896),
('L-6.0 Norm Loss', 0.062293455),
('TV(2.0) Loss', 58.491325)], overall loss: -44.3353424072
Iteration: 239, named_losses: [('ActivationMax Loss', -104.80802),
('L-6.0 Norm Loss', 0.062297061),
('TV(2.0) Loss', 56.09906)], overall loss: -48.6466674805
Iteration: 240, named_losses: [('ActivationMax Loss', -107.4565),
('L-6.0 Norm Loss', 0.062297102),
('TV(2.0) Loss', 57.593307)], overall loss: -49.8009033203
Iteration: 241, named_losses: [('ActivationMax Loss', -103.24889),
('L-6.0 Norm Loss', 0.062300507),
('TV(2.0) Loss', 56.637703)], overall loss: -46.5488815308
Iteration: 242, named_losses: [('ActivationMax Loss', -104.22279),
('L-6.0 Norm Loss', 0.062303975),
('TV(2.0) Loss', 58.854889)], overall loss: -45.3056030273
Iteration: 243, named_losses: [('ActivationMax Loss', -106.06935),
('L-6.0 Norm Loss', 0.062306654),
('TV(2.0) Loss', 57.842873)], overall loss: -48.1641693115
Iteration: 244, named_losses: [('ActivationMax Loss', -107.12758),
('L-6.0 Norm Loss', 0.062309526),
('TV(2.0) Loss', 57.721554)], overall loss: -49.3437156677
Iteration: 245, named_losses: [('ActivationMax Loss', -110.45367),
('L-6.0 Norm Loss', 0.06231118),
('TV(2.0) Loss', 56.592937)], overall loss: -53.7984199524
Iteration: 246, named_losses: [('ActivationMax Loss', -105.25899),
('L-6.0 Norm Loss', 0.062314849),
('TV(2.0) Loss', 59.685715)], overall loss: -45.5109558105
Iteration: 247, named_losses: [('ActivationMax Loss', -106.68048),
('L-6.0 Norm Loss', 0.06231707),
('TV(2.0) Loss', 56.97971)], overall loss: -49.6384544373
Iteration: 248, named_losses: [('ActivationMax Loss', -107.46439),
('L-6.0 Norm Loss', 0.062318288),
('TV(2.0) Loss', 58.886314)], overall loss: -48.5157546997
Iteration: 249, named_losses: [('ActivationMax Loss', -107.52404),
('L-6.0 Norm Loss', 0.062321611),
('TV(2.0) Loss', 57.354012)], overall loss: -50.1077041626
Iteration: 250, named_losses: [('ActivationMax Loss', -105.10503),
('L-6.0 Norm Loss', 0.062323544),
('TV(2.0) Loss', 58.235168)], overall loss: -46.8075332642
Iteration: 251, named_losses: [('ActivationMax Loss', -110.02692),
('L-6.0 Norm Loss', 0.06232591),
('TV(2.0) Loss', 57.247654)], overall loss: -52.7169456482
Iteration: 252, named_losses: [('ActivationMax Loss', -109.05452),
('L-6.0 Norm Loss', 0.062328301),
('TV(2.0) Loss', 58.03125)], overall loss: -50.9609451294
Iteration: 253, named_losses: [('ActivationMax Loss', -110.25841),
('L-6.0 Norm Loss', 0.062332455),
('TV(2.0) Loss', 58.473946)], overall loss: -51.7221298218
Iteration: 254, named_losses: [('ActivationMax Loss', -109.35121),
('L-6.0 Norm Loss', 0.062333677),
('TV(2.0) Loss', 59.535042)], overall loss: -49.7538375854
Iteration: 255, named_losses: [('ActivationMax Loss', -108.26953),
('L-6.0 Norm Loss', 0.062336184),
('TV(2.0) Loss', 58.487885)], overall loss: -49.7193069458
Iteration: 256, named_losses: [('ActivationMax Loss', -111.23071),
('L-6.0 Norm Loss', 0.062341016),
('TV(2.0) Loss', 58.887402)], overall loss: -52.2809638977
Iteration: 257, named_losses: [('ActivationMax Loss', -106.85623),
('L-6.0 Norm Loss', 0.062343534),
('TV(2.0) Loss', 59.184872)], overall loss: -47.6090202332
Iteration: 258, named_losses: [('ActivationMax Loss', -109.48892),
('L-6.0 Norm Loss', 0.062346313),
('TV(2.0) Loss', 58.449951)], overall loss: -50.9766235352
Iteration: 259, named_losses: [('ActivationMax Loss', -105.8782),
('L-6.0 Norm Loss', 0.062348217),
('TV(2.0) Loss', 58.297905)], overall loss: -47.5179519653
Iteration: 260, named_losses: [('ActivationMax Loss', -109.84384),
('L-6.0 Norm Loss', 0.062351409),
('TV(2.0) Loss', 57.896324)], overall loss: -51.8851623535
Iteration: 261, named_losses: [('ActivationMax Loss', -105.68251),
('L-6.0 Norm Loss', 0.062352248),
('TV(2.0) Loss', 59.137436)], overall loss: -46.4827194214
Iteration: 262, named_losses: [('ActivationMax Loss', -106.68652),
('L-6.0 Norm Loss', 0.062354982),
('TV(2.0) Loss', 58.494118)], overall loss: -48.1300506592
Iteration: 263, named_losses: [('ActivationMax Loss', -109.27451),
('L-6.0 Norm Loss', 0.062356852),
('TV(2.0) Loss', 58.842148)], overall loss: -50.3700027466
Iteration: 264, named_losses: [('ActivationMax Loss', -110.98386),
('L-6.0 Norm Loss', 0.062358826),
('TV(2.0) Loss', 57.431625)], overall loss: -53.4898834229
Iteration: 265, named_losses: [('ActivationMax Loss', -109.06016),
('L-6.0 Norm Loss', 0.062362604),
('TV(2.0) Loss', 59.318092)], overall loss: -49.6797027588
Iteration: 266, named_losses: [('ActivationMax Loss', -113.14455),
('L-6.0 Norm Loss', 0.062365636),
('TV(2.0) Loss', 58.969494)], overall loss: -54.1126976013
Iteration: 267, named_losses: [('ActivationMax Loss', -107.90953),
('L-6.0 Norm Loss', 0.062368978),
('TV(2.0) Loss', 59.832138)], overall loss: -48.0150222778
Iteration: 268, named_losses: [('ActivationMax Loss', -112.32379),
('L-6.0 Norm Loss', 0.062371835),
('TV(2.0) Loss', 58.449043)], overall loss: -53.8123779297
Iteration: 269, named_losses: [('ActivationMax Loss', -105.80854),
('L-6.0 Norm Loss', 0.062376611),
('TV(2.0) Loss', 60.501823)], overall loss: -45.2443389893
Iteration: 270, named_losses: [('ActivationMax Loss', -114.73972),
('L-6.0 Norm Loss', 0.062378291),
('TV(2.0) Loss', 57.394127)], overall loss: -57.2832107544
Iteration: 271, named_losses: [('ActivationMax Loss', -112.65745),
('L-6.0 Norm Loss', 0.062381741),
('TV(2.0) Loss', 60.287937)], overall loss: -52.3071327209
Iteration: 272, named_losses: [('ActivationMax Loss', -110.16896),
('L-6.0 Norm Loss', 0.062383883),
('TV(2.0) Loss', 58.064457)], overall loss: -52.0421180725
Iteration: 273, named_losses: [('ActivationMax Loss', -112.46786),
('L-6.0 Norm Loss', 0.062387586),
('TV(2.0) Loss', 61.057594)], overall loss: -51.3478851318
Iteration: 274, named_losses: [('ActivationMax Loss', -112.55803),
('L-6.0 Norm Loss', 0.062390991),
('TV(2.0) Loss', 59.246513)], overall loss: -53.2491226196
Iteration: 275, named_losses: [('ActivationMax Loss', -113.26693),
('L-6.0 Norm Loss', 0.062393028),
('TV(2.0) Loss', 60.512764)], overall loss: -52.6917724609
Iteration: 276, named_losses: [('ActivationMax Loss', -110.98032),
('L-6.0 Norm Loss', 0.062395461),
('TV(2.0) Loss', 59.328346)], overall loss: -51.5895843506
Iteration: 277, named_losses: [('ActivationMax Loss', -113.67271),
('L-6.0 Norm Loss', 0.062397864),
('TV(2.0) Loss', 60.281197)], overall loss: -53.3291168213
Iteration: 278, named_losses: [('ActivationMax Loss', -113.04047),
('L-6.0 Norm Loss', 0.062401105),
('TV(2.0) Loss', 59.962502)], overall loss: -53.0155715942
Iteration: 279, named_losses: [('ActivationMax Loss', -110.9659),
('L-6.0 Norm Loss', 0.062404241),
('TV(2.0) Loss', 61.521461)], overall loss: -49.3820343018
Iteration: 280, named_losses: [('ActivationMax Loss', -115.2794),
('L-6.0 Norm Loss', 0.062406976),
('TV(2.0) Loss', 59.865814)], overall loss: -55.3511810303
Iteration: 281, named_losses: [('ActivationMax Loss', -113.83675),
('L-6.0 Norm Loss', 0.062409297),
('TV(2.0) Loss', 61.184658)], overall loss: -52.5896873474
Iteration: 282, named_losses: [('ActivationMax Loss', -113.69312),
('L-6.0 Norm Loss', 0.062412113),
('TV(2.0) Loss', 61.390404)], overall loss: -52.2403030396
Iteration: 283, named_losses: [('ActivationMax Loss', -110.56762),
('L-6.0 Norm Loss', 0.062414687),
('TV(2.0) Loss', 61.344093)], overall loss: -49.1611099243
Iteration: 284, named_losses: [('ActivationMax Loss', -117.77048),
('L-6.0 Norm Loss', 0.062416639),
('TV(2.0) Loss', 60.605007)], overall loss: -57.1030540466
Iteration: 285, named_losses: [('ActivationMax Loss', -112.71196),
('L-6.0 Norm Loss', 0.062421471),
('TV(2.0) Loss', 60.330814)], overall loss: -52.3187217712
Iteration: 286, named_losses: [('ActivationMax Loss', -114.80564),
('L-6.0 Norm Loss', 0.062424406),
('TV(2.0) Loss', 61.049095)], overall loss: -53.6941223145
Iteration: 287, named_losses: [('ActivationMax Loss', -113.91931),
('L-6.0 Norm Loss', 0.062425874),
('TV(2.0) Loss', 60.590805)], overall loss: -53.2660827637
Iteration: 288, named_losses: [('ActivationMax Loss', -111.59972),
('L-6.0 Norm Loss', 0.062429011),
('TV(2.0) Loss', 61.732868)], overall loss: -49.8044166565
Iteration: 289, named_losses: [('ActivationMax Loss', -111.73795),
('L-6.0 Norm Loss', 0.062431265),
('TV(2.0) Loss', 60.907837)], overall loss: -50.7676849365
Iteration: 290, named_losses: [('ActivationMax Loss', -117.24828),
('L-6.0 Norm Loss', 0.06243369),
('TV(2.0) Loss', 61.591736)], overall loss: -55.5941085815
Iteration: 291, named_losses: [('ActivationMax Loss', -111.78905),
('L-6.0 Norm Loss', 0.062436443),
('TV(2.0) Loss', 60.753647)], overall loss: -50.9729690552
Iteration: 292, named_losses: [('ActivationMax Loss', -118.45187),
('L-6.0 Norm Loss', 0.062438887),
('TV(2.0) Loss', 62.12117)], overall loss: -56.2682647705
Iteration: 293, named_losses: [('ActivationMax Loss', -116.6367),
('L-6.0 Norm Loss', 0.062440295),
('TV(2.0) Loss', 60.871086)], overall loss: -55.7031707764
Iteration: 294, named_losses: [('ActivationMax Loss', -115.75291),
('L-6.0 Norm Loss', 0.062444028),
('TV(2.0) Loss', 63.294563)], overall loss: -52.395904541
Iteration: 295, named_losses: [('ActivationMax Loss', -120.34003),
('L-6.0 Norm Loss', 0.06244716),
('TV(2.0) Loss', 61.570621)], overall loss: -58.7069664001
Iteration: 296, named_losses: [('ActivationMax Loss', -117.56786),
('L-6.0 Norm Loss', 0.062450472),
('TV(2.0) Loss', 63.104797)], overall loss: -54.400604248
Iteration: 297, named_losses: [('ActivationMax Loss', -113.33498),
('L-6.0 Norm Loss', 0.062454388),
('TV(2.0) Loss', 61.661232)], overall loss: -51.611289978
Iteration: 298, named_losses: [('ActivationMax Loss', -118.31075),
('L-6.0 Norm Loss', 0.062455233),
('TV(2.0) Loss', 63.642811)], overall loss: -54.6054878235
Iteration: 299, named_losses: [('ActivationMax Loss', -117.61665),
('L-6.0 Norm Loss', 0.06245964),
('TV(2.0) Loss', 62.600723)], overall loss: -54.9534606934
Iteration: 300, named_losses: [('ActivationMax Loss', -119.81812),
('L-6.0 Norm Loss', 0.062461127),
('TV(2.0) Loss', 64.406975)], overall loss: -55.3486862183
Iteration: 301, named_losses: [('ActivationMax Loss', -115.15378),
('L-6.0 Norm Loss', 0.062465556),
('TV(2.0) Loss', 63.529228)], overall loss: -51.5620880127
Iteration: 302, named_losses: [('ActivationMax Loss', -120.78297),
('L-6.0 Norm Loss', 0.06246718),
('TV(2.0) Loss', 64.324959)], overall loss: -56.3955383301
Iteration: 303, named_losses: [('ActivationMax Loss', -117.20642),
('L-6.0 Norm Loss', 0.062469989),
('TV(2.0) Loss', 62.703556)], overall loss: -54.4403953552
Iteration: 304, named_losses: [('ActivationMax Loss', -118.35577),
('L-6.0 Norm Loss', 0.06247329),
('TV(2.0) Loss', 62.835968)], overall loss: -55.4573364258
Iteration: 305, named_losses: [('ActivationMax Loss', -114.39941),
('L-6.0 Norm Loss', 0.062474035),
('TV(2.0) Loss', 62.751347)], overall loss: -51.5855827332
Iteration: 306, named_losses: [('ActivationMax Loss', -120.15311),
('L-6.0 Norm Loss', 0.062477037),
('TV(2.0) Loss', 63.238388)], overall loss: -56.8522415161
Iteration: 307, named_losses: [('ActivationMax Loss', -120.37479),
('L-6.0 Norm Loss', 0.062480524),
('TV(2.0) Loss', 63.184933)], overall loss: -57.1273841858
Iteration: 308, named_losses: [('ActivationMax Loss', -121.32096),
('L-6.0 Norm Loss', 0.062484484),
('TV(2.0) Loss', 63.428288)], overall loss: -57.8301887512
Iteration: 309, named_losses: [('ActivationMax Loss', -115.52107),
('L-6.0 Norm Loss', 0.062486794),
('TV(2.0) Loss', 63.913368)], overall loss: -51.5452194214
Iteration: 310, named_losses: [('ActivationMax Loss', -124.97328),
('L-6.0 Norm Loss', 0.062488586),
('TV(2.0) Loss', 63.717896)], overall loss: -61.1928939819
Iteration: 311, named_losses: [('ActivationMax Loss', -118.34658),
('L-6.0 Norm Loss', 0.062492564),
('TV(2.0) Loss', 64.756485)], overall loss: -53.5276031494
Iteration: 312, named_losses: [('ActivationMax Loss', -119.46532),
('L-6.0 Norm Loss', 0.062495649),
('TV(2.0) Loss', 63.611916)], overall loss: -55.7909088135
Iteration: 313, named_losses: [('ActivationMax Loss', -122.28691),
('L-6.0 Norm Loss', 0.062497236),
('TV(2.0) Loss', 64.647598)], overall loss: -57.5768127441
Iteration: 314, named_losses: [('ActivationMax Loss', -122.96639),
('L-6.0 Norm Loss', 0.062500447),
('TV(2.0) Loss', 62.292442)], overall loss: -60.6114501953
Iteration: 315, named_losses: [('ActivationMax Loss', -119.77802),
('L-6.0 Norm Loss', 0.062504053),
('TV(2.0) Loss', 64.834229)], overall loss: -54.8812789917
Iteration: 316, named_losses: [('ActivationMax Loss', -117.60001),
('L-6.0 Norm Loss', 0.06250526),
('TV(2.0) Loss', 63.93232)], overall loss: -53.6051864624
Iteration: 317, named_losses: [('ActivationMax Loss', -123.99913),
('L-6.0 Norm Loss', 0.062507831),
('TV(2.0) Loss', 64.633331)], overall loss: -59.3032913208
Iteration: 318, named_losses: [('ActivationMax Loss', -121.27096),
('L-6.0 Norm Loss', 0.062513039),
('TV(2.0) Loss', 64.345695)], overall loss: -56.8627471924
Iteration: 319, named_losses: [('ActivationMax Loss', -117.6347),
('L-6.0 Norm Loss', 0.062513158),
('TV(2.0) Loss', 65.665977)], overall loss: -51.906211853
Iteration: 320, named_losses: [('ActivationMax Loss', -117.37431),
('L-6.0 Norm Loss', 0.062515974),
('TV(2.0) Loss', 64.24369)], overall loss: -53.0680999756
Iteration: 321, named_losses: [('ActivationMax Loss', -121.93172),
('L-6.0 Norm Loss', 0.062519491),
('TV(2.0) Loss', 65.83168)], overall loss: -56.0375137329
Iteration: 322, named_losses: [('ActivationMax Loss', -122.42195),
('L-6.0 Norm Loss', 0.062522121),
('TV(2.0) Loss', 64.549812)], overall loss: -57.8096160889
Iteration: 323, named_losses: [('ActivationMax Loss', -119.37366),
('L-6.0 Norm Loss', 0.062525228),
('TV(2.0) Loss', 66.0327)], overall loss: -53.2784347534
Iteration: 324, named_losses: [('ActivationMax Loss', -122.82781),
('L-6.0 Norm Loss', 0.0625275),
('TV(2.0) Loss', 64.045074)], overall loss: -58.7202072144
Iteration: 325, named_losses: [('ActivationMax Loss', -125.21674),
('L-6.0 Norm Loss', 0.062531345),
('TV(2.0) Loss', 66.365929)], overall loss: -58.7882766724
Iteration: 326, named_losses: [('ActivationMax Loss', -119.97245),
('L-6.0 Norm Loss', 0.062533632),
('TV(2.0) Loss', 64.306755)], overall loss: -55.6031646729
Iteration: 327, named_losses: [('ActivationMax Loss', -124.83936),
('L-6.0 Norm Loss', 0.062537),
('TV(2.0) Loss', 67.791237)], overall loss: -56.9855804443
Iteration: 328, named_losses: [('ActivationMax Loss', -123.57245),
('L-6.0 Norm Loss', 0.062539473),
('TV(2.0) Loss', 65.386154)], overall loss: -58.1237564087
Iteration: 329, named_losses: [('ActivationMax Loss', -128.15799),
('L-6.0 Norm Loss', 0.062542871),
('TV(2.0) Loss', 67.297089)], overall loss: -60.7983551025
Iteration: 330, named_losses: [('ActivationMax Loss', -125.39252),
('L-6.0 Norm Loss', 0.062543713),
('TV(2.0) Loss', 66.658058)], overall loss: -58.671913147
Iteration: 331, named_losses: [('ActivationMax Loss', -124.52459),
('L-6.0 Norm Loss', 0.062547408),
('TV(2.0) Loss', 67.887253)], overall loss: -56.5747909546
Iteration: 332, named_losses: [('ActivationMax Loss', -126.75299),
('L-6.0 Norm Loss', 0.062550858),
('TV(2.0) Loss', 66.926544)], overall loss: -59.7638931274
Iteration: 333, named_losses: [('ActivationMax Loss', -126.92239),
('L-6.0 Norm Loss', 0.062551469),
('TV(2.0) Loss', 67.777855)], overall loss: -59.0819854736
Iteration: 334, named_losses: [('ActivationMax Loss', -128.23158),
('L-6.0 Norm Loss', 0.062553898),
('TV(2.0) Loss', 66.244347)], overall loss: -61.9246749878
Iteration: 335, named_losses: [('ActivationMax Loss', -126.21046),
('L-6.0 Norm Loss', 0.062556483),
('TV(2.0) Loss', 68.55365)], overall loss: -57.59425354
Iteration: 336, named_losses: [('ActivationMax Loss', -131.94002),
('L-6.0 Norm Loss', 0.062559016),
('TV(2.0) Loss', 65.856522)], overall loss: -66.0209350586
Iteration: 337, named_losses: [('ActivationMax Loss', -121.91496),
('L-6.0 Norm Loss', 0.062561005),
('TV(2.0) Loss', 68.109398)], overall loss: -53.7430038452
Iteration: 338, named_losses: [('ActivationMax Loss', -127.00276),
('L-6.0 Norm Loss', 0.062565938),
('TV(2.0) Loss', 66.57309)], overall loss: -60.3671035767
Iteration: 339, named_losses: [('ActivationMax Loss', -128.661),
('L-6.0 Norm Loss', 0.062566891),
('TV(2.0) Loss', 68.095688)], overall loss: -60.502746582
Iteration: 340, named_losses: [('ActivationMax Loss', -128.08061),
('L-6.0 Norm Loss', 0.062571406),
('TV(2.0) Loss', 66.769478)], overall loss: -61.2485580444
Iteration: 341, named_losses: [('ActivationMax Loss', -126.48823),
('L-6.0 Norm Loss', 0.062574655),
('TV(2.0) Loss', 67.75621)], overall loss: -58.6694412231
Iteration: 342, named_losses: [('ActivationMax Loss', -122.09871),
('L-6.0 Norm Loss', 0.06257762),
('TV(2.0) Loss', 68.08297)], overall loss: -53.953163147
Iteration: 343, named_losses: [('ActivationMax Loss', -129.41879),
('L-6.0 Norm Loss', 0.062582485),
('TV(2.0) Loss', 68.20504)], overall loss: -61.1511764526
Iteration: 344, named_losses: [('ActivationMax Loss', -129.94044),
('L-6.0 Norm Loss', 0.062582567),
('TV(2.0) Loss', 67.162056)], overall loss: -62.7158126831
Iteration: 345, named_losses: [('ActivationMax Loss', -128.48019),
('L-6.0 Norm Loss', 0.062585577),
('TV(2.0) Loss', 67.913086)], overall loss: -60.5045166016
Iteration: 346, named_losses: [('ActivationMax Loss', -127.74873),
('L-6.0 Norm Loss', 0.062588036),
('TV(2.0) Loss', 67.574097)], overall loss: -60.1120376587
Iteration: 347, named_losses: [('ActivationMax Loss', -132.95065),
('L-6.0 Norm Loss', 0.062591277),
('TV(2.0) Loss', 67.622467)], overall loss: -65.2655944824
Iteration: 348, named_losses: [('ActivationMax Loss', -128.11633),
('L-6.0 Norm Loss', 0.062594086),
('TV(2.0) Loss', 67.88636)], overall loss: -60.1673812866
Iteration: 349, named_losses: [('ActivationMax Loss', -131.07497),
('L-6.0 Norm Loss', 0.062597908),
('TV(2.0) Loss', 68.484192)], overall loss: -62.5281829834
Iteration: 350, named_losses: [('ActivationMax Loss', -129.94107),
('L-6.0 Norm Loss', 0.062600285),
('TV(2.0) Loss', 67.976715)], overall loss: -61.9017486572
Iteration: 351, named_losses: [('ActivationMax Loss', -129.11758),
('L-6.0 Norm Loss', 0.06260372),
('TV(2.0) Loss', 68.361275)], overall loss: -60.6937026978
Iteration: 352, named_losses: [('ActivationMax Loss', -133.146),
('L-6.0 Norm Loss', 0.062606096),
('TV(2.0) Loss', 67.894104)], overall loss: -65.1892852783
Iteration: 353, named_losses: [('ActivationMax Loss', -132.46341),
('L-6.0 Norm Loss', 0.062606864),
('TV(2.0) Loss', 67.656242)], overall loss: -64.7445602417
Iteration: 354, named_losses: [('ActivationMax Loss', -132.15623),
('L-6.0 Norm Loss', 0.062608726),
('TV(2.0) Loss', 68.552078)], overall loss: -63.5415496826
Iteration: 355, named_losses: [('ActivationMax Loss', -129.22891),
('L-6.0 Norm Loss', 0.062612206),
('TV(2.0) Loss', 69.048164)], overall loss: -60.1181411743
Iteration: 356, named_losses: [('ActivationMax Loss', -133.20561),
('L-6.0 Norm Loss', 0.06261757),
('TV(2.0) Loss', 70.207153)], overall loss: -62.935836792
Iteration: 357, named_losses: [('ActivationMax Loss', -131.26804),
('L-6.0 Norm Loss', 0.06261877),
('TV(2.0) Loss', 69.271111)], overall loss: -61.9343032837
Iteration: 358, named_losses: [('ActivationMax Loss', -128.54488),
('L-6.0 Norm Loss', 0.06262318),
('TV(2.0) Loss', 68.563278)], overall loss: -59.9189758301
Iteration: 359, named_losses: [('ActivationMax Loss', -130.45735),
('L-6.0 Norm Loss', 0.062625036),
('TV(2.0) Loss', 70.652313)], overall loss: -59.7424163818
Iteration: 360, named_losses: [('ActivationMax Loss', -132.21051),
('L-6.0 Norm Loss', 0.062629029),
('TV(2.0) Loss', 68.366829)], overall loss: -63.7810592651
Iteration: 361, named_losses: [('ActivationMax Loss', -130.64563),
('L-6.0 Norm Loss', 0.062630475),
('TV(2.0) Loss', 70.482147)], overall loss: -60.1008453369
Iteration: 362, named_losses: [('ActivationMax Loss', -132.06813),
('L-6.0 Norm Loss', 0.062631987),
('TV(2.0) Loss', 67.446716)], overall loss: -64.5587768555
Iteration: 363, named_losses: [('ActivationMax Loss', -133.40018),
('L-6.0 Norm Loss', 0.062635988),
('TV(2.0) Loss', 70.401825)], overall loss: -62.9357147217
Iteration: 364, named_losses: [('ActivationMax Loss', -135.07903),
('L-6.0 Norm Loss', 0.062636971),
('TV(2.0) Loss', 68.114494)], overall loss: -66.9018936157
Iteration: 365, named_losses: [('ActivationMax Loss', -131.06651),
('L-6.0 Norm Loss', 0.062642239),
('TV(2.0) Loss', 70.215195)], overall loss: -60.7886810303
Iteration: 366, named_losses: [('ActivationMax Loss', -136.80736),
('L-6.0 Norm Loss', 0.062643565),
('TV(2.0) Loss', 69.126923)], overall loss: -67.6177978516
Iteration: 367, named_losses: [('ActivationMax Loss', -133.58646),
('L-6.0 Norm Loss', 0.062649891),
('TV(2.0) Loss', 71.965187)], overall loss: -61.5586166382
Iteration: 368, named_losses: [('ActivationMax Loss', -131.34685),
('L-6.0 Norm Loss', 0.062649332),
('TV(2.0) Loss', 69.641693)], overall loss: -61.6425018311
Iteration: 369, named_losses: [('ActivationMax Loss', -130.57747),
('L-6.0 Norm Loss', 0.06265381),
('TV(2.0) Loss', 71.754013)], overall loss: -58.7608032227
Iteration: 370, named_losses: [('ActivationMax Loss', -137.10349),
('L-6.0 Norm Loss', 0.062655605),
('TV(2.0) Loss', 69.931778)], overall loss: -67.1090545654
Iteration: 371, named_losses: [('ActivationMax Loss', -132.79715),
('L-6.0 Norm Loss', 0.062658906),
('TV(2.0) Loss', 71.617867)], overall loss: -61.1166305542
Iteration: 372, named_losses: [('ActivationMax Loss', -137.56659),
('L-6.0 Norm Loss', 0.062662616),
('TV(2.0) Loss', 69.219322)], overall loss: -68.2845993042
Iteration: 373, named_losses: [('ActivationMax Loss', -134.81746),
('L-6.0 Norm Loss', 0.06266503),
('TV(2.0) Loss', 71.961533)], overall loss: -62.793258667
Iteration: 374, named_losses: [('ActivationMax Loss', -139.00591),
('L-6.0 Norm Loss', 0.062667996),
('TV(2.0) Loss', 69.214539)], overall loss: -69.7286987305
Iteration: 375, named_losses: [('ActivationMax Loss', -139.2256),
('L-6.0 Norm Loss', 0.062671885),
('TV(2.0) Loss', 71.610886)], overall loss: -67.5520477295
Iteration: 376, named_losses: [('ActivationMax Loss', -133.77788),
('L-6.0 Norm Loss', 0.062674493),
('TV(2.0) Loss', 69.77211)], overall loss: -63.9430999756
Iteration: 377, named_losses: [('ActivationMax Loss', -134.90303),
('L-6.0 Norm Loss', 0.06267599),
('TV(2.0) Loss', 72.281776)], overall loss: -62.5585708618
Iteration: 378, named_losses: [('ActivationMax Loss', -141.00626),
('L-6.0 Norm Loss', 0.062678412),
('TV(2.0) Loss', 69.659378)], overall loss: -71.2841949463
Iteration: 379, named_losses: [('ActivationMax Loss', -134.6557),
('L-6.0 Norm Loss', 0.0626828),
('TV(2.0) Loss', 71.964676)], overall loss: -62.6283416748
Iteration: 380, named_losses: [('ActivationMax Loss', -137.4716),
('L-6.0 Norm Loss', 0.062684275),
('TV(2.0) Loss', 70.77964)], overall loss: -66.6292800903
Iteration: 381, named_losses: [('ActivationMax Loss', -132.44173),
('L-6.0 Norm Loss', 0.062688865),
('TV(2.0) Loss', 72.374557)], overall loss: -60.004486084
Iteration: 382, named_losses: [('ActivationMax Loss', -138.54382),
('L-6.0 Norm Loss', 0.062690705),
('TV(2.0) Loss', 70.004189)], overall loss: -68.4769515991
Iteration: 383, named_losses: [('ActivationMax Loss', -134.41895),
('L-6.0 Norm Loss', 0.062693328),
('TV(2.0) Loss', 72.389236)], overall loss: -61.967010498
Iteration: 384, named_losses: [('ActivationMax Loss', -139.12358),
('L-6.0 Norm Loss', 0.062696755),
('TV(2.0) Loss', 70.258804)], overall loss: -68.8020782471
Iteration: 385, named_losses: [('ActivationMax Loss', -136.84123),
('L-6.0 Norm Loss', 0.06269788),
('TV(2.0) Loss', 72.239212)], overall loss: -64.5393218994
Iteration: 386, named_losses: [('ActivationMax Loss', -140.56212),
('L-6.0 Norm Loss', 0.062701583),
('TV(2.0) Loss', 70.644363)], overall loss: -69.8550567627
Iteration: 387, named_losses: [('ActivationMax Loss', -139.22398),
('L-6.0 Norm Loss', 0.062704168),
('TV(2.0) Loss', 74.081116)], overall loss: -65.0801696777
Iteration: 388, named_losses: [('ActivationMax Loss', -138.96829),
('L-6.0 Norm Loss', 0.062707596),
('TV(2.0) Loss', 70.89389)], overall loss: -68.0116882324
Iteration: 389, named_losses: [('ActivationMax Loss', -139.73682),
('L-6.0 Norm Loss', 0.062710412),
('TV(2.0) Loss', 72.549789)], overall loss: -67.1243133545
Iteration: 390, named_losses: [('ActivationMax Loss', -139.44196),
('L-6.0 Norm Loss', 0.06271337),
('TV(2.0) Loss', 71.712288)], overall loss: -67.6669540405
Iteration: 391, named_losses: [('ActivationMax Loss', -134.49084),
('L-6.0 Norm Loss', 0.06271641),
('TV(2.0) Loss', 72.558838)], overall loss: -61.8692932129
Iteration: 392, named_losses: [('ActivationMax Loss', -140.21024),
('L-6.0 Norm Loss', 0.062720843),
('TV(2.0) Loss', 72.034454)], overall loss: -68.113067627
Iteration: 393, named_losses: [('ActivationMax Loss', -136.07448),
('L-6.0 Norm Loss', 0.062722653),
('TV(2.0) Loss', 72.450783)], overall loss: -63.5609664917
Iteration: 394, named_losses: [('ActivationMax Loss', -139.89977),
('L-6.0 Norm Loss', 0.062726103),
('TV(2.0) Loss', 71.97332)], overall loss: -67.8637161255
Iteration: 395, named_losses: [('ActivationMax Loss', -137.41159),
('L-6.0 Norm Loss', 0.062728763),
('TV(2.0) Loss', 72.810127)], overall loss: -64.538734436
Iteration: 396, named_losses: [('ActivationMax Loss', -142.73474),
('L-6.0 Norm Loss', 0.062729523),
('TV(2.0) Loss', 71.824142)], overall loss: -70.847869873
Iteration: 397, named_losses: [('ActivationMax Loss', -140.91222),
('L-6.0 Norm Loss', 0.062735163),
('TV(2.0) Loss', 72.537704)], overall loss: -68.3117828369
Iteration: 398, named_losses: [('ActivationMax Loss', -141.6562),
('L-6.0 Norm Loss', 0.062736541),
('TV(2.0) Loss', 72.590141)], overall loss: -69.0033187866
Iteration: 399, named_losses: [('ActivationMax Loss', -142.24733),
('L-6.0 Norm Loss', 0.062739797),
('TV(2.0) Loss', 73.031548)], overall loss: -69.1530380249
Iteration: 400, named_losses: [('ActivationMax Loss', -140.57739),
('L-6.0 Norm Loss', 0.062742479),
('TV(2.0) Loss', 71.91481)], overall loss: -68.5998382568
Iteration: 401, named_losses: [('ActivationMax Loss', -139.50906),
('L-6.0 Norm Loss', 0.062745459),
('TV(2.0) Loss', 72.750412)], overall loss: -66.6959075928
Iteration: 402, named_losses: [('ActivationMax Loss', -143.27655),
('L-6.0 Norm Loss', 0.062746421),
('TV(2.0) Loss', 72.066795)], overall loss: -71.1470108032
Iteration: 403, named_losses: [('ActivationMax Loss', -139.51357),
('L-6.0 Norm Loss', 0.0627506),
('TV(2.0) Loss', 72.188675)], overall loss: -67.2621459961
Iteration: 404, named_losses: [('ActivationMax Loss', -142.9973),
('L-6.0 Norm Loss', 0.062753551),
('TV(2.0) Loss', 73.722336)], overall loss: -69.2122039795
Iteration: 405, named_losses: [('ActivationMax Loss', -138.62294),
('L-6.0 Norm Loss', 0.062756181),
('TV(2.0) Loss', 72.825424)], overall loss: -65.7347564697
Iteration: 406, named_losses: [('ActivationMax Loss', -144.35088),
('L-6.0 Norm Loss', 0.062758714),
('TV(2.0) Loss', 72.848717)], overall loss: -71.4393997192
Iteration: 407, named_losses: [('ActivationMax Loss', -142.30759),
('L-6.0 Norm Loss', 0.062761292),
('TV(2.0) Loss', 72.600616)], overall loss: -69.6442108154
Iteration: 408, named_losses: [('ActivationMax Loss', -139.07095),
('L-6.0 Norm Loss', 0.062764049),
('TV(2.0) Loss', 73.714333)], overall loss: -65.2938613892
Iteration: 409, named_losses: [('ActivationMax Loss', -142.22777),
('L-6.0 Norm Loss', 0.062768973),
('TV(2.0) Loss', 73.164162)], overall loss: -69.000831604
Iteration: 410, named_losses: [('ActivationMax Loss', -141.81583),
('L-6.0 Norm Loss', 0.062771544),
('TV(2.0) Loss', 74.028221)], overall loss: -67.7248306274
Iteration: 411, named_losses: [('ActivationMax Loss', -145.58228),
('L-6.0 Norm Loss', 0.062774956),
('TV(2.0) Loss', 72.881424)], overall loss: -72.6380767822
Iteration: 412, named_losses: [('ActivationMax Loss', -141.39627),
('L-6.0 Norm Loss', 0.062777713),
('TV(2.0) Loss', 73.894768)], overall loss: -67.4387283325
Iteration: 413, named_losses: [('ActivationMax Loss', -144.37086),
('L-6.0 Norm Loss', 0.062781513),
('TV(2.0) Loss', 73.111526)], overall loss: -71.1965637207
Iteration: 414, named_losses: [('ActivationMax Loss', -138.76392),
('L-6.0 Norm Loss', 0.062785141),
('TV(2.0) Loss', 75.106598)], overall loss: -63.5945281982
Iteration: 415, named_losses: [('ActivationMax Loss', -143.54672),
('L-6.0 Norm Loss', 0.062787428),
('TV(2.0) Loss', 73.301186)], overall loss: -70.1827468872
Iteration: 416, named_losses: [('ActivationMax Loss', -145.38876),
('L-6.0 Norm Loss', 0.062790059),
('TV(2.0) Loss', 73.513397)], overall loss: -71.8125762939
Iteration: 417, named_losses: [('ActivationMax Loss', -141.94406),
('L-6.0 Norm Loss', 0.062794261),
('TV(2.0) Loss', 73.975876)], overall loss: -67.9053955078
Iteration: 418, named_losses: [('ActivationMax Loss', -145.29156),
('L-6.0 Norm Loss', 0.062796727),
('TV(2.0) Loss', 74.410286)], overall loss: -70.8184890747
Iteration: 419, named_losses: [('ActivationMax Loss', -143.43077),
('L-6.0 Norm Loss', 0.062801622),
('TV(2.0) Loss', 74.717628)], overall loss: -68.6503372192
Iteration: 420, named_losses: [('ActivationMax Loss', -145.27054),
('L-6.0 Norm Loss', 0.062802516),
('TV(2.0) Loss', 74.45179)], overall loss: -70.7559432983
Iteration: 421, named_losses: [('ActivationMax Loss', -144.51248),
('L-6.0 Norm Loss', 0.06280712),
('TV(2.0) Loss', 75.463799)], overall loss: -68.9858779907
Iteration: 422, named_losses: [('ActivationMax Loss', -143.84959),
('L-6.0 Norm Loss', 0.062810346),
('TV(2.0) Loss', 73.957306)], overall loss: -69.8294830322
Iteration: 423, named_losses: [('ActivationMax Loss', -145.58594),
('L-6.0 Norm Loss', 0.062813528),
('TV(2.0) Loss', 75.771721)], overall loss: -69.7513961792
Iteration: 424, named_losses: [('ActivationMax Loss', -145.62358),
('L-6.0 Norm Loss', 0.062815405),
('TV(2.0) Loss', 74.289368)], overall loss: -71.2713928223
Iteration: 425, named_losses: [('ActivationMax Loss', -146.09689),
('L-6.0 Norm Loss', 0.062818445),
('TV(2.0) Loss', 75.823204)], overall loss: -70.2108688354
Iteration: 426, named_losses: [('ActivationMax Loss', -146.34875),
('L-6.0 Norm Loss', 0.062820926),
('TV(2.0) Loss', 75.21714)], overall loss: -71.0687942505
Iteration: 427, named_losses: [('ActivationMax Loss', -146.32544),
('L-6.0 Norm Loss', 0.062824123),
('TV(2.0) Loss', 76.579735)], overall loss: -69.6828842163
Iteration: 428, named_losses: [('ActivationMax Loss', -143.18855),
('L-6.0 Norm Loss', 0.062829934),
('TV(2.0) Loss', 75.339134)], overall loss: -67.7865829468
Iteration: 429, named_losses: [('ActivationMax Loss', -147.66838),
('L-6.0 Norm Loss', 0.062832028),
('TV(2.0) Loss', 75.044785)], overall loss: -72.560760498
Iteration: 430, named_losses: [('ActivationMax Loss', -144.30286),
('L-6.0 Norm Loss', 0.062835753),
('TV(2.0) Loss', 75.958275)], overall loss: -68.2817459106
Iteration: 431, named_losses: [('ActivationMax Loss', -147.18196),
('L-6.0 Norm Loss', 0.062836736),
('TV(2.0) Loss', 74.678429)], overall loss: -72.4406967163
Iteration: 432, named_losses: [('ActivationMax Loss', -147.75696),
('L-6.0 Norm Loss', 0.062842377),
('TV(2.0) Loss', 75.894943)], overall loss: -71.7991790771
Iteration: 433, named_losses: [('ActivationMax Loss', -147.81818),
('L-6.0 Norm Loss', 0.062843263),
('TV(2.0) Loss', 75.730606)], overall loss: -72.0247344971
Iteration: 434, named_losses: [('ActivationMax Loss', -146.59891),
('L-6.0 Norm Loss', 0.062848829),
('TV(2.0) Loss', 76.10627)], overall loss: -70.4297866821
Iteration: 435, named_losses: [('ActivationMax Loss', -149.23871),
('L-6.0 Norm Loss', 0.06285169),
('TV(2.0) Loss', 76.117828)], overall loss: -73.0580291748
Iteration: 436, named_losses: [('ActivationMax Loss', -147.08241),
('L-6.0 Norm Loss', 0.062855311),
('TV(2.0) Loss', 75.71904)], overall loss: -71.3005218506
Iteration: 437, named_losses: [('ActivationMax Loss', -144.91626),
('L-6.0 Norm Loss', 0.062857568),
('TV(2.0) Loss', 76.941376)], overall loss: -67.9120330811
Iteration: 438, named_losses: [('ActivationMax Loss', -147.75507),
('L-6.0 Norm Loss', 0.062860459),
('TV(2.0) Loss', 76.416542)], overall loss: -71.2756576538
Iteration: 439, named_losses: [('ActivationMax Loss', -146.35825),
('L-6.0 Norm Loss', 0.062862925),
('TV(2.0) Loss', 76.040337)], overall loss: -70.2550430298
Iteration: 440, named_losses: [('ActivationMax Loss', -148.93884),
('L-6.0 Norm Loss', 0.062865987),
('TV(2.0) Loss', 76.496346)], overall loss: -72.3796310425
Iteration: 441, named_losses: [('ActivationMax Loss', -149.18591),
('L-6.0 Norm Loss', 0.062868237),
('TV(2.0) Loss', 75.888054)], overall loss: -73.234992981
Iteration: 442, named_losses: [('ActivationMax Loss', -147.30473),
('L-6.0 Norm Loss', 0.062870204),
('TV(2.0) Loss', 76.196472)], overall loss: -71.0453948975
Iteration: 443, named_losses: [('ActivationMax Loss', -151.31589),
('L-6.0 Norm Loss', 0.062872902),
('TV(2.0) Loss', 75.758423)], overall loss: -75.4945983887
Iteration: 444, named_losses: [('ActivationMax Loss', -150.51529),
('L-6.0 Norm Loss', 0.062876485),
('TV(2.0) Loss', 76.557373)], overall loss: -73.89503479
Iteration: 445, named_losses: [('ActivationMax Loss', -148.82353),
('L-6.0 Norm Loss', 0.062880047),
('TV(2.0) Loss', 76.311279)], overall loss: -72.4493713379
Iteration: 446, named_losses: [('ActivationMax Loss', -151.71295),
('L-6.0 Norm Loss', 0.062881567),
('TV(2.0) Loss', 77.365166)], overall loss: -74.28490448
Iteration: 447, named_losses: [('ActivationMax Loss', -149.31421),
('L-6.0 Norm Loss', 0.062885903),
('TV(2.0) Loss', 76.143028)], overall loss: -73.1082992554
Iteration: 448, named_losses: [('ActivationMax Loss', -150.26318),
('L-6.0 Norm Loss', 0.062887043),
('TV(2.0) Loss', 76.581001)], overall loss: -73.6193008423
Iteration: 449, named_losses: [('ActivationMax Loss', -151.68604),
('L-6.0 Norm Loss', 0.062892638),
('TV(2.0) Loss', 76.890915)], overall loss: -74.7322235107
Iteration: 450, named_losses: [('ActivationMax Loss', -148.28377),
('L-6.0 Norm Loss', 0.062893935),
('TV(2.0) Loss', 75.592422)], overall loss: -72.6284484863
Iteration: 451, named_losses: [('ActivationMax Loss', -147.35678),
('L-6.0 Norm Loss', 0.062899612),
('TV(2.0) Loss', 78.092506)], overall loss: -69.2013778687
Iteration: 452, named_losses: [('ActivationMax Loss', -153.25745),
('L-6.0 Norm Loss', 0.062901981),
('TV(2.0) Loss', 75.079391)], overall loss: -78.1151580811
Iteration: 453, named_losses: [('ActivationMax Loss', -149.76364),
('L-6.0 Norm Loss', 0.062906131),
('TV(2.0) Loss', 78.377541)], overall loss: -71.3231887817
Iteration: 454, named_losses: [('ActivationMax Loss', -152.04784),
('L-6.0 Norm Loss', 0.062906317),
('TV(2.0) Loss', 75.977638)], overall loss: -76.0072860718
Iteration: 455, named_losses: [('ActivationMax Loss', -148.44231),
('L-6.0 Norm Loss', 0.062910795),
('TV(2.0) Loss', 77.3032)], overall loss: -71.0761947632
Iteration: 456, named_losses: [('ActivationMax Loss', -153.95691),
('L-6.0 Norm Loss', 0.062912397),
('TV(2.0) Loss', 76.434631)], overall loss: -77.4593658447
Iteration: 457, named_losses: [('ActivationMax Loss', -146.44548),
('L-6.0 Norm Loss', 0.062915578),
('TV(2.0) Loss', 79.674469)], overall loss: -66.7080993652
Iteration: 458, named_losses: [('ActivationMax Loss', -153.75627),
('L-6.0 Norm Loss', 0.062918916),
('TV(2.0) Loss', 76.898636)], overall loss: -76.7947235107
Iteration: 459, named_losses: [('ActivationMax Loss', -148.89221),
('L-6.0 Norm Loss', 0.062920913),
('TV(2.0) Loss', 78.139786)], overall loss: -70.6894989014
Iteration: 460, named_losses: [('ActivationMax Loss', -151.31757),
('L-6.0 Norm Loss', 0.062923267),
('TV(2.0) Loss', 77.551994)], overall loss: -73.7026443481
Iteration: 461, named_losses: [('ActivationMax Loss', -149.07819),
('L-6.0 Norm Loss', 0.062927485),
('TV(2.0) Loss', 78.698586)], overall loss: -70.3166732788
Iteration: 462, named_losses: [('ActivationMax Loss', -152.49811),
('L-6.0 Norm Loss', 0.062929422),
('TV(2.0) Loss', 76.873299)], overall loss: -75.561882019
Iteration: 463, named_losses: [('ActivationMax Loss', -153.04091),
('L-6.0 Norm Loss', 0.062933609),
('TV(2.0) Loss', 78.172409)], overall loss: -74.8055725098
Iteration: 464, named_losses: [('ActivationMax Loss', -153.95583),
('L-6.0 Norm Loss', 0.062936828),
('TV(2.0) Loss', 77.590683)], overall loss: -76.3022003174
Iteration: 465, named_losses: [('ActivationMax Loss', -156.10953),
('L-6.0 Norm Loss', 0.062940717),
('TV(2.0) Loss', 79.119041)], overall loss: -76.9275436401
Iteration: 466, named_losses: [('ActivationMax Loss', -145.71866),
('L-6.0 Norm Loss', 0.062943906),
('TV(2.0) Loss', 77.459)], overall loss: -68.1967163086
Iteration: 467, named_losses: [('ActivationMax Loss', -154.64474),
('L-6.0 Norm Loss', 0.06294553),
('TV(2.0) Loss', 79.444298)], overall loss: -75.1375045776
Iteration: 468, named_losses: [('ActivationMax Loss', -151.04881),
('L-6.0 Norm Loss', 0.062949196),
('TV(2.0) Loss', 76.980843)], overall loss: -74.005027771
Iteration: 469, named_losses: [('ActivationMax Loss', -150.35945),
('L-6.0 Norm Loss', 0.062951736),
('TV(2.0) Loss', 78.674477)], overall loss: -71.6220169067
Iteration: 470, named_losses: [('ActivationMax Loss', -156.62335),
('L-6.0 Norm Loss', 0.062953971),
('TV(2.0) Loss', 77.191483)], overall loss: -79.3689117432
Iteration: 471, named_losses: [('ActivationMax Loss', -150.9823),
('L-6.0 Norm Loss', 0.062956445),
('TV(2.0) Loss', 79.235382)], overall loss: -71.6839599609
Iteration: 472, named_losses: [('ActivationMax Loss', -154.03825),
('L-6.0 Norm Loss', 0.062961489),
('TV(2.0) Loss', 78.472809)], overall loss: -75.5024871826
Iteration: 473, named_losses: [('ActivationMax Loss', -155.60004),
('L-6.0 Norm Loss', 0.062963262),
('TV(2.0) Loss', 79.945732)], overall loss: -75.5913467407
Iteration: 474, named_losses: [('ActivationMax Loss', -152.65337),
('L-6.0 Norm Loss', 0.062968515),
('TV(2.0) Loss', 78.46608)], overall loss: -74.1243133545
Iteration: 475, named_losses: [('ActivationMax Loss', -155.58034),
('L-6.0 Norm Loss', 0.062968753),
('TV(2.0) Loss', 79.334038)], overall loss: -76.1833267212
Iteration: 476, named_losses: [('ActivationMax Loss', -154.90883),
('L-6.0 Norm Loss', 0.062973261),
('TV(2.0) Loss', 77.762535)], overall loss: -77.0833206177
Iteration: 477, named_losses: [('ActivationMax Loss', -154.97806),
('L-6.0 Norm Loss', 0.062974989),
('TV(2.0) Loss', 79.10482)], overall loss: -75.8102645874
Iteration: 478, named_losses: [('ActivationMax Loss', -152.09059),
('L-6.0 Norm Loss', 0.06297908),
('TV(2.0) Loss', 77.791771)], overall loss: -74.2358474731
Iteration: 479, named_losses: [('ActivationMax Loss', -156.22099),
('L-6.0 Norm Loss', 0.062981963),
('TV(2.0) Loss', 80.056015)], overall loss: -76.1019897461
Iteration: 480, named_losses: [('ActivationMax Loss', -153.39764),
('L-6.0 Norm Loss', 0.062984273),
('TV(2.0) Loss', 78.513351)], overall loss: -74.8213043213
Iteration: 481, named_losses: [('ActivationMax Loss', -151.23335),
('L-6.0 Norm Loss', 0.062989578),
('TV(2.0) Loss', 79.159126)], overall loss: -72.0112380981
Iteration: 482, named_losses: [('ActivationMax Loss', -154.5715),
('L-6.0 Norm Loss', 0.062990718),
('TV(2.0) Loss', 79.108345)], overall loss: -75.4001693726
Iteration: 483, named_losses: [('ActivationMax Loss', -152.37679),
('L-6.0 Norm Loss', 0.062994026),
('TV(2.0) Loss', 79.806961)], overall loss: -72.5068359375
Iteration: 484, named_losses: [('ActivationMax Loss', -155.35219),
('L-6.0 Norm Loss', 0.062995881),
('TV(2.0) Loss', 78.537399)], overall loss: -76.7518005371
Iteration: 485, named_losses: [('ActivationMax Loss', -150.86581),
('L-6.0 Norm Loss', 0.062999263),
('TV(2.0) Loss', 79.860237)], overall loss: -70.9425735474
Iteration: 486, named_losses: [('ActivationMax Loss', -156.02452),
('L-6.0 Norm Loss', 0.063002303),
('TV(2.0) Loss', 78.534882)], overall loss: -77.4266357422
Iteration: 487, named_losses: [('ActivationMax Loss', -152.9613),
('L-6.0 Norm Loss', 0.063006014),
('TV(2.0) Loss', 79.599541)], overall loss: -73.2987594604
Iteration: 488, named_losses: [('ActivationMax Loss', -156.22141),
('L-6.0 Norm Loss', 0.063007407),
('TV(2.0) Loss', 78.018753)], overall loss: -78.1396484375
Iteration: 489, named_losses: [('ActivationMax Loss', -151.55534),
('L-6.0 Norm Loss', 0.063012823),
('TV(2.0) Loss', 78.805367)], overall loss: -72.686958313
Iteration: 490, named_losses: [('ActivationMax Loss', -158.52295),
('L-6.0 Norm Loss', 0.06301301),
('TV(2.0) Loss', 78.155159)], overall loss: -80.3047714233
Iteration: 491, named_losses: [('ActivationMax Loss', -155.44188),
('L-6.0 Norm Loss', 0.06301821),
('TV(2.0) Loss', 79.538498)], overall loss: -75.8403625488
Iteration: 492, named_losses: [('ActivationMax Loss', -157.68198),
('L-6.0 Norm Loss', 0.063019305),
('TV(2.0) Loss', 78.353966)], overall loss: -79.2649917603
Iteration: 493, named_losses: [('ActivationMax Loss', -152.27164),
('L-6.0 Norm Loss', 0.063024096),
('TV(2.0) Loss', 80.214958)], overall loss: -71.9936599731
Iteration: 494, named_losses: [('ActivationMax Loss', -158.29787),
('L-6.0 Norm Loss', 0.063024633),
('TV(2.0) Loss', 78.70845)], overall loss: -79.5263977051
Iteration: 495, named_losses: [('ActivationMax Loss', -153.49835),
('L-6.0 Norm Loss', 0.063029215),
('TV(2.0) Loss', 81.051971)], overall loss: -72.3833465576
Iteration: 496, named_losses: [('ActivationMax Loss', -158.92822),
('L-6.0 Norm Loss', 0.063031986),
('TV(2.0) Loss', 78.715805)], overall loss: -80.1493835449
Iteration: 497, named_losses: [('ActivationMax Loss', -155.50137),
('L-6.0 Norm Loss', 0.063035131),
('TV(2.0) Loss', 81.072197)], overall loss: -74.3661422729
Iteration: 498, named_losses: [('ActivationMax Loss', -156.21909),
('L-6.0 Norm Loss', 0.063035503),
('TV(2.0) Loss', 78.621414)], overall loss: -77.5346374512
Iteration: 499, named_losses: [('ActivationMax Loss', -157.51538),
('L-6.0 Norm Loss', 0.063039958),
('TV(2.0) Loss', 81.570572)], overall loss: -75.8817749023
Iteration: 500, named_losses: [('ActivationMax Loss', -157.952),
('L-6.0 Norm Loss', 0.063040897),
('TV(2.0) Loss', 79.478874)], overall loss: -78.4100875854
Out[10]:
<matplotlib.image.AxesImage at 0x7f5418480e90>
We can see that the loss appears to be converging. So more iterations definitely seem to give better output. One way to get crisper results is to use Jitter input_modifier. As the name suggests, Jitter moves pixels around in the image. Lets try this out.
In [11]:
from vis.input_modifiers import Jitter
# 20 is the imagenet category for 'ouzel'
# Jitter 16 pixels along all dimensions to during the optimization process.
img = visualize_activation(model, layer_idx, filter_indices=20, max_iter=500, input_modifiers=[Jitter(16)])
plt.imshow(img)
Out[11]:
<matplotlib.image.AxesImage at 0x7f5418100150>
Look at that! Not only has the conv net captured what it means to be an ouzel, but it also seems to encode for different orientations and scales, a further proof of rotational and scale invariance.
Lets try this for a bunch of other random categories. This will take a while. Go grab a nice cup of coffee and prepare to be amused :)
In [12]:
import numpy as np
categories = np.random.permutation(1000)[:15]
vis_images = []
image_modifiers = [Jitter(16)]
for idx in categories:
img = visualize_activation(model, layer_idx, filter_indices=idx, max_iter=500, input_modifiers=image_modifiers)
# Reverse lookup index to imagenet label and overlay it on the image.
img = utils.draw_text(img, utils.get_imagenet_label(idx))
vis_images.append(img)
# Generate stitched images with 5 cols (so it will have 3 rows).
plt.rcParams['figure.figsize'] = (50, 50)
stitched = utils.stitch_images(vis_images, cols=5)
plt.axis('off')
plt.imshow(stitched)
plt.show()
Some of them make sense if you stare at ot for a while. There are ways of improving this. We will cover some ideas for this in the next section. You can come back here and try those out as an exercise.
In a CNN, each Conv layer has several learned template matching filters that maximize their output when a similar template pattern is found in the input image. First Conv layer is easy to interpret; simply visualize the weights as an image. To see what the Conv layer is doing, a simple option is to apply the filter over raw input pixels. Subsequent Conv filters operate over the outputs of previous Conv filters (which indicate the presence or absence of some templates), making them hard to interpret.
One way of interpreting them is to generate an input image that maximizes the filter output. This allows us to generate an input that activates the filter.
Lets start by visualizing the second conv layer of vggnet (named as 'block1_conv2'). Here is the VGG16 model for reference.
In [13]:
model.summary()
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) (None, 224, 224, 3) 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
_________________________________________________________________
flatten (Flatten) (None, 25088) 0
_________________________________________________________________
fc1 (Dense) (None, 4096) 102764544
_________________________________________________________________
fc2 (Dense) (None, 4096) 16781312
_________________________________________________________________
predictions (Dense) (None, 1000) 4097000
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________
In [14]:
from vis.visualization import get_num_filters
# The name of the layer we want to visualize
# You can see this in the model definition.
layer_name = 'block1_conv2'
layer_idx = utils.find_layer_idx(model, layer_name)
# Visualize all filters in this layer.
filters = np.arange(get_num_filters(model.layers[layer_idx]))
# Generate input image for each filter.
vis_images = []
for idx in filters:
img = visualize_activation(model, layer_idx, filter_indices=idx)
# Utility to overlay text on image.
img = utils.draw_text(img, 'Filter {}'.format(idx))
vis_images.append(img)
# Generate stitched image palette with 8 cols.
stitched = utils.stitch_images(vis_images, cols=8)
plt.axis('off')
plt.imshow(stitched)
plt.title(layer_name)
plt.show()
They mostly seem to match for specific color and directional patterns. Lets try a bunch of other layers. We will randomly visualize 10 filters within various layers.
In [15]:
selected_indices = []
for layer_name in ['block2_conv2', 'block3_conv3', 'block4_conv3', 'block5_conv3']:
layer_idx = utils.find_layer_idx(model, layer_name)
# Visualize all filters in this layer.
filters = np.random.permutation(get_num_filters(model.layers[layer_idx]))[:10]
selected_indices.append(filters)
# Generate input image for each filter.
vis_images = []
for idx in filters:
img = visualize_activation(model, layer_idx, filter_indices=idx)
# Utility to overlay text on image.
img = utils.draw_text(img, 'Filter {}'.format(idx))
vis_images.append(img)
# Generate stitched image palette with 5 cols so we get 2 rows.
stitched = utils.stitch_images(vis_images, cols=5)
plt.figure()
plt.axis('off')
plt.imshow(stitched)
plt.show()
We can see how filters evolved to look for simple -> complex abstract patterns.
We also notice that some of the filters in block5_conv3 (the last one) failed to converge. This is usually because regularization losses (total variation and LP norm) are overtaking activation maximization loss (set verbose=True to observe). There are a couple of options to make this work better,
Jitter input_modifier.seed_input with regularization enabled.I will show a subset of these ideas here. Lets start by adidng Jitter and disabling total variation.
In [16]:
layer_idx = utils.find_layer_idx(model, 'block5_conv3')
# We need to select the same random filters in order to compare the results.
filters = selected_indices[-1]
selected_indices.append(filters)
# Generate input image for each filter.
vis_images = []
for idx in filters:
# We will jitter 5% relative to the image size.
img = visualize_activation(model, layer_idx, filter_indices=idx,
tv_weight=0.,
input_modifiers=[Jitter(0.05)])
# Utility to overlay text on image.
img = utils.draw_text(img, 'Filter {}'.format(idx))
vis_images.append(img)
# Generate stitched image palette with 5 cols so we get 2 rows.
stitched = utils.stitch_images(vis_images, cols=5)
plt.figure()
plt.axis('off')
plt.imshow(stitched)
plt.show()
We can see how previously unconverged filters show something this time. Lets take a specific output from here and use it as a seed_input with total_variation enabled this time.
In [17]:
# Generate input image for each filter.
new_vis_images = []
for i, idx in enumerate(filters):
# We will seed with optimized image this time.
img = visualize_activation(model, layer_idx, filter_indices=idx,
seed_input=vis_images[i],
input_modifiers=[Jitter(0.05)])
# Utility to overlay text on image.
img = utils.draw_text(img, 'Filter {}'.format(idx))
new_vis_images.append(img)
# Generate stitched image palette with 5 cols so we get 2 rows.
stitched = utils.stitch_images(new_vis_images, cols=5)
plt.figure()
plt.axis('off')
plt.imshow(stitched)
plt.show()
And that, folks, is how we roll :) This trick works pretty well to get those stubborn filters to converge.
The API to visualize_activation accepts filter_indices. This is generally meant for multi label classifiers, but nothing prevents us from having some fun.
By setting filter_indices, to multiple output categories, we can generate an input that the network thinks is both those categories. Maybe we can generate a cool looking crab fish. I will leave this as an exersice to the reader. You mgith have to experiment with regularization weights a lot.
Ideally, we can use a GAN trained on imagenet and use the discriminator loss as a regularizer. This is easily done using visualize_activations_with_losses API. If you ever do this, please consider submitting a PR :)
As alluded at the beginning of the tutorial, we want to compare and see what happens if we didnt swap out softmax for linear activation.
Lets try the ouzel visualization again.
In [21]:
layer_idx = utils.find_layer_idx(model, 'predictions')
# Swap linear back with softmax
model.layers[layer_idx].activation = activations.softmax
model = utils.apply_modifications(model)
img = visualize_activation(model, layer_idx, filter_indices=20, input_modifiers=[Jitter(16)])
plt.rcParams['figure.figsize'] = (18, 6)
plt.imshow(img)
Out[21]:
<matplotlib.image.AxesImage at 0x7f53c40eae90>
It does not work! The reason is that maximizing an output node can be done by minimizing other outputs. Softmax is weird that way. It is the only activation that depends on other node output(s) in the layer.
Content source: raghakot/keras-vis
Similar notebooks: