2018-11-24 02:12:25
In [1]:
import numpy as np
In [5]:
# from fastai.core
def even_mults(start:float, stop:float, n:int)->np.ndarray:
"Build evenly stepped schedule from `star` to `stop` in `n` steps."
mult = stop/start
step = mult**(1/(n-1))
return np.array([start*(step**i) for i in range(n)])
let's say for a hypothetical network with 3 layer groups (conv_group_1, conv_group_2, linear_group).
In [21]:
layer_groups = ['conv_group_1', 'conv_group_2', 'linear_group']
In [17]:
def lr_range(lr:[float,slice])->np.ndarray:
if not isinstance(lr, slice): return lr
if lr.start: res = even_mults(lr.start, lr.stop, len(layer_groups))
else: res = [lr.stop/3]*(len(layer_groups)-1)+[lr.stop]
return np.array(res)
In [18]:
lr = slice(1e-3)
lr_range(lr)
Out[18]:
In [19]:
lr = 1e-3
lr_range(lr)
Out[19]:
Interesting, so if you have multiple trainable layer groups, and pass in a slice with only a stop element, you'll get the lr for the last group, and the lr / 3 for all preceeding groups.
In [20]:
# 10 layer groups
layer_groups = [i for i in range(10)]
lr = slice(1e-3)
lr_range(lr)
Out[20]:
Now what happens when I pass in a start and stop value:
In [22]:
lr = slice(1e-6, 1e-3)
In [23]:
lr_range(lr)
Out[23]:
In [31]:
1e-3/30
Out[31]:
In [30]:
1e-6*30
Out[30]:
In [32]:
(1e-3/30 + 1e-6/30)*2
Out[32]:
This is so cool. Fastai finds the order / magnitude / exponential / logorithmic mean, not the absolute mean. This is why the step multiplier is (stop/start)**1/(n-1)) where n is the number of layer groups.
In [34]:
even_mults(1e-6, 1e-3, 3)
Out[34]:
In [35]:
even_mults(1e-6, 1e-3, 10)
Out[35]:
So the question I have, and why I'm here, is: can I have discriminative learning rates with a magnitude separation of 3? So: $\frac{lr}{3^2}, \frac{lr}{3^1}, \frac{lr}{3^0} = $ lr/9, lr/3, lr
In [37]:
lr_stop = 1e-3
lr_start= lr_stop / 3**2
In [38]:
even_mults(lr_start, lr_stop, 3)
Out[38]:
In [39]:
1e-3/9
Out[39]:
This is very exciting.
It also means, for my planet resnet34 thing, I don't need to worry about the internals of the learning rate calculation & assignment. I just need to specify the correct start and end lrs.
Which means all I have to do is provide the appropriate aggression for training. This I like.
In [40]:
(1/9 + 1)/2
Out[40]:
In [47]:
5/9
Out[47]:
In [48]:
even_mults(1/9, 1, 3)
Out[48]:
In [49]:
lr_range(3)
Out[49]:
In [57]:
even_mults(1e-10, 1, 11)
Out[57]:
In [2]:
from fastai import *
from fastai.vision import *
__version__
Out[2]:
In [3]:
import torchvision
In [4]:
path = untar_data(URLs.MNIST_TINY)
tfms = get_transforms()
data = (ImageItemList.from_folder(path).split_by_folder()
.label_from_folder().transform(tfms).databunch())
learn = create_cnn(data, torchvision.models.inception_v3)
In [22]:
??models.resnet18
In [7]:
??torchvision.models.inception_v3
In [25]:
def inception_v3_2(pretrained=False, **kwargs):
r"""Inception v3 model architecture from
`"Rethinking the Inception Architecture for Computer Vision" <http://arxiv.org/abs/1512.00567>`_.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = torchvision.models.Inception3(**kwargs)
# if pretrained:
# if 'transform_input' not in kwargs:
# kwargs['transform_input'] = True
# model.load_state_dict(model_zoo.load_url(model_urls['inception_v3_google']))
return model
In [26]:
create_cnn(data, inception_v3_2)
In [ ]:
In [ ]:
??learn.fit_one_cycle
In [ ]:
??learn.lr_range
In [ ]:
??even_mults
In [ ]: