In this notebook, we'll demonstrate how to automatically select hyperparameters based on a labeled development set. Also check out the CDR tutorial for an example use case of grid search!
In [1]:
from snorkel.learning import GenerativeModelWeights
from snorkel.learning.structure import generate_label_matrix
weights = GenerativeModelWeights(10)
for i in range(10):
weights.lf_accuracy[i] = 2.5
weights.dep_similar[0, 1] = 0.25
weights.dep_similar[2, 3] = 0.25
L_gold_train, L_train = generate_label_matrix(weights, 10000)
L_gold_dev, L_dev = generate_label_matrix(weights, 1000)
Below we use RandomSearch to search over a random set (with n=5 below) of hyperparameter configurations, validating on the development set and saving the best model. Note we can also use the GridSearch class to search over all combinations of hyperparameters. We start by defining the model, then the parameter ranges to search over, then the searcher:
In [2]:
from snorkel.learning import GenerativeModel
from snorkel.learning import RandomSearch
param_ranges = {
'step_size' : [1e-2, 1e-3, 1e-4, 1e-5, 1e-6],
'decay' : [1.0, 0.95, 0.9],
'epochs' : [20, 50, 100]
}
searcher = RandomSearch(GenerativeModel, param_ranges, L_train, n=5)
Finally we fit to the dev set--i.e. we execute the search, scoring and selecting the best configuration according to the dev set score:
In [3]:
%%time
gen_model, run_stats = searcher.fit(L_dev, L_gold_dev)
run_stats
Now we can apply the best learned model.
Note that this is just a toy example- the above search may not be that interesting (i.e. all models may get perfect scores). But in practice, hyperparameter tuning is extremely important, especially for the end discriminative model (see below).
In [4]:
# Note that in other tutorials, this is often named `train_marginals`!
Y_train = gen_model.marginals(L_train)
GridSearchMPYou can run GridSearch (and any subclasses like RandomSearch) in parallel just by specifying the n_threads keyword argument in the fit method!
Note: When running TensorFlow models in parallel, it is often necessary to throttle the number of threads that each individual model / process tries to use. You can set that by settting n_threads in the initialization of the GridSearch object (here, this kwarg is for the discriminative model class).
Note #2: There is currently an unknown issue with running parallel grid search in a notebook where other models have already been run (see Issue #707), so recommended usage for now is to run in a separate notebook / from the command line.
Hyperparameter search can also be used in the same way for the discriminative model, and in fact this is currently the most popular use of it! Below is example code:
from snorkel.learning import reRNN
param_ranges = {
'lr' : [1e-2, 1e-3, 1e-4, 1e-5, 1e-6],
'dropout' : [0.0, 0.5]
}
model_hyperparams = {
'n_epochs' : 50,
'rebalance' : 0.5,
'print_freq' : 25
}
# We now add a session and probabilistic labels, as well as pass in the candidates
# instead of the label matrix
searcher = RandomSearch(reRNN, param_ranges, train_candidates, Y_train=Y_train, n=5,
model_hyperparams=model_hyperparams)
# We now pass in the development candidates and the gold development labels
lstm, run_stats = searcher.fit(dev_candidates, L_gold_dev)