Theanets

Korepanova Natalia

Moscow, 2015

The theanets is a deep learning and neural network toolkit.

• Written in Python to iteroperate with numpy and scikit-learn
• theano is used to accelerate computations when possible using GPU
• Simple API for building and training common types of neural network models
• Easy-to-read code
• Quite detailed user guide and documentation
• Under th hood, a fully expressive graph computation framework

Installation

The easiest way is by using pip:

This command will install all the dependdeces of theanets including numpy and theano.

You can also download package from https://github.com/lmjohns3/theanets and run the code from your local copy:

Package overview

Three basin steps in theanets:

1. Define the structure of the model
   • Classification (theanets.Classifier)
   • Regression (theanets.Regressor)
   • Parametric mapping to the input space (theanets.Autoencoder)
   • Recurrent modesl (theanets.reccurent module)
2. Train the model with respect to some task or cost function
3. Use the model for making predictions and/or visualizing the learned features

The usual sceleton of the code:

1. Creating a Model

All network models theanets are instances of the base class Network , which maintain two pieces of information:

1. a list of layers
2. a list of (possibly regularized) loss functions

1.1 Specifying Layers

The easiest and the most common architecture to create is a network with a single “chain” of layers.

Example:

In general, layers argument must be a sequence of values each of which specifies the configuration of a single layer in the model:

Layer Attributes

• size: The number of “neurons” in the layer.
• form: A string specifying the type of layer to use. This defaults to “feedforward”
• name: A string name for the layer. The default names for the first and last layers - 'in' and 'out', the layers in between are assigned the name “hidN” where N is the number of existing layers.
• activation: A string describing the activation function to use for the layer. This defaults to 'relu' - $max(0,z)$.
• etc.

Activation Functions

Some possible values:

• relu: $g(z) = \max(0,z)$
• linear: $g(z) = z$
• logistic, sigmoid: $g(z) = (1 + \exp(-z))^{-1}$
• tanh: $g(z) = \tanh(z)$
• softmax: $g(z) = \exp(z)/\sum_v(\exp(v))$
• norm:z: $g(z) = (z - \bar{z})/\mathbb{E}[(z - \bar{z})^2]$
• etc.

Activation functions can also be composed by concatenating multiple function names togather using a +.

Examples:

If there is a string in the tuple that names a registered layer type (e.g., 'tied', 'rnn', etc.), then this type of layer will be created.

If there is a string in the tuple and it does not name a registered layer type, the string is assumed to name an activation function—for example, 'logistic', 'relu+norm:z', and so on.

If a layer configuration value is a dictionary, its keyword arguments are passed directly to theanets.Layer.build() to construct a new layer instance. The dictionary must contain a size key. It can additionally contain any other keyword arguments that you wish to use when constructing the layer.

sparsity: A float giving the proportion of parameter values in the layer that should be initialized to zero. Nonzero values in the parameters will be drawn from a Gaussian distribution and then an appropriate number of these parameter values will randomly be reset to zero to make the parameter “sparse.”

Custom Layers

To create a custom layer, just create a subclass of theanets.Layer and give it the functionality you want.

An example of a normal feedforward layer but without a bias term:

1.2 Specifying a Loss

All of the predefined models in theanets are created by default with one loss function appropriate for that type of model.

• Autoencoder: MSE between network's output and input $$\mathcal{L}(X,\theta) = \frac{1}{mn}\sum_{i=1}^m\|F_\theta(x_i)-x_i\|_2^2, \quad X \in \mathbb{R}^{m\times n}$$
• Regressor: MSE between true and predicted values of the target $$\mathcal{L}(X, Y,\theta) = \frac{1}{mn}\sum_{i=1}^m\|F_\theta(x_i)-y_i\|_2^2, \quad X \in \mathbb{R}^{m\times n}, Y \in \mathbb{R}^{m\times o}$$
• Classifier: Cross-entropy between the network output and the true target labels $$\mathcal{L}(X, Y,\theta) = \frac{1}{m}\sum_{i=1}^m \sum_{j=1}^{k}\delta_{j, y_i}\log F_\theta(x_i)_j, \quad X \in \mathbb{R}^{m\times n}, Y \in \{1,...,k\}^m$$

For example, to use a mean-absolute error instead of the default mean-squared error for a regression model:

Multiple Loss

Let’s say that you want to optimize a model using both the mean absolute and the mean squared error. You could first create a regular regression model:

You can specify the relative weight of the two losses by manipulating the weight attribute of each loss instance. For instance, if you want the MAE loss to be twice as strong as the MSE loss:

Weighted Targets

Include weighted=True when you create a model. Example:

The training and validation datasets require an additional component: an array of floating-point values with the same shape as the expected output of the model, so that the training and validation datasets would each have three pieces: sample, label, and weight. Each value in the weight array is used as the weight for the corresponding error when computing the loss.

Custom Losses

Create a new theanets.Loss subclass and specify its name when you create your model.

For example, to create a regression model that uses a step function averaged over all of the model inputs with weighted outputs:

1.3 Recurrent Models

Time is an explicit part of the model:

• all the data shapes are one dimension larger than the corresponding shapes for a feedforward network
• the extra dimension represents time
• the extra dimension is located on:
  • the first (0) axis in theanets versions through 0.6
  • the second (1) axis in theanets versions 0.7 and up.

Recurrent versions of the three types of models:

• theanets.recurrent.Autoencoder: takes as input $X \in \mathbb{R}^{m\times t \times n}$ and recreates the same data at the output under squared-error loss
• theanets.recurrent.Regressor: input data $X \in \mathbb{R}^{m\times t \times n}$ and output data $Y \in \mathbb{R}^{m\times t \times o}$, fit the output under squared-error loss
• theanets.recurrent.Classifier: input data $X \in \mathbb{R}^{m\times t \times n}$ and set of integer labels $Y \in \mathbb{Z}^{m \times t}$, the default error is cross-enthropy

1.4 Custom Models

To create a custom model, just define a new subclass of theanets.Network.

For instance, the feedforward autoencoder model is defined basically like this:

2. Training a Model

2.1 Specifying a Trainer

The easiest way train a model with theanets is to invoke the train() method:

Here, a classifier model is being trained using Nesterov’s accelerated gradient, with a learning rate of 0.01 and momentum of 0.9.

Multiple calls to train() are possible and can be used to implement things like custom annealing schedules (e.g., the “newbob” training strategy):

Trainers

In theanets the most of the trainers are provided by downhill package which provides algorithms for minimizing scalar loss functions that are defined using theano.

• sgd: Stochastic gradient descent
• nag: Nesterov’s accelerated gradient
• rprop: Resilient backpropagation
• rmsprop: RMSProp
• adadelta: ADADELTA
• esgd: Equilibrated SGD
• adam: Adam

Also theanets defines a few algorithms which are more specific to neural networks:

• sampler: This trainer sets model parameters directly to samples drawn from the training data. This is a very fast “training” algorithm since all updates take place at once; however, often features derived directly from the training data require further tuning to perform well.
• layerwise: Greedy supervised layerwise pre-training: This trainer applies RMSProp to each layer sequentially.
• pretrain: Greedy unsupervised layerwise pre-training: This trainer applies RMSProp to a tied-weights “shadow” autoencoder using an unlabeled dataset, and then transfers the learned autoencoder weights to the model being trained.

2.2 Providing Data

There are two ways of passing data: using numpy arrays and callables.

Instead of an array of data, you can provide a callable for a Dataset. This callable must take no arguments and must return a list of numpy arrays of the proper shape for your loss.

For example, this code defines a batch() helper that could be used for a loss that needs one input. The callable chooses a random dataset and a random offset for each batch:

2.3 Specifying Regularizers

Regularizers in theanets are specified during training, in calls to Network.train(), or during use, in calls to Network.predict().

Built-in Regularizers

• Decay: $$\mathcal{L}(\cdot)=...+\lambda\|\theta\|_2^2$$
• Sparcity: $$\mathcal{L}(\cdot)=...+\lambda\|\theta\|_1$$
• Hidden representations: $$\mathcal{L}(\cdot)=...+\lambda \sum_{i=2}^{N-1}\|f_i(x)\|_1$$ where $f_i(x)$ - activation function of i-th hidden layer
• Zero-mean Gaussian noise is added to input data or hidden layers
• Multiplicative dropout (binary) noise set some inputs or hidden layers parameters to zero

Custom Regularizers

To create a custom regularizer in theanets, you need to create a custom subclass of the theanets.Regularizer class, and then provide this regularizer when you run your model.

Example:

2.4 Training as Iteration

2.5 Saving Progress

The theanets.Network base class can snapshot your model automatically during training. When you call theanets.Network.train(), you can provide the following keyword arguments:

• save_progress: a string containing a filename where the model should be saved
  • include an empty format string {} to format filename with the UTC Unix timestamp at the moment the model is saved.
• save_every: a numeric value specifying how often the model should be saved during training
  • integer value: the number of training iterations between checkpoints
  • float value: the number of minutes that are allowed to elapse between checkpoints.

Mannualy:

• theanets.Network.save()
• theanets.Network.load()

3. Using a Model

3.1 Predicting New Data

Regardless of the model, you pass to predict() a numpy array containing data examples along the rows, and the method returns an array containing one row of output predictions for each row of input data.

You can also compute the activations of all layer outputs in the network using the theanets.Network.feed_forward() method:

This method returns a dictionary that maps layer output names to their corresponding values for the given input. Like predict(), each output array contains one row for every row of input data.

3.2 Inspecting Parameters

The parameters in each layer of the model are available using theanets.Network.find(). The first query term finds a layer in the network, and the second finds a parameter within that layer.

The find() method returns a theano shared variable. To get a numpy array of the current values of the variable, call get_value() on the result from find(), like so:

Computation Graphs

In theanets it is also possible to create network graphs that have arbitrary, acyclic connections among layers. Creating a nonlinear network graph requires using the inputs keyword argument when creating a layer.

Examples

MNIST Autoencoder (PCA)

Many extremely common dimensionality reduction techniques can be expressed as autoencoders. For instance, Principal Component Analysis (PCA) can be expressed as a model with two tied, linear layers:

MNIST Classifier

In this example, the weights in layer 1 connect the inputs to the first hidden layer; these weights have one column of 784 values for each hidden node in the network, so we can iterate over the transpose and put each column—properly reshaped—into a giant image.