Morten Hjorth-Jensen, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
Date: Dec 17, 2019
Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
Till now our focus has been, including convolutional neural networks as well, on feedforward neural networks. The output or the activations flow only in one direction, from the input layer to the output layer.
A recurrent neural network (RNN) looks very much like a feedforward neural network, except that it also has connections pointing backward.
RNNs are used to analyze time series data such as stock prices, and tell you when to buy or sell. In autonomous driving systems, they can anticipate car trajectories and help avoid accidents. More generally, they can work on sequences of arbitrary lengths, rather than on fixed-sized inputs like all the nets we have discussed so far. For example, they can take sentences, documents, or audio samples as input, making them extremely useful for natural language processing systems such as automatic translation and speech-to-text.
The text here is under development. Planned finished mid Jan 2020.
The figure here displays a simple example of an RNN, with inputs $x_t$ at a given time $t$ and outputs $y_t$. Introducing time as a variable offers an intutitive way of understanding these networks. In addition to the inputs $x_t$, the layer at a time $t$ receives also as input the output from the previous layer $t-1$, that is $y_{t1}$.
This means also that we need to have weights that link both the inputs $x_t$ to the outputs $y_t$ as well as weights that link the output from the previous time $y_{t-1}$ and $y_t$. The figure here shows an example of a simple RNN.
In our discussions of ordinary differential equations and partial differential equations using neural networks. Here we will discuss how we can solve say ordinary differential equations and eigenvalue problems using RNNs. Eigenvalue problems can be solved using RNNs by rewriting such a problems as a non-linear differential equation.
Instead of starting with a well-known ordinary differential equation, we start directly with an eigenvaule problem.
Discussions about dynamic unrolling through time. discuss memory cells, input and output