Recurrent neural network

A recurrent neural network (RNN) is a class of neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior.

Recurrent neural networks must be approached differently from feedforward neural networks, both when analyzing their behavior and training them. Recurrent neural networks can also behave chaotically. Usually, dynamical systems theory is used to model and analyze them. While a feedforward network propagates data linearly from input to output, recurrent networks (RN) also propagate data from later processing stages to earlier stages.

Architectures

Some of the most common recurrent neural network architectures are described here. The Elman and Jordan networks are also known as "simple recurrent networks" (SRN).

Elman network

This variation on the multilayer perceptron was invented by Jeff Elman. A three-layer network is used, with the addition of a set of "context units" in the input layer. There are connections from the middle (hidden) layer to these context units fixed with a weight of one [Neural Networks as Cybernetic Systems 2nd and revised edition, Holk Cruse [http://www.brains-minds-media.org/archive/615/bmm615.pdf] ] . At each time step, the input is propagated in a standard feed-forward fashion, and then a learning rule is applied. The fixed back connections result in the context units always maintaining a copy of the previous values of the hidden units (since they propagate over the connections before the learning rule is applied). Thus the network can maintain a sort of state, allowing it to perform such tasks as sequence-prediction that are beyond the power of a standard multilayer perceptron.

Jordan network

This network architecture is similar to the Elman network. The context units are however fed from the output layer instead of the hidden layer.

MMC network

See [http://www.tech.plym.ac.uk/socce/ncpw9/Kuehn.pdf]

Hopfield network

The Hopfield network is a recurrent neural network in which all connections are symmetric. Invented by John Hopfield in 1982, this network guarantees that its dynamics will converge. If the connections are trained using Hebbian learning then the Hopfield network can perform as robust content-addressable memory, resistant to connection alteration.

Echo state network

The echo state network (ESN) is a recurrent neural network with a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change and be trained. ESN are good to (re)produce temporal patterns.

Long short term memory network

The Long short term memory (LSTM) is an artificial neural net structure that unlike traditional RNNs doesn't have the problem of vanishing gradients. It can therefore use long delays and can handle signals that have a mix of low and high frequency components.

RNN with parametric bias

In this setup the recurrent neural network with parametric bias (RNNPB) is trained to reproduce a sequence with a constant bias input. The network is capable of learning different sequences with different parametric biases. With a trained network is also possible to find the associated parameter for an observed sequence. The sequence is backpropagated through the network to recover the bias which would produce the given sequence.

Continuous-time RNN

(CTRNN)

Hierarchical RNN

Here RNN are sparsely connected together through bottlenecks with the idea to isolate different hierarchical functions to different parts of the composite network. [ [http://www.bdc.brain.riken.go.jp/~rpaine/PaineTaniSAB2004_h.pdf Dynamic Representation of Movement Primitives in an Evolved Recurrent Neural Network ] ] [ [http://www-clmc.usc.edu/publications/P/paine-NN2004.pdf doi:10.1016/j.neunet.2004.08.005 ] ] [http://adb.sagepub.com/cgi/reprint/13/3/211.pdf]

Recurrent Multilayer Perceptron

(RMLP)

Pollack’s Sequential Cascaded Networks

Training

Training in recurrent neural networks is generally very slow.

Backpropagation through time (BPTT)

In this approach the simple recurrent network is unfolded in time for some iterations and then trained through backpropagation as the feed forward network.

Real-time recurrent learning (RTRL)

Unlike BPTT this algorithm is "local in time but not local in space" [ Neural and Adaptive Systems: Fundamentals through Simulation. J.C. Principe, N.R. Euliano, W.C. Lefebvre]

Genetic algorithms

Since RNN learning is very slow, genetic algorithms is a feasible alternative for weight optimization, especially in unstructured networks.

References

*cite book | author=Mandic, D. & Chambers, J. | title=Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability | publisher=Wiley | year=2001
*cite journal | last=Elman | first=J.L. | title=Finding Structure in Time | journal = Cognitive Science | year=1990 | volume=14 | pages=179–211 | doi=10.1016/0364-0213(90)90002-E