Deep Learning Neural Networks NLP Advanced Demo content

Recurrent Neural Network

الشبكة العصبية التكرارية

Abbreviation: RNN

Definition

A neural network for processing sequences that keeps a hidden state passed from one time step to the next, so it remembers what came earlier in the sequence.

Definition

A recurrent neural network processes sequences by applying the same weights at each time step and carrying a hidden state that summarises the sequence seen so far.

Mathematical equations

h_t = \tanh\left(W_{xh} x_t + W_{hh} h_{t-1} + b_h\right)

Scientific sources

References and scientific sources

  • Research paper
    Finding Structure in Time Jeffrey L. Elman · 1990 · Cognitive Science 14(2) The simple recurrent neural network (Elman network).
  • Research paper
    Long Short-Term Memory Sepp Hochreiter, Jürgen Schmidhuber · 1997 · Neural Computation 9(8) The LSTM architecture, addressing the vanishing gradient problem in recurrent networks.
  • Book
    Deep Learning Ian Goodfellow, Yoshua Bengio, Aaron Courville · 2016 · MIT Press A comprehensive reference book on deep learning, free to read on the web.

Learn and review

Suggested terms to learn next