BasicTemporal and Sequential Networks

Gated recurrent unit (GRU)

A simplification of LSTM with only two gates (update and reset) and no separate cell state, with comparable performance on many tasks and fewer parameters.

Overview

NameGated recurrent unit
Year introduced2014
ResearchersKyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio
Original paperLearning phrase representations using RNN encoder–decoder for statistical machine translation DOI arXiv
Data typeSequences and time series, Text
DomainsTime series forecasting, Audio and speech, Natural language processing

Problem it solves

LSTM is effective but has many parameters; researchers wanted a simpler unit that keeps the benefit of gates.

Background and evolution

What came before: Long short-term memory, Recurrent neural network

  1. 2014

    Introduced within the RNN Encoder–Decoder model for translation. [ref]

  2. 2014

    Chung et al. compared GRU and LSTM empirically and found similar performance. [ref]

Core idea

The update gate $z_t$ decides how much of the old state is kept versus the new candidate state (linear interpolation). The reset gate $r_t$ decides how much of the previous state is used when computing the candidate.

Example: In a daily temperature series, when values are stable $z_t$ stays close to keeping the state, and on a sudden change it leans towards updating it.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Update gate z_tBalancing old and new
Reset gate r_tEffect of the past on the candidate
Candidate ĥ_tProposed state
Hidden state h_tLinear interpolation

Equations

1. GRU equations (PyTorch form)

$$z_t=\sigma(W_zx_t+U_zh_{t-1}),\; r_t=\sigma(W_rx_t+U_rh_{t-1}),\; \tilde h_t=\tanh\!\left(W x_t+r_t\odot(U h_{t-1})\right),\; h_t=(1-z_t)\odot\tilde h_t+z_t\odot h_{t-1}$$

Symbols
$z_t$Update gate
$r_t$Reset gate
Dimensions
Parameters roughly $3(h(h+d)+2h)$ in PyTorch
Numerical example
$z=0.8$, $h_{t-1}=1$, $\tilde h=-1$ ⇒ $h_t=0.2(-1)+0.8(1)=0.6$.
Relation to the architecture
Note: the original paper swaps $z$ and $1-z$; PyTorch uses the form above, and the meaning is equivalent.

How it works, step by step

  1. The update and reset gates are computed.
  2. The candidate state is computed.
  3. Interpolate between the old state and the candidate.
  4. Repeat for every step.

Strengths and limitations

Strengths

  • Fewer parameters than LSTM (three sets instead of four).
  • Comparable performance on many tasks.

Limitations

  • No separation between memory and output as in LSTM.
  • Does not consistently beat LSTM; the choice is empirical.

When to use it, and when not?

Good for

  • When resources are limited and memory is needed.

Not suited for

  • Very long contexts that require parallelism.

Applications

DomainUse
Time series forecastingForecasting demand, energy and prices.
Natural language processingLanguage modelling and speech recognition (historically).
Medical image analysisAnalysing biosignals such as ECG and EEG.
RoboticsSequential control and motion prediction.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# GRU — next-value prediction
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class GRUForecaster(nn.Module):
    def __init__(self, d_in=1, h=32):
        super().__init__()
        self.gru = nn.GRU(d_in, h, batch_first=True)
        self.out = nn.Linear(h, 1)
    def forward(self, x):
        H, _ = self.gru(x)
        return self.out(H[:, -1])

# 3) Prepare the data
# Predicting the next value of a sum of two waves with random phases and frequencies
def make(n, T=30):
    w1, w2 = torch.rand(n, 1) * 0.3 + 0.1, torch.rand(n, 1) * 0.1 + 0.02
    t = torch.rand(n, 1) * 50 + torch.arange(T + 1).float()
    s = torch.sin(w1 * t) + 0.5 * torch.sin(w2 * t) + 0.03 * torch.randn(n, T + 1)
    return s[:, :T, None], s[:, T:]
X_train, y_train = make(800)
X_test, y_test = make(200)

model = GRUForecaster()

# 4) Loss function and optimizer
loss_fn = nn.MSELoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 15, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    mse = F.mse_loss(out, y_test).item()
    print(f"test MSE = {mse:.4f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("prediction:", out.squeeze().tolist())

This code was run on 2026-11-20 in 4.7 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test MSE = 0.0047

Trainable parameters in the example: model = 3,393.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic1997

Long short-term memory

Long Short-Term Memory (LSTM)

A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…

Temporal and Sequential Networks

Beginner1990

Recurrent neural network

Recurrent Neural Network (RNN)

A network that processes a sequence element by element and carries a hidden state that passes information from previous steps, using the same weights at every step.

Temporal and Sequential Networks

Basic2014

Hybrid CNN–GRU Networks

CNN-GRU

One-dimensional convolutions capture local patterns in the signal and shorten it, then a GRU (often bidirectional) models relationships over time with fewer parameters t…

Temporal and Sequential NetworksHybrid Architectures

Advanced2020

Temporal Graph Networks

Temporal Graph Networks (TGN)

A framework for learning representations on dynamic graphs represented as a stream of timed events, using a memory for each node that is updated by events and an embeddi…

Temporal and Sequential NetworksGraph Neural Networks

Research ideas

  • A systematic comparison of GRU and LSTM on Arabic EEG data with equal hyperparameter tuning.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    How many gates does a GRU have?

  2. Mathematical

    In the PyTorch form, if $z_t=1$, what is $h_t$?

  3. Comparison

    Which claim about GRU vs LSTM is most accurate?

References

  1. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. Proceedings of EMNLP 2014, 1724–1734. https://doi.org/10.3115/v1/D14-1179

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS 2014 Workshop on Deep Learning. https://arxiv.org/abs/1412.3555

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    DOIAutomatically verified via Crossref – 2026-11-20