BasicTemporal and Sequential Networks

Bidirectional LSTM (BiLSTM)

Two recurrent layers, one reading the sequence from the start and the other from the end; their states are concatenated so that every position sees both its preceding and following context.

Overview

NameBidirectional LSTM
Year introduced1997
ResearchersMike Schuster, Kuldip K. Paliwal (bidirectional RNN); Alex Graves, Jürgen Schmidhuber (BiLSTM, 2005)
Original paperBidirectional recurrent neural networks DOI
Data typeSequences and time series, Text
DomainsAudio and speech, Medical image analysis, Natural language processing

Problem it solves

In tasks such as word tagging or phoneme classification, the meaning of an element also depends on what follows it, and a unidirectional network cannot see the future.

Background and evolution

What came before: Recurrent neural network, Long short-term memory

  1. 1997

    Schuster and Paliwal introduced bidirectional recurrent networks. [ref]

  2. 2005

    Graves and Schmidhuber applied BiLSTM to frame-level phoneme classification. [ref]

Core idea

A forward state $\overrightarrow{h}_t$ is computed from the start up to t, and a backward state $\overleftarrow{h}_t$ from the end down to t, and the two are concatenated. It is unsuitable for live forecasting because it needs the whole sequence.

Example: In "I read the book that I bought yesterday", tagging the word "book" benefits from the words after it as well as before it.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Forward LSTMLeft to right
Backward LSTMRight to left
ConcatenationConcatenating the two states
Output layerPer position or per sequence

Equations

1. Concatenating both directions

$$h_t=\left[\overrightarrow{h}_t\,;\,\overleftarrow{h}_t\right],\quad \overrightarrow{h}_t=\mathrm{LSTM}_f(x_t,\overrightarrow{h}_{t-1}),\quad \overleftarrow{h}_t=\mathrm{LSTM}_b(x_t,\overleftarrow{h}_{t+1})$$

Symbols
$\overrightarrow{h}_t$Forward state
$\overleftarrow{h}_t$Backward state
Dimensions
$h_t\in\mathbb{R}^{2h}$
Numerical example
With $h=32$, each position's output is 64.
Relation to the architecture
Combines both contexts for every element.

How it works, step by step

  1. The forward network passes over the sequence.
  2. The backward network passes in reverse.
  3. The two states are concatenated for every position.
  4. The classifier is applied to every position or to a pooled summary.

Strengths and limitations

Strengths

  • Full context for every position.
  • Clear gains in tagging and speech recognition tasks.

Limitations

  • Unsuitable for real-time prediction or step-by-step generation.
  • Twice the compute of a single LSTM.

When to use it, and when not?

Good for

  • Word tagging and entity recognition.
  • Classifying complete recorded signals.

Not suited for

  • Real-time forecasting of the future.

Applications

DomainUse
Natural language processingNamed-entity recognition and part-of-speech tagging.
Audio and speechFrame-level phoneme classification.
Medical image analysisClassifying recorded ECG beats.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Bidirectional LSTM — per-position tagging
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class BiLSTMTagger(nn.Module):
    def __init__(self, d_in=1, h=32, n_tags=2):
        super().__init__()
        self.lstm = nn.LSTM(d_in, h, batch_first=True, bidirectional=True)
        self.out = nn.Linear(2 * h, n_tags)
    def forward(self, x):
        H, _ = self.lstm(x)                 # B×T×2h
        return self.out(H).transpose(1, 2)  # B×tags×T (CrossEntropy shape for sequences)

# 3) Prepare the data
# Correct tag for each position: is the next value larger than the current one? (requires seeing the future)
X = torch.randn(800, 20, 1)
nxt = torch.cat([X[:, 1:, 0], X[:, -1:, 0]], 1)
Y = (nxt > X[:, :, 0]).long()
X_train, y_train, X_test, y_test = X[:640], Y[:640], X[640:], Y[640:]
model = BiLSTMTagger()

# 4) The loss and the optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)

# 5) Training loop
for epoch in range(15):
    model.train()
    for i in range(0, len(X_train), 64):
        loss = loss_fn(model(X_train[i:i + 64]), y_train[i:i + 64])
        opt.zero_grad(); loss.backward(); opt.step()
    if epoch % 5 == 0: print(f"epoch {epoch}  loss={loss.item():.4f}")

# 6) Evaluation (position-level accuracy)
model.eval()
with torch.no_grad():
    acc = (model(X_test).argmax(1) == y_test).float().mean().item()
print(f"test accuracy = {acc:.3f}")

# 7) Inference: tags of the first 10 positions
with torch.no_grad():
    print("tags:", model(X_test[:1]).argmax(1)[0, :10].tolist())

This code was run on 2026-11-20 in 3.5 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 0.989

Trainable parameters in the example: model = 9,090.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic1997

Long short-term memory

Long Short-Term Memory (LSTM)

A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…

Temporal and Sequential Networks

Basic2014

Gated recurrent unit

Gated Recurrent Unit (GRU)

A simplification of LSTM with only two gates (update and reset) and no separate cell state, with comparable performance on many tasks and fewer parameters.

Temporal and Sequential Networks

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Research ideas

  • BiLSTM-CRF vs small Transformer models for entity recognition in Arabic medical texts.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Why is BiLSTM unsuitable for real-time forecasting of the future?

  2. Concept

    What is the per-position output size of a BiLSTM with hidden state h=128 per direction?

  3. Concept

    In which task does BiLSTM usually beat a unidirectional LSTM?

References

  1. Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11), 2673–2681. https://doi.org/10.1109/78.650093

    DOIAutomatically verified via Crossref – 2026-11-20

  2. Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5–6), 602–610. https://doi.org/10.1016/j.neunet.2005.06.042

    DOIAutomatically verified via Crossref – 2026-11-20

  3. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    DOIAutomatically verified via Crossref – 2026-11-20