BeginnerFoundationsGenerative Networks

Autoencoder (AE)

A network that compresses the input into a small latent representation (the encoder) and then reconstructs it (the decoder), so it learns a compressed representation without labels.

Overview

NameAutoencoder
Year introduced2006
ResearchersGeoffrey E. Hinton, Ruslan R. Salakhutdinov
Original paperReducing the dimensionality of data with neural networks DOI
Data typeImages, Tabular data, Sequences and time series
DomainsTime series forecasting, Computer vision, Medical image analysis

Problem it solves

The need to reduce dimensionality in a nonlinear way that goes beyond principal component analysis (PCA), and to learn representations from unlabelled data.

Background and evolution

What came before: Multilayer Perceptron, PCA

  1. 2006

    Hinton and Salakhutdinov showed that deep autoencoders outperform PCA in dimensionality reduction after layer-by-layer pretraining. [ref]

  2. 2014

    VAE turned the autoencoder into a probabilistic generative model. [ref]

Core idea

It consists of two functions: $z=f_\phi(x)$ compresses the input and $\hat{x}=g_\theta(z)$ reconstructs it. The bottleneck forces the network to keep only the most important information. Training minimises the reconstruction error.

Example: A 28×28 image (784 values) is compressed to 16 values and then restored. If the reconstruction error for a new image is very large, it may be an anomaly.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Input$x\in\mathbb{R}^{d}$
EncoderLayers that shrink the dimensions
Latent CodeA representation $z\in\mathbb{R}^{k}$ where $k\ll d$
DecoderLayers that expand the dimensions
Loss FunctionMSE or BCE for reconstruction

Equations

1. Reconstruction loss

$$\mathcal{L}(\phi,\theta)=\frac{1}{N}\sum_{n=1}^{N}\left\|x_n-g_\theta(f_\phi(x_n))\right\|_2^2$$

Symbols
$f_\phi$The encoder
$g_\theta$The decoder
Dimensions
A scalar
Numerical example
$x=(1,0)$ and $\hat{x}=(0.8,0.1)$ ⇒ $\|x-\hat x\|^2=0.04+0.01=0.05$.
Relation to the architecture
It is the only training objective in a plain autoencoder.

How it works, step by step

  1. The vector enters the encoder.
  2. The dimensions shrink gradually down to the bottleneck.
  3. The decoder expands the latent representation.
  4. The output is compared with the input and the error computed.
  5. The weights of both parts are updated together.

Strengths and limitations

Strengths

  • Needs no labels.
  • Useful for dimensionality reduction, anomaly detection and denoising.

Limitations

  • The latent space is irregular, so it is not directly suitable for generation.
  • It may learn the identity if the bottleneck is not tight enough.

When to use it, and when not?

Good for

  • Anomaly detection in signals or images.
  • Initialising representations for downstream tasks.

Not suited for

  • Generating realistic new samples (use a VAE or diffusion).

Applications

DomainUse
Medical image analysisDetecting anomalous cases in medical images by reconstruction error.
Time series forecastingFault detection in sensor signals.
Computer visionImage denoising (Denoising AE).

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Autoencoder
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class AE(nn.Module):
    def __init__(self, d=20, k=3):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, 16), nn.ReLU(), nn.Linear(16, k))
        self.dec = nn.Sequential(nn.Linear(k, 16), nn.ReLU(), nn.Linear(16, d))
    def forward(self, x):
        return self.dec(self.enc(x))

# 3) Prepare the data
# Data lying near a three-dimensional subspace inside 20 dimensions
basis = torch.randn(3, 20)
X = torch.randn(800, 3) @ basis + 0.05 * torch.randn(800, 20)
X_train, X_test = X[:640], X[640:]
y_train, y_test = X_train, X_test        # the target is the input itself
model = AE()

# 4) Loss function and optimizer
loss_fn = nn.MSELoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 40, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    mse = F.mse_loss(out, y_test).item()
    print(f"test MSE = {mse:.4f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("prediction:", out.squeeze().tolist())

This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test MSE = 0.0028

Trainable parameters in the example: model = 791.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Intermediate2014

Variational Autoencoder

Variational Autoencoder (VAE)

A probabilistic generative model with a latent variable: an encoder that estimates an approximate distribution q(z x) and a decoder that represents p(x z), trained toget…

Generative Networks

Advanced2017

Vector-Quantised Variational Autoencoder

Vector Quantized Variational Autoencoder (VQ-VAE)

An autoencoder that quantises the encoder's output to the nearest vector in a learned codebook, so the representation becomes discrete codes; an autoregressive prior is…

Generative Networks

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Research ideas

  • Autoencoders for anomaly detection in MRI images, compared with diffusion models.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the training objective of a plain autoencoder?

  2. Concept

    Why is it hard to generate new samples from a plain AE?

  3. Concept

    How is an AE used for anomaly detection?

References

  1. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647

    DOIAutomatically verified via Crossref – 2026-11-20

  2. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1312.6114

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20