Autoencoder (AE)
A network that compresses the input into a small latent representation (the encoder) and then reconstructs it (the decoder), so it learns a compressed representation without labels.
Overview
| Name | Autoencoder |
|---|---|
| Year introduced | 2006 |
| Researchers | Geoffrey E. Hinton, Ruslan R. Salakhutdinov |
| Original paper | Reducing the dimensionality of data with neural networks DOI |
| Data type | Images, Tabular data, Sequences and time series |
| Domains | Time series forecasting, Computer vision, Medical image analysis |
Problem it solves
The need to reduce dimensionality in a nonlinear way that goes beyond principal component analysis (PCA), and to learn representations from unlabelled data.
Background and evolution
What came before: Multilayer Perceptron, PCA
Core idea
It consists of two functions: $z=f_\phi(x)$ compresses the input and $\hat{x}=g_\theta(z)$ reconstructs it. The bottleneck forces the network to keep only the most important information. Training minimises the reconstruction error.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Input | $x\in\mathbb{R}^{d}$ |
| Encoder | Layers that shrink the dimensions |
| Latent Code | A representation $z\in\mathbb{R}^{k}$ where $k\ll d$ |
| Decoder | Layers that expand the dimensions |
| Loss Function | MSE or BCE for reconstruction |
Equations
1. Reconstruction loss
| $f_\phi$ | The encoder |
| $g_\theta$ | The decoder |
How it works, step by step
- The vector enters the encoder.
- The dimensions shrink gradually down to the bottleneck.
- The decoder expands the latent representation.
- The output is compared with the input and the error computed.
- The weights of both parts are updated together.
Strengths and limitations
Strengths
- Needs no labels.
- Useful for dimensionality reduction, anomaly detection and denoising.
Limitations
- The latent space is irregular, so it is not directly suitable for generation.
- It may learn the identity if the bottleneck is not tight enough.
When to use it, and when not?
Good for
- Anomaly detection in signals or images.
- Initialising representations for downstream tasks.
Not suited for
- Generating realistic new samples (use a VAE or diffusion).
Applications
| Domain | Use |
|---|---|
| Medical image analysis | Detecting anomalous cases in medical images by reconstruction error. |
| Time series forecasting | Fault detection in sensor signals. |
| Computer vision | Image denoising (Denoising AE). |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# Autoencoder
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class AE(nn.Module):
def __init__(self, d=20, k=3):
super().__init__()
self.enc = nn.Sequential(nn.Linear(d, 16), nn.ReLU(), nn.Linear(16, k))
self.dec = nn.Sequential(nn.Linear(k, 16), nn.ReLU(), nn.Linear(16, d))
def forward(self, x):
return self.dec(self.enc(x))
# 3) Prepare the data
# Data lying near a three-dimensional subspace inside 20 dimensions
basis = torch.randn(3, 20)
X = torch.randn(800, 3) @ basis + 0.05 * torch.randn(800, 20)
X_train, X_test = X[:640], X[640:]
y_train, y_test = X_train, X_test # the target is the input itself
model = AE()
# 4) Loss function and optimizer
loss_fn = nn.MSELoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 40, 32
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb)
mse = F.mse_loss(out, y_test).item()
print(f"test MSE = {mse:.4f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb)
print("prediction:", out.squeeze().tolist())
This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test MSE = 0.0028
Trainable parameters in the example: model = 791.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- AI Model Atlas Another section of the site
Related architectures
Variational Autoencoder
Variational Autoencoder (VAE)
A probabilistic generative model with a latent variable: an encoder that estimates an approximate distribution q(z x) and a decoder that represents p(x z), trained toget…
Vector-Quantised Variational Autoencoder
Vector Quantized Variational Autoencoder (VQ-VAE)
An autoencoder that quantises the encoder's output to the nearest vector in a learned codebook, so the representation becomes discrete codes; an autoregressive prior is…
U-Net
U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…
No results match the filters.
Research ideas
- Autoencoders for anomaly detection in MRI images, compared with diffusion models.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the training objective of a plain autoencoder?
Why? The loss compares the output with the input itself.
The other options: Making the latent distribution normal is the goal of a VAE, and the discriminator belongs to GANs.
Why is it hard to generate new samples from a plain AE?
Why? A random point in the latent space may fall in a region the decoder has never seen.
How is an AE used for anomaly detection?
Why? The network is trained on normal data so it reconstructs it well, and it fails relatively on anomalous data.
References
Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647
Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1312.6114
