Fourier Neural Operator (FNO)
A neural operator that applies a "spectral convolution" in each layer: a Fourier transform of the features, then multiplication of a limited number of low frequencies by learned complex weights, then the inverse transform, with a local linear path; it learns the solutions of families of differential equations and works at different resolutions.
Overview
| Name | Fourier Neural Operator |
|---|---|
| Year introduced | 2021 |
| Researchers | Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar |
| Institution | Caltech / Purdue University |
| Original paper | Fourier neural operator for parametric partial differential equations arXiv |
| Data type | Physical fields and functions |
| Domains | Scientific computing, Medical image analysis |
| Computational cost | $O(N\log N\cdot w)+O(N w^2)$ per layer. |
| Data size | Hundreds to thousands of solutions. |
Problem it solves
Learning solution operators with ordinary convolutional networks is tied to the mesh resolution, and the computational cost of classic solvers is high when solving repeatedly for many parameters.
Background and evolution
What came before: Deep Operator Network, Convolutional Neural Network
Core idea
An integral operator with a kernel that depends on the difference $\kappa(x-y)$ is a convolution, and convolution in Fourier space is multiplication. So the function is lifted into many channels, then in each layer: FFT → keep the first $k_{max}$ frequencies → multiply them by a learned matrix → IFFT, and a pointwise linear transformation $W$ is added, followed by an activation. Because the weights are defined on frequencies rather than pixels, the model can be evaluated on a finer discretisation.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Lifting P | Lift into channels |
| Spectral convolution R | Complex weights on k_max frequencies |
| Local linear W | 1×1 convolution |
| Nonlinearity σ | GELU |
| Projection Q | To the solution value |
Equations
1. Fourier layer
| $\mathcal{F}$ | Fourier transform |
| $R_\phi$ | Complex weights for each retained frequency |
| $W$ | Local linear transformation |
2. The integral operator as a convolution
| $\kappa$ | Integral kernel |
| $v$ | The function entering the layer |
| $\mathcal F,\mathcal F^{-1}$ | The Fourier transform and its inverse |
How it works, step by step
- Generate (condition, solution) pairs with a numerical solver.
- Lift the input with the coordinates into channels.
- Apply Fourier layers.
- Project to the solution.
- Train with a relative L2 loss.
- Evaluate at other resolutions.
Strengths and limitations
Strengths
- Very fast after training (according to the authors: orders of magnitude faster than classic solvers in their experiments).
- Resolution-independent.
- Effective on periodic and smooth problems.
Limitations
- Assumes a regular domain (FFT); complex shapes need modifications.
- Truncating frequencies limits capturing very sharp detail.
- Needs data from a solver.
When to use it, and when not?
Good for
- Repeated simulation of a family of equations on a regular grid.
- Weather and fluid forecasting.
Not suited for
- Irregular geometries without adaptation.
- Very little data.
Applications
| Domain | Use |
|---|---|
| Scientific computing | Simulating fluids, heat, elasticity and inverse problems. |
| Medical image analysis | Modelling blood flow and drug diffusion in tissue. |
| Robotics | Dynamical models for control. |
| Time series forecasting | Forecasting spatio-temporal fields such as weather. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
The training data is generated inside the example with a spectral solver (RK4) whose convergence we checked by reducing the step; the "identity" baseline is large because viscosity and shocks dissipate energy strongly.
# A 1D FNO for the Burgers equation, tested at a higher resolution
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class SpectralConv1d(nn.Module):
"""Spectral convolution: FFT → multiply the first `modes` frequencies by learned complex weights → IFFT."""
def __init__(self, cin, cout, modes):
super().__init__()
self.modes = modes
self.w = nn.Parameter(torch.randn(cin, cout, modes, dtype=torch.cfloat) / (cin * cout))
def forward(self, x): # x: B×C×N
xf = torch.fft.rfft(x)
out = torch.zeros(x.shape[0], self.w.shape[1], xf.shape[-1], dtype=torch.cfloat)
out[..., :self.modes] = torch.einsum('bin,ion->bon', xf[..., :self.modes], self.w)
return torch.fft.irfft(out, n=x.shape[-1])
class FNO1d(nn.Module):
def __init__(self, modes=12, width=32, layers=3):
super().__init__()
self.lift = nn.Linear(2, width) # (u0(x), x) → channels
self.spec = nn.ModuleList([SpectralConv1d(width, width, modes) for _ in range(layers)])
self.w = nn.ModuleList([nn.Conv1d(width, width, 1) for _ in range(layers)]) # the local path W
self.proj = nn.Sequential(nn.Linear(width, 64), nn.GELU(), nn.Linear(64, 1))
def forward(self, u0): # u0: B×N (any resolution N)
N = u0.shape[-1]
grid = torch.linspace(0, 1, N + 1)[:-1].expand_as(u0)
v = self.lift(torch.stack([u0, grid], -1)).transpose(1, 2) # B×width×N
for k, (s, w) in enumerate(zip(self.spec, self.w)):
v = s(v) + w(v)
if k < len(self.spec) - 1: v = F.gelu(v)
return self.proj(v.transpose(1, 2)).squeeze(-1)
# 3) Prepare the data
# The operator: solving the periodic viscous Burgers equation u_t + u u_x = ν u_xx from u(x,0) to u(x,T)
# The data is generated with a simple spectral solver (pseudo-spectral + RK4) on a fine grid and then sampled
NU, T_END = 0.02, 0.5
def burgers(u0, steps=1000): # dt = 5e-4 satisfies the stability condition for the viscous term on 128 points
N = u0.shape[-1]; k = torch.fft.rfftfreq(N, 1 / N) * 2 * torch.pi
def rhs(u):
uh = torch.fft.rfft(u)
ux = torch.fft.irfft(1j * k * uh, n=N); uxx = torch.fft.irfft(-(k ** 2) * uh, n=N)
return -u * ux + NU * uxx
u, dt = u0.clone(), T_END / steps
for _ in range(steps):
a = rhs(u); b = rhs(u + dt / 2 * a); c = rhs(u + dt / 2 * b); d = rhs(u + dt * c)
u = u + dt / 6 * (a + 2 * b + 2 * c + d)
return u
def make(n, N=128):
x = torch.linspace(0, 1, N + 1)[:-1]
c = torch.randn(n, 4, 2) / torch.arange(1, 5)[None, :, None]
u0 = sum(c[:, j, 0, None] * torch.sin(2 * torch.pi * (j + 1) * x) + c[:, j, 1, None] * torch.cos(2 * torch.pi * (j + 1) * x) for j in range(4))
return u0, burgers(u0)
U0, UT = make(1000)
X_train, y_train = U0[:800, ::2], UT[:800, ::2] # training at resolution 64
X_test, y_test = U0[800:, ::2], UT[800:, ::2]
X_test_hi, y_test_hi = U0[800:], UT[800:] # testing at resolution 128, which it was not trained on
model = FNO1d()
# 4) The loss (relative L2, as in the FNO paper) and the optimizer
def rel_l2(p, y): return (torch.norm(p - y, dim=1) / torch.norm(y, dim=1)).mean()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
sched = torch.optim.lr_scheduler.StepLR(opt, 25, 0.5)
# 5) Training loop
EPOCHS, BATCH = 80, 20
for epoch in range(EPOCHS):
perm, tot = torch.randperm(len(X_train)), 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
loss = rel_l2(model(X_train[idx]), y_train[idx]) # Forward Pass
opt.zero_grad(); loss.backward(); opt.step()
tot += loss.item() * len(idx)
sched.step()
if epoch % 20 == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} rel L2={tot / len(X_train):.4f}")
# 6) Evaluation: at the same resolution, then at a higher one (zero-shot super-resolution)
with torch.no_grad():
print(f"test relative L2 error @64 = {rel_l2(model(X_test), y_test).item():.4f}")
print(f"test relative L2 error @128 = {rel_l2(model(X_test_hi), y_test_hi).item():.4f} (no retraining)")
print(f"identity baseline (u_T ≈ u_0) relative error = {rel_l2(X_test, y_test).item():.4f}")
# 7) Inference on a new initial condition
with torch.no_grad():
x = torch.linspace(0, 1, 65)[:-1]
u0 = torch.sin(2 * torch.pi * x)[None]
print("max |FNO − solver| for u0 = sin(2πx):", round((model(u0) - burgers(u0)).abs().max().item(), 4))
This code was run on 2026-11-20 in 26.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test relative L2 error @64 = 0.0059 test relative L2 error @128 = 0.0059 (no retraining) identity baseline (u_T ≈ u_0) relative error = 2.8821
Trainable parameters in the example: model = 42,305.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Neural Operators
Neural Operators (NO)
A family of architectures that learn operators between function spaces rather than between fixed-size vectors, by stacking pointwise linear layers with integral operator…
Deep Operator Network
Deep Operator Network (DeepONet)
A network that learns an operator mapping a function to a function (such as: condition → solution), through a "branch" network that encodes the input function from its v…
Physics-Informed Neural Networks
Physics-Informed Neural Networks (PINN)
A network that approximates the solution of a partial differential equation u_θ(x,t) and is trained with a loss that combines the equation's residual (computed by automa…
Scientific machine learning architectures
Scientific Machine Learning Architectures (SciML)
A family of architectures that blends mechanistic models (known equations) with neural networks that learn the unknown parts, trained through differentiable solvers. Not…
No results match the filters.
Research ideas
- An FNO for short-term forecasting of heat fields in Arab urban environments with satellite data.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
Why can an FNO be evaluated at a resolution it was not trained on?
Why? The same low frequencies exist at any sufficient resolution.
What happens to frequencies above $k_{max}$ in the spectral path?
Why? The local path W helps recover some detail.
What is the cost of an FFT of length N?
Why? The basis of FNO's efficiency.
References
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2021). Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2010.08895
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2023). Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89), 1–97. https://arxiv.org/abs/2108.08481
Lu, L., Jin, P., Pang, G., Zhang, Z., & Karniadakis, G. E. (2021). Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218–229. https://doi.org/10.1038/s42256-021-00302-5
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422–440. https://doi.org/10.1038/s42254-021-00314-5
