AdvancedPhysics-Informed Neural Networks

Fourier Neural Operator (FNO)

A neural operator that applies a "spectral convolution" in each layer: a Fourier transform of the features, then multiplication of a limited number of low frequencies by learned complex weights, then the inverse transform, with a local linear path; it learns the solutions of families of differential equations and works at different resolutions.

Overview

NameFourier Neural Operator
Year introduced2021
ResearchersZongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar
InstitutionCaltech / Purdue University
Original paperFourier neural operator for parametric partial differential equations arXiv
Data typePhysical fields and functions
DomainsScientific computing, Medical image analysis
Computational cost$O(N\log N\cdot w)+O(N w^2)$ per layer.
Data sizeHundreds to thousands of solutions.

Problem it solves

Learning solution operators with ordinary convolutional networks is tied to the mesh resolution, and the computational cost of classic solvers is high when solving repeatedly for many parameters.

Background and evolution

What came before: Deep Operator Network, Convolutional Neural Network

  1. 2021

    Li and colleagues introduced FNO (ICLR 2021, arXiv 2020). [ref]

  2. 2023

    Kovachki and colleagues set out the general framework for neural operators in JMLR. [ref]

Core idea

An integral operator with a kernel that depends on the difference $\kappa(x-y)$ is a convolution, and convolution in Fourier space is multiplication. So the function is lifted into many channels, then in each layer: FFT → keep the first $k_{max}$ frequencies → multiply them by a learned matrix → IFFT, and a pointwise linear transformation $W$ is added, followed by an activation. Because the weights are defined on frequencies rather than pixels, the model can be evaluated on a finer discretisation.

Example: Training an FNO on solutions of the Burgers equation at 64 points, then evaluating it at 128 points without retraining.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Lifting PLift into channels
Spectral convolution RComplex weights on k_max frequencies
Local linear W1×1 convolution
Nonlinearity σGELU
Projection QTo the solution value

Equations

1. Fourier layer

$$v_{t+1}(x)=\sigma\Big(W\,v_t(x)+\mathcal{F}^{-1}\big(R_\phi\cdot(\mathcal{F}v_t)\big)(x)\Big)$$

Symbols
$\mathcal{F}$Fourier transform
$R_\phi$Complex weights for each retained frequency
$W$Local linear transformation
Dimensions
$R_\phi\in\mathbb{C}^{k_{max}\times w\times w}$
Numerical example
$w=32$, $k_{max}=12$ in one dimension: $12\times32\times32=12{,}288$ complex parameters per layer, regardless of the resolution N.
Relation to the architecture
The parameter count is independent of the discretisation.

2. The integral operator as a convolution

$$(\mathcal{K}v)(x)=\int\kappa(x-y)\,v(y)\,dy=\mathcal{F}^{-1}\big(\mathcal{F}\kappa\cdot\mathcal{F}v\big)(x)$$

Symbols
$\kappa$Integral kernel
$v$The function entering the layer
$\mathcal F,\mathcal F^{-1}$The Fourier transform and its inverse
Dimensions
Function to function
Numerical example
FFT costs $O(N\log N)$ versus $O(N^2)$ for direct integration; at $N=1024$: about $10^4$ versus $10^6$ operations.
Relation to the architecture
It justifies working in frequency space.

How it works, step by step

  1. Generate (condition, solution) pairs with a numerical solver.
  2. Lift the input with the coordinates into channels.
  3. Apply Fourier layers.
  4. Project to the solution.
  5. Train with a relative L2 loss.
  6. Evaluate at other resolutions.

Strengths and limitations

Strengths

  • Very fast after training (according to the authors: orders of magnitude faster than classic solvers in their experiments).
  • Resolution-independent.
  • Effective on periodic and smooth problems.

Limitations

  • Assumes a regular domain (FFT); complex shapes need modifications.
  • Truncating frequencies limits capturing very sharp detail.
  • Needs data from a solver.

When to use it, and when not?

Good for

  • Repeated simulation of a family of equations on a regular grid.
  • Weather and fluid forecasting.

Not suited for

  • Irregular geometries without adaptation.
  • Very little data.

Applications

DomainUse
Scientific computingSimulating fluids, heat, elasticity and inverse problems.
Medical image analysisModelling blood flow and drug diffusion in tissue.
RoboticsDynamical models for control.
Time series forecastingForecasting spatio-temporal fields such as weather.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

The training data is generated inside the example with a spectral solver (RK4) whose convergence we checked by reducing the step; the "identity" baseline is large because viscosity and shocks dissipate energy strongly.

Python (PyTorch)
# A 1D FNO for the Burgers equation, tested at a higher resolution
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class SpectralConv1d(nn.Module):
    """Spectral convolution: FFT → multiply the first `modes` frequencies by learned complex weights → IFFT."""
    def __init__(self, cin, cout, modes):
        super().__init__()
        self.modes = modes
        self.w = nn.Parameter(torch.randn(cin, cout, modes, dtype=torch.cfloat) / (cin * cout))
    def forward(self, x):                                   # x: B×C×N
        xf = torch.fft.rfft(x)
        out = torch.zeros(x.shape[0], self.w.shape[1], xf.shape[-1], dtype=torch.cfloat)
        out[..., :self.modes] = torch.einsum('bin,ion->bon', xf[..., :self.modes], self.w)
        return torch.fft.irfft(out, n=x.shape[-1])

class FNO1d(nn.Module):
    def __init__(self, modes=12, width=32, layers=3):
        super().__init__()
        self.lift = nn.Linear(2, width)                     # (u0(x), x) → channels
        self.spec = nn.ModuleList([SpectralConv1d(width, width, modes) for _ in range(layers)])
        self.w = nn.ModuleList([nn.Conv1d(width, width, 1) for _ in range(layers)])   # the local path W
        self.proj = nn.Sequential(nn.Linear(width, 64), nn.GELU(), nn.Linear(64, 1))
    def forward(self, u0):                                  # u0: B×N (any resolution N)
        N = u0.shape[-1]
        grid = torch.linspace(0, 1, N + 1)[:-1].expand_as(u0)
        v = self.lift(torch.stack([u0, grid], -1)).transpose(1, 2)        # B×width×N
        for k, (s, w) in enumerate(zip(self.spec, self.w)):
            v = s(v) + w(v)
            if k < len(self.spec) - 1: v = F.gelu(v)
        return self.proj(v.transpose(1, 2)).squeeze(-1)

# 3) Prepare the data
# The operator: solving the periodic viscous Burgers equation u_t + u u_x = ν u_xx from u(x,0) to u(x,T)
# The data is generated with a simple spectral solver (pseudo-spectral + RK4) on a fine grid and then sampled
NU, T_END = 0.02, 0.5
def burgers(u0, steps=1000):          # dt = 5e-4 satisfies the stability condition for the viscous term on 128 points
    N = u0.shape[-1]; k = torch.fft.rfftfreq(N, 1 / N) * 2 * torch.pi
    def rhs(u):
        uh = torch.fft.rfft(u)
        ux = torch.fft.irfft(1j * k * uh, n=N); uxx = torch.fft.irfft(-(k ** 2) * uh, n=N)
        return -u * ux + NU * uxx
    u, dt = u0.clone(), T_END / steps
    for _ in range(steps):
        a = rhs(u); b = rhs(u + dt / 2 * a); c = rhs(u + dt / 2 * b); d = rhs(u + dt * c)
        u = u + dt / 6 * (a + 2 * b + 2 * c + d)
    return u
def make(n, N=128):
    x = torch.linspace(0, 1, N + 1)[:-1]
    c = torch.randn(n, 4, 2) / torch.arange(1, 5)[None, :, None]
    u0 = sum(c[:, j, 0, None] * torch.sin(2 * torch.pi * (j + 1) * x) + c[:, j, 1, None] * torch.cos(2 * torch.pi * (j + 1) * x) for j in range(4))
    return u0, burgers(u0)
U0, UT = make(1000)
X_train, y_train = U0[:800, ::2], UT[:800, ::2]          # training at resolution 64
X_test, y_test = U0[800:, ::2], UT[800:, ::2]
X_test_hi, y_test_hi = U0[800:], UT[800:]                # testing at resolution 128, which it was not trained on
model = FNO1d()

# 4) The loss (relative L2, as in the FNO paper) and the optimizer
def rel_l2(p, y): return (torch.norm(p - y, dim=1) / torch.norm(y, dim=1)).mean()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
sched = torch.optim.lr_scheduler.StepLR(opt, 25, 0.5)

# 5) Training loop
EPOCHS, BATCH = 80, 20
for epoch in range(EPOCHS):
    perm, tot = torch.randperm(len(X_train)), 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        loss = rel_l2(model(X_train[idx]), y_train[idx])                     # Forward Pass
        opt.zero_grad(); loss.backward(); opt.step()
        tot += loss.item() * len(idx)
    sched.step()
    if epoch % 20 == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  rel L2={tot / len(X_train):.4f}")

# 6) Evaluation: at the same resolution, then at a higher one (zero-shot super-resolution)
with torch.no_grad():
    print(f"test relative L2 error @64  = {rel_l2(model(X_test), y_test).item():.4f}")
    print(f"test relative L2 error @128 = {rel_l2(model(X_test_hi), y_test_hi).item():.4f}  (no retraining)")
    print(f"identity baseline (u_T ≈ u_0) relative error = {rel_l2(X_test, y_test).item():.4f}")

# 7) Inference on a new initial condition
with torch.no_grad():
    x = torch.linspace(0, 1, 65)[:-1]
    u0 = torch.sin(2 * torch.pi * x)[None]
    print("max |FNO − solver| for u0 = sin(2πx):", round((model(u0) - burgers(u0)).abs().max().item(), 4))

This code was run on 2026-11-20 in 26.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test relative L2 error @64  = 0.0059
test relative L2 error @128 = 0.0059  (no retraining)
identity baseline (u_T ≈ u_0) relative error = 2.8821

Trainable parameters in the example: model = 42,305.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Research2023

Neural Operators

Neural Operators (NO)

A family of architectures that learn operators between function spaces rather than between fixed-size vectors, by stacking pointwise linear layers with integral operator…

Physics-Informed Neural Networks

Advanced2021

Deep Operator Network

Deep Operator Network (DeepONet)

A network that learns an operator mapping a function to a function (such as: condition → solution), through a "branch" network that encodes the input function from its v…

Physics-Informed Neural Networks

Intermediate2019

Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINN)

A network that approximates the solution of a partial differential equation u_θ(x,t) and is trained with a loss that combines the equation's residual (computed by automa…

Physics-Informed Neural Networks

Advanced2020

Scientific machine learning architectures

Scientific Machine Learning Architectures (SciML)

A family of architectures that blends mechanistic models (known equations) with neural networks that learn the unknown parts, trained through differentiable solvers. Not…

Physics-Informed Neural NetworksHybrid Architectures

Research ideas

  • An FNO for short-term forecasting of heat fields in Arab urban environments with satellite data.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Why can an FNO be evaluated at a resolution it was not trained on?

  2. Mathematical

    What happens to frequencies above $k_{max}$ in the spectral path?

  3. Concept

    What is the cost of an FFT of length N?

References

  1. Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2021). Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2010.08895

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2023). Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89), 1–97. https://arxiv.org/abs/2108.08481

    arXivPublisherAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Lu, L., Jin, P., Pang, G., Zhang, Z., & Karniadakis, G. E. (2021). Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218–229. https://doi.org/10.1038/s42256-021-00302-5

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  4. Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422–440. https://doi.org/10.1038/s42254-021-00314-5

    DOIAutomatically verified via Crossref – 2026-11-20