AdvancedPhysics-Informed Neural NetworksHybrid Architectures

Scientific machine learning architectures (SciML)

A family of architectures that blends mechanistic models (known equations) with neural networks that learn the unknown parts, trained through differentiable solvers. Notable examples are Universal Differential Equations (UDE), Neural ODEs and differentiable simulators.

Overview

NameScientific machine learning architectures
Year introduced2020
ResearchersReference example: Christopher Rackauckas et al. (Universal Differential Equations)
InstitutionMIT
Original paperUniversal differential equations for scientific machine learning arXiv
Data typeSequences and time series, Physical fields and functions
DomainsTime series forecasting, Scientific computing, Medical image analysis
Computational costScales with the number of solver steps.
Data sizeSmall to medium (time trajectories).

Problem it solves

Mechanistic models miss poorly understood phenomena, and purely neural models need a lot of data and are not interpretable. Scientists want to keep what they know and learn only what is missing.

Background and evolution

What came before: Neural ordinary differential equations, Physics-Informed Neural Networks, Multilayer Perceptron

  1. 2018

    Neural ODE: differentiating through solvers. [ref]

  2. 2020

    UDE: a framework that places networks inside known equations to learn the unknown terms. [ref]

  3. 2021

    Review of physics-informed machine learning. [ref]

Core idea

$\dot z=f_{known}(z;p)+NN_\theta(z)$. The system is simulated with an ODE solver, trajectories are compared with measurements, and the error is differentiated through the solver steps to update $\theta$. After training, what the network learned can be inspected and a symbolic formula extracted (symbolic regression), making the model interpretable.

Example: An epidemic model: the SIR equations are known, but the infection rate changes with behaviour; a small network learns this term from data.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Known mechanistic termsKnown equations
Neural correctionNetwork for the unknown terms
Differentiable ODE solverRK4 or adjoint
Trajectory loss
Symbolic recovery (optional)Formula extraction

Equations

1. Universal differential equation

$$\frac{dz}{dt}=f_{known}(z,t;p)+NN_\theta(z,t)$$

Symbols
$p$Known parameters
$NN_\theta$Learned part
Dimensions
$z\in\mathbb{R}^{d}$
Numerical example
Lotka–Volterra: $\dot x=1.3x+NN_1$, and the true missing term is $-0.9xy$; at $x=1.5$, $y=1$: $-1.35$.
Relation to the architecture
The network learns only what the model is missing.

2. Trajectory loss

$$\mathcal{L}(\theta)=\frac{1}{T}\sum_{k=1}^{T}\big\lVert\mathrm{ODESolve}(f,z_0,t_k)-z^{obs}(t_k)\big\rVert^2$$

Symbols
$\mathrm{ODESolve}$Differentiable solver
Dimensions
A scalar
Numerical example
An error of $0.02$ at two of three points and zero at the third: $\mathcal{L}=(0.0004+0.0004)/3\approx0.00027$.
Relation to the architecture
The gradient passes through every solver step.

How it works, step by step

  1. Write down what you know of the equations.
  2. Add a network for the unknown terms.
  3. Simulate with a differentiable solver.
  4. Train on trajectories (short, then longer).
  5. Inspect the learned term and extract a formula if possible.

Strengths and limitations

Strengths

  • Less data thanks to prior knowledge.
  • Higher interpretability.
  • Better generalization to new conditions.

Limitations

  • Differentiating through the solver is costly and may be unstable for long trajectories.
  • Identifiability is not guaranteed.

When to use it, and when not?

Good for

  • Systems with partial mechanistic models: biology, epidemics, engineering.

Not suited for

  • No reliable mechanistic knowledge at all.

Applications

DomainUse
Scientific computingModelling biological and chemical systems.
Medical image analysisHybrid pharmacological (PK/PD) models.
Time series forecastingForecasting dynamical systems.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Universal Differential Equation: learning the unknown interaction terms in Lotka–Volterra
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class UDE(nn.Module):
    """Universal differential equation: the known parts of the physical model + a network that learns the unknown terms.
    Lotka–Volterra: dx/dt = a x − [?] , dy/dt = −d y + [?] (interaction terms unknown)."""
    def __init__(self, a=1.3, d=1.8):
        super().__init__()
        self.a, self.d = a, d
        self.nn = nn.Sequential(nn.Linear(2, 32), nn.Tanh(), nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 2))
    def forward(self, z):
        x, y = z[:, :1], z[:, 1:]
        known = torch.cat([self.a * x, -self.d * y], 1)
        return known + self.nn(z)

def rk4_rollout(f, z0, dt, steps):
    zs, z = [z0], z0
    for _ in range(steps):
        a = f(z); b = f(z + dt / 2 * a); c = f(z + dt / 2 * b); d = f(z + dt * c)
        z = z + dt / 6 * (a + 2 * b + 2 * c + d); zs.append(z)
    return torch.stack(zs, 1)                                          # B×(steps+1)×2

# 3) Prepare the data
# The true system (partly unknown to the model): terms −β x y and +γ x y
A_, B_, G_, D_ = 1.3, 0.9, 0.8, 1.8
true_f = lambda z: torch.stack([A_ * z[:, 0] - B_ * z[:, 0] * z[:, 1], -D_ * z[:, 1] + G_ * z[:, 0] * z[:, 1]], 1)
dt, STEPS = 0.05, 60
with torch.no_grad():
    Z0 = torch.rand(64, 2) * 2 + 0.5
    traj = rk4_rollout(true_f, Z0, dt, STEPS)
    traj_obs = traj * (1 + 0.01 * torch.randn_like(traj))                 # 1% measurement noise
model = UDE()

# 4) Loss: difference between the model-simulated trajectory and the measured one (differentiating through the solver itself)
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop: short trajectories first, then longer (simplified multiple shooting)
for epoch in range(600):
    H = 10 if epoch < 200 else 30
    start = torch.randint(0, STEPS - H, (1,)).item()
    pred = rk4_rollout(model, traj_obs[:, start], dt, H)                  # Forward Pass through the solver
    loss = F.mse_loss(pred, traj_obs[:, start:start + H + 1])
    opt.zero_grad(); loss.backward(); opt.step()
    if epoch % 150 == 0 or epoch == 599:
        print(f"epoch {epoch:3d}  trajectory MSE={loss.item():.5f}")

# 6) Evaluation: did the network recover the interaction terms −βxy and +γxy? And prediction for new initial conditions
with torch.no_grad():
    zz = torch.rand(500, 2) * 2 + 0.5
    learned = model.nn(zz)
    true_missing = torch.stack([-B_ * zz[:, 0] * zz[:, 1], G_ * zz[:, 0] * zz[:, 1]], 1)
    print(f"relative error of recovered interaction terms = {(torch.norm(learned - true_missing) / torch.norm(true_missing)).item():.4f}")
    z0 = torch.rand(16, 2) * 2 + 0.5
    err = (rk4_rollout(model, z0, dt, STEPS) - rk4_rollout(true_f, z0, dt, STEPS)).abs().mean().item()
    print(f"mean abs trajectory error on new initial conditions = {err:.4f}")

# 7) Inference: estimate β from the network (first term ≈ −β x y)
with torch.no_grad():
    x, y = 1.5, 1.0
    print(f"estimated beta ≈ {(-model.nn(torch.tensor([[x, y]]))[0, 0] / (x * y)).item():.3f} (true {B_})")

This code was run on 2026-11-20 in 15.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

relative error of recovered interaction terms = 0.0343
mean abs trajectory error on new initial conditions = 0.0431

Trainable parameters in the example: model = 1,218.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2018

Neural ordinary differential equations

Neural Ordinary Differential Equations (Neural ODE)

The hidden state is defined by a continuous differential equation whose derivative is given by a neural network, and the output is computed with a numerical solver, so t…

Temporal and Sequential NetworksPhysics-Informed Neural Networks

Intermediate2019

Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINN)

A network that approximates the solution of a partial differential equation u_θ(x,t) and is trained with a loss that combines the equation's residual (computed by automa…

Physics-Informed Neural Networks

Advanced2021

Physics-informed deep learning

Physics-Informed Deep Learning (PIML)

A general approach to embedding physical knowledge in deep learning in three ways: through data (observational bias), through the architecture (inductive bias, such as H…

Physics-Informed Neural Networks

Advanced2021

Deep Operator Network

Deep Operator Network (DeepONet)

A network that learns an operator mapping a function to a function (such as: condition → solution), through a "branch" network that encodes the input function from its v…

Physics-Informed Neural Networks

Advanced2021

Fourier Neural Operator

Fourier Neural Operator (FNO)

A neural operator that applies a "spectral convolution" in each layer: a Fourier transform of the features, then multiplication of a limited number of low frequencies by…

Physics-Informed Neural Networks

Advanced2021

Liquid Neural Networks

Liquid Neural Networks (LNN)

Continuous-time recurrent networks in which each neuron's time constant changes with the input (Liquid Time-Constant), so their dynamics adapt to the data with a small n…

Temporal and Sequential NetworksAdvanced Neural Architectures

Research ideas

  • UDE for modelling blood-glucose response to insulin in type 1 diabetes patients.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the core idea of universal differential equations?

  2. Concept

    How does the gradient flow in UDE training?

  3. Concept

    What is the benefit of extracting a symbolic formula for the learned term?

References

  1. Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., Ramadhan, A., & Edelman, A. (2020). Universal differential equations for scientific machine learning. arXiv preprint. https://arxiv.org/abs/2001.04385

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems (NeurIPS) 31. https://arxiv.org/abs/1806.07366

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422–440. https://doi.org/10.1038/s42254-021-00314-5

    DOIAutomatically verified via Crossref – 2026-11-20

  4. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707. https://doi.org/10.1016/j.jcp.2018.10.045

    DOIAutomatically verified via Crossref – 2026-11-20