Scientific machine learning architectures (SciML)
A family of architectures that blends mechanistic models (known equations) with neural networks that learn the unknown parts, trained through differentiable solvers. Notable examples are Universal Differential Equations (UDE), Neural ODEs and differentiable simulators.
Overview
| Name | Scientific machine learning architectures |
|---|---|
| Year introduced | 2020 |
| Researchers | Reference example: Christopher Rackauckas et al. (Universal Differential Equations) |
| Institution | MIT |
| Original paper | Universal differential equations for scientific machine learning arXiv |
| Data type | Sequences and time series, Physical fields and functions |
| Domains | Time series forecasting, Scientific computing, Medical image analysis |
| Computational cost | Scales with the number of solver steps. |
| Data size | Small to medium (time trajectories). |
Problem it solves
Mechanistic models miss poorly understood phenomena, and purely neural models need a lot of data and are not interpretable. Scientists want to keep what they know and learn only what is missing.
Background and evolution
What came before: Neural ordinary differential equations, Physics-Informed Neural Networks, Multilayer Perceptron
Core idea
$\dot z=f_{known}(z;p)+NN_\theta(z)$. The system is simulated with an ODE solver, trajectories are compared with measurements, and the error is differentiated through the solver steps to update $\theta$. After training, what the network learned can be inspected and a symbolic formula extracted (symbolic regression), making the model interpretable.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Known mechanistic terms | Known equations |
| Neural correction | Network for the unknown terms |
| Differentiable ODE solver | RK4 or adjoint |
| Trajectory loss | |
| Symbolic recovery (optional) | Formula extraction |
Equations
1. Universal differential equation
| $p$ | Known parameters |
| $NN_\theta$ | Learned part |
2. Trajectory loss
| $\mathrm{ODESolve}$ | Differentiable solver |
How it works, step by step
- Write down what you know of the equations.
- Add a network for the unknown terms.
- Simulate with a differentiable solver.
- Train on trajectories (short, then longer).
- Inspect the learned term and extract a formula if possible.
Strengths and limitations
Strengths
- Less data thanks to prior knowledge.
- Higher interpretability.
- Better generalization to new conditions.
Limitations
- Differentiating through the solver is costly and may be unstable for long trajectories.
- Identifiability is not guaranteed.
When to use it, and when not?
Good for
- Systems with partial mechanistic models: biology, epidemics, engineering.
Not suited for
- No reliable mechanistic knowledge at all.
Applications
| Domain | Use |
|---|---|
| Scientific computing | Modelling biological and chemical systems. |
| Medical image analysis | Hybrid pharmacological (PK/PD) models. |
| Time series forecasting | Forecasting dynamical systems. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# Universal Differential Equation: learning the unknown interaction terms in Lotka–Volterra
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class UDE(nn.Module):
"""Universal differential equation: the known parts of the physical model + a network that learns the unknown terms.
Lotka–Volterra: dx/dt = a x − [?] , dy/dt = −d y + [?] (interaction terms unknown)."""
def __init__(self, a=1.3, d=1.8):
super().__init__()
self.a, self.d = a, d
self.nn = nn.Sequential(nn.Linear(2, 32), nn.Tanh(), nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 2))
def forward(self, z):
x, y = z[:, :1], z[:, 1:]
known = torch.cat([self.a * x, -self.d * y], 1)
return known + self.nn(z)
def rk4_rollout(f, z0, dt, steps):
zs, z = [z0], z0
for _ in range(steps):
a = f(z); b = f(z + dt / 2 * a); c = f(z + dt / 2 * b); d = f(z + dt * c)
z = z + dt / 6 * (a + 2 * b + 2 * c + d); zs.append(z)
return torch.stack(zs, 1) # B×(steps+1)×2
# 3) Prepare the data
# The true system (partly unknown to the model): terms −β x y and +γ x y
A_, B_, G_, D_ = 1.3, 0.9, 0.8, 1.8
true_f = lambda z: torch.stack([A_ * z[:, 0] - B_ * z[:, 0] * z[:, 1], -D_ * z[:, 1] + G_ * z[:, 0] * z[:, 1]], 1)
dt, STEPS = 0.05, 60
with torch.no_grad():
Z0 = torch.rand(64, 2) * 2 + 0.5
traj = rk4_rollout(true_f, Z0, dt, STEPS)
traj_obs = traj * (1 + 0.01 * torch.randn_like(traj)) # 1% measurement noise
model = UDE()
# 4) Loss: difference between the model-simulated trajectory and the measured one (differentiating through the solver itself)
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop: short trajectories first, then longer (simplified multiple shooting)
for epoch in range(600):
H = 10 if epoch < 200 else 30
start = torch.randint(0, STEPS - H, (1,)).item()
pred = rk4_rollout(model, traj_obs[:, start], dt, H) # Forward Pass through the solver
loss = F.mse_loss(pred, traj_obs[:, start:start + H + 1])
opt.zero_grad(); loss.backward(); opt.step()
if epoch % 150 == 0 or epoch == 599:
print(f"epoch {epoch:3d} trajectory MSE={loss.item():.5f}")
# 6) Evaluation: did the network recover the interaction terms −βxy and +γxy? And prediction for new initial conditions
with torch.no_grad():
zz = torch.rand(500, 2) * 2 + 0.5
learned = model.nn(zz)
true_missing = torch.stack([-B_ * zz[:, 0] * zz[:, 1], G_ * zz[:, 0] * zz[:, 1]], 1)
print(f"relative error of recovered interaction terms = {(torch.norm(learned - true_missing) / torch.norm(true_missing)).item():.4f}")
z0 = torch.rand(16, 2) * 2 + 0.5
err = (rk4_rollout(model, z0, dt, STEPS) - rk4_rollout(true_f, z0, dt, STEPS)).abs().mean().item()
print(f"mean abs trajectory error on new initial conditions = {err:.4f}")
# 7) Inference: estimate β from the network (first term ≈ −β x y)
with torch.no_grad():
x, y = 1.5, 1.0
print(f"estimated beta ≈ {(-model.nn(torch.tensor([[x, y]]))[0, 0] / (x * y)).item():.3f} (true {B_})")
This code was run on 2026-11-20 in 15.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
relative error of recovered interaction terms = 0.0343 mean abs trajectory error on new initial conditions = 0.0431
Trainable parameters in the example: model = 1,218.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Time Series Lab Another section of the site
Related architectures
Neural ordinary differential equations
Neural Ordinary Differential Equations (Neural ODE)
The hidden state is defined by a continuous differential equation whose derivative is given by a neural network, and the output is computed with a numerical solver, so t…
Physics-Informed Neural Networks
Physics-Informed Neural Networks (PINN)
A network that approximates the solution of a partial differential equation u_θ(x,t) and is trained with a loss that combines the equation's residual (computed by automa…
Physics-informed deep learning
Physics-Informed Deep Learning (PIML)
A general approach to embedding physical knowledge in deep learning in three ways: through data (observational bias), through the architecture (inductive bias, such as H…
Deep Operator Network
Deep Operator Network (DeepONet)
A network that learns an operator mapping a function to a function (such as: condition → solution), through a "branch" network that encodes the input function from its v…
Fourier Neural Operator
Fourier Neural Operator (FNO)
A neural operator that applies a "spectral convolution" in each layer: a Fourier transform of the features, then multiplication of a limited number of low frequencies by…
Liquid Neural Networks
Liquid Neural Networks (LNN)
Continuous-time recurrent networks in which each neuron's time constant changes with the input (Liquid Time-Constant), so their dynamics adapt to the data with a small n…
No results match the filters.
Research ideas
- UDE for modelling blood-glucose response to insulin in type 1 diabetes patients.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the core idea of universal differential equations?
Why? $\dot z=f_{known}+NN_\theta$.
How does the gradient flow in UDE training?
Why? Or via the adjoint method.
What is the benefit of extracting a symbolic formula for the learned term?
Why? It turns the network into an explicit law.
References
Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., Ramadhan, A., & Edelman, A. (2020). Universal differential equations for scientific machine learning. arXiv preprint. https://arxiv.org/abs/2001.04385
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems (NeurIPS) 31. https://arxiv.org/abs/1806.07366
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422–440. https://doi.org/10.1038/s42254-021-00314-5
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707. https://doi.org/10.1016/j.jcp.2018.10.045
