AdvancedTemporal and Sequential NetworksGraph Neural Networks

Temporal Graph Networks (TGN)

A framework for learning representations on dynamic graphs represented as a stream of timed events, using a memory for each node that is updated by events and an embedding module that aggregates temporal neighbours.

Overview

NameTemporal Graph Networks
Year introduced2020
ResearchersEmanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein
InstitutionTwitter
Original paperTemporal graph networks for deep learning on dynamic graphs arXiv
Data typeGraphs, Sequences and time series
DomainsRecommender systems, Fraud detection, Social network analysis, Learning on graph data

Problem it solves

Many graphs change continuously (interactions, transactions, messages), and static models ignore the timing and order of events.

Background and evolution

What came before: GraphSAGE, Gated recurrent unit, JODIE/DyRep/TGAT

  1. 2020

    Rossi and colleagues introduced TGN as a general framework that includes earlier models as special cases. [ref]

Core idea

Each node has a memory vector $s_i(t)$ that summarises its history. When an event occurs between $i$ and $j$ at time $t$, a message is built from their two memories, the time gap and the event features, and the memory is updated with a GRU. At prediction time the node embedding is computed from its memory and its temporal neighbours with attention or aggregation.

Example: On a payment platform, an account active for years suddenly makes tens of transfers in minutes: the memory and the time gaps capture this change.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Event streamEvents $(i,j,t,e_{ij})$
Message functionFrom the two memories, the time and the event
Time encodingA periodic encoding of $\Delta t$
Memory updaterA GRU for each node
Embedding moduleAttention over temporal neighbours
DecoderPredicting future links

Equations

1. Memory update

$$m_i(t)=\mathrm{msg}\!\left(s_i(t^-),\,s_j(t^-),\,\phi(t-t_i^-),\,e_{ij}(t)\right),\qquad s_i(t)=\mathrm{GRU}\!\left(m_i(t),\,s_i(t^-)\right)$$

Symbols
$s_i(t^-)$Memory before the event
$\phi$Time encoding
$t_i^-$Time of the node's last event
Dimensions
$s_i\in\mathbb{R}^{d}$
Numerical example
If 10 time units have passed since the last event, $\phi(10)$ enters the message, so the model knows how long the gap was.
Relation to the architecture
This is the mechanism that makes the representation depend on history.

2. Time encoding

$$\phi(\Delta t)=\cos(\omega\,\Delta t+b)$$

Symbols
$\omega,b$Learned vectors
Dimensions
$\phi(\Delta t)\in\mathbb{R}^{d_t}$
Numerical example
With $\omega=(1,0.1)$, $b=0$ and $\Delta t=\pi$: $\phi=(-1,\cos 0.314)\approx(-1,0.951)$.
Relation to the architecture
It turns the time gap into a vector suited to the network.

How it works, step by step

  1. Events are ordered in time and processed in batches.
  2. Embeddings are computed from the current memory to predict the batch's events.
  3. The loss is computed against negative samples.
  4. Messages are built from the batch's events.
  5. The memory is updated after the computation (to avoid information leakage).

Strengths and limitations

Strengths

  • It captures the dynamics of graphs and the timing of events.
  • A general framework that includes earlier models.

Limitations

  • The update order is sensitive; a mistake in it leaks data from the future.
  • Training is relatively sequential and harder to parallelise.

When to use it, and when not?

Good for

  • Predicting upcoming interactions.
  • Fraud detection in transaction streams.

Not suited for

  • Static graphs or a few widely spaced snapshots.

Applications

DomainUse
Recommender systemsPredicting the next interaction between a user and an item.
Fraud detectionDetecting anomalous behaviour in a transaction stream.
Social network analysisPredicting follows and messages.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# A simplified Temporal Graph Network (TGN) for link prediction
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class TimeEnc(nn.Module):
    def __init__(self, d):
        super().__init__(); self.lin = nn.Linear(1, d)
    def forward(self, dt):
        return torch.cos(self.lin(dt[:, None]))                 # φ(Δt)

class TGN(nn.Module):
    def __init__(self, N, d=16, dt_dim=8):
        super().__init__()
        self.N, self.d = N, d
        self.time = TimeEnc(dt_dim)
        self.msg = nn.Linear(2 * d + dt_dim, d)
        self.gru = nn.GRUCell(d, d)
        self.emb = nn.Embedding(N, d)                           # fixed node features
        self.dec = nn.Sequential(nn.Linear(2 * d, d), nn.ReLU(), nn.Linear(d, 1))
        self.reset()
    def reset(self):
        self.mem = torch.zeros(self.N, self.d); self.last = torch.zeros(self.N)
    def z(self, nodes):
        return self.mem[nodes] + self.emb(nodes)                # embedding = memory + feature
    def score(self, src, dst):
        return self.dec(torch.cat([self.z(src), self.z(dst)], 1)).squeeze(1)
    def update(self, src, dst, t):
        for a, b in ((src, dst), (dst, src)):
            m = self.msg(torch.cat([self.mem[a], self.mem[b], self.time(t - self.last[a])], 1))
            self.mem = self.mem.clone(); self.mem[a] = self.gru(torch.relu(m), self.mem[a])
            self.last = self.last.clone(); self.last[a] = t

# 3) Prepare the data
# A synthetic event stream: 40 users in 4 groups, interacting mostly within their group
N = 40
group = torch.arange(N) % 4
src = torch.randint(0, N, (1200,))
same = torch.rand(1200) < 0.85
dst = torch.where(same, (src + 4 * torch.randint(1, 10, (1200,))) % N, torch.randint(0, N, (1200,)))
t = torch.arange(1200).float()
split = 1000
model = TGN(N)

# 4) The loss: BCE for a real event against a random negative event
loss_fn = nn.BCEWithLogitsLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)

# 5) Training loop: we predict the batch first, then update the memory (preventing leakage from the future)
for epoch in range(4):
    model.reset(); total = 0.0
    for i in range(0, split, 50):
        s, d_, tt = src[i:i + 50], dst[i:i + 50], t[i:i + 50]
        neg = torch.randint(0, N, (len(s),))
        logits = torch.cat([model.score(s, d_), model.score(s, neg)])
        labels = torch.cat([torch.ones(len(s)), torch.zeros(len(s))])
        loss = loss_fn(logits, labels)
        opt.zero_grad(); loss.backward(); opt.step(); total += loss.item()
        with torch.no_grad():
            model.update(s, d_, tt)
        model.mem, model.last = model.mem.detach(), model.last.detach()
    print(f"epoch {epoch}  loss={total / (split // 50):.4f}")

# 6) Evaluation on the later events in time
with torch.no_grad():
    s, d_ = src[split:], dst[split:]
    pos, neg = model.score(s, d_), model.score(s, torch.randint(0, N, (len(s),)))
    acc = ((pos > neg).float().mean()).item()
print(f"test accuracy = {acc:.3f}   (share of real events scored higher than the negative)")

# 7) Inference: probability that user 0 interacts with 4 (same group) and with 1 (another group)
with torch.no_grad():
    print("p(0→4) =", torch.sigmoid(model.score(torch.tensor([0]), torch.tensor([4]))).item(),
          " p(0→1) =", torch.sigmoid(model.score(torch.tensor([0]), torch.tensor([1]))).item())

This code was run on 2026-11-20 in 2.9 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 0.705   (share of real events scored higher than the negative)

Trainable parameters in the example: model = 3,489.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2017

GraphSAGE

GraphSAGE

An inductive framework that learns aggregation functions over a sampled neighbourhood, so it can compute representations for new nodes the network did not see during tra…

Graph Neural Networks

Basic2014

Gated recurrent unit

Gated Recurrent Unit (GRU)

A simplification of LSTM with only two gates (update and reset) and no separate cell state, with comparable performance on many tasks and fewer parameters.

Temporal and Sequential Networks

Intermediate2018

Graph Attention Network

Graph Attention Network (GAT)

It replaces GCN's fixed weights with learned attention coefficients for each edge, so each node decides how important each neighbour is, with multiple attention heads.

Graph Neural Networks

Research ideas

  • Apply TGN to fraud detection in transactions with a strict temporal split and evaluation by AUC-PR.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the function of the memory in TGN?

  2. Concept

    Why is the memory updated after predicting the batch and not before?

  3. Mathematical

    What does $\phi(\Delta t)$ encode?

References

  1. Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., & Bronstein, M. (2020). Temporal graph networks for deep learning on dynamic graphs. ICML 2020 Workshop on Graph Representation Learning and Beyond. https://arxiv.org/abs/2006.10637

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Hamilton, W. L., Ying, R., & Leskovec, J. (2017). Inductive representation learning on large graphs. Advances in Neural Information Processing Systems (NeurIPS) 30. https://arxiv.org/abs/1706.02216

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. Proceedings of EMNLP 2014, 1724–1734. https://doi.org/10.3115/v1/D14-1179

    DOIarXivAutomatically verified via Crossref – 2026-11-20