IntermediateGraph Neural Networks

Graph Attention Network (GAT)

It replaces GCN's fixed weights with learned attention coefficients for each edge, so each node decides how important each neighbour is, with multiple attention heads.

Overview

NameGraph Attention Network
Year introduced2018
ResearchersPetar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio
Original paperGraph attention networks arXiv
Data typeGraphs
DomainsFraud detection, Scientific computing, Social network analysis, Learning on graph data
Computational cost$O(|E|\,F\,K)$ per layer.
Data sizeMedium to large graphs; with neighbourhood sampling for huge graphs.

Problem it solves

In GCN neighbours are treated with weights that depend only on degree, while some neighbours may be far more important than others.

Background and evolution

What came before: Graph Convolutional Network, Attention Mechanism

  1. 2018

    GAT was introduced at ICLR 2018. [ref]

  2. 2022

    Brody and colleagues showed that GAT attention is "static" and proposed GATv2 with dynamic attention. [ref]

Core idea

For each edge $(i,j)$ a score $e_{ij}$ is computed from the two node representations after a linear transformation, then normalised with Softmax over the neighbours of $i$. The new representation is a weighted average with these coefficients. Several independent heads are used, then concatenated or averaged.

Example: In a network of scientific papers, a paper may give more weight to a paper in the same field than to a heavily cited general paper.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
InputNode features and the graph
Linear projection$Wh$ for each node
Attention scores$a^\top[Wh_i\|Wh_j]$ with LeakyReLU
Masked SoftmaxOver the neighbours only
Multi-headK independent heads
OutputConcatenating or averaging the heads

Equations

1. Attention coefficients

$$\alpha_{ij}=\frac{\exp\!\left(\mathrm{LeakyReLU}\!\left(a^{\top}[Wh_i\,\|\,Wh_j]\right)\right)}{\sum_{k\in\mathcal{N}(i)\cup\{i\}}\exp\!\left(\mathrm{LeakyReLU}\!\left(a^{\top}[Wh_i\,\|\,Wh_k]\right)\right)}$$

Symbols
$W$Projection matrix $F\times d$
$a$Attention vector of length $2F$
$\|$Concatenation
Dimensions
$\alpha_{ij}\in(0,1)$, summing to 1 over the neighbours
Numerical example
If the scores for three neighbours are $(1,0,0)$, then $\alpha\approx(0.576,0.212,0.212)$.
Relation to the architecture
It sets the importance of each neighbour in a learned way.

2. Multi-head node update

$$h_i^{\prime}=\Big\|_{k=1}^{K}\sigma\!\left(\sum_{j\in\mathcal{N}(i)\cup\{i\}}\alpha_{ij}^{k}W^{k}h_j\right)$$

Symbols
$K$Number of heads
$W^k$Projection of head k
Dimensions
$h_i^{\prime}\in\mathbb{R}^{KF}$
Numerical example
With 8 heads and F=8 the output dimension becomes 64.
Relation to the architecture
Multiple heads stabilise learning and capture different relationship patterns.

How it works, step by step

  1. Each node's features are projected linearly.
  2. A score is computed for each edge from its two nodes.
  3. The scores are normalised over each node's neighbours.
  4. The neighbours' representations are aggregated with the weights.
  5. The process repeats for each head, then the results are concatenated.

Strengths and limitations

Strengths

  • Learned, flexible neighbour weights.
  • Works inductively on new graphs.
  • The attention weights can be inspected.

Limitations

  • Slower and more memory-hungry than GCN.
  • Attention in the original GAT is relatively static (addressed by GATv2).

When to use it, and when not?

Good for

  • When the importance of neighbours varies.
  • Medium-sized graphs with rich features.

Not suited for

  • Very large graphs with limited memory and no sampling.

Applications

DomainUse
Learning on graph dataClassifying nodes, links and whole graphs.
Recommender systemsRecommender systems that represent users and items as a bipartite graph.
Scientific computingPredicting molecular properties in computational chemistry.
Fraud detectionFraud detection in financial transaction networks.
Social network analysisSocial network analysis and community detection.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Graph Attention Network (GAT) — a simplified dense implementation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class GATLayer(nn.Module):
    def __init__(self, d_in, d_out, heads=4, concat=True):
        super().__init__()
        self.h, self.f, self.concat = heads, d_out, concat
        self.W = nn.Linear(d_in, heads * d_out, bias=False)
        self.a_src = nn.Parameter(torch.randn(heads, d_out) * 0.1)
        self.a_dst = nn.Parameter(torch.randn(heads, d_out) * 0.1)
    def forward(self, X, A):
        N = X.shape[0]
        Wh = self.W(X).view(N, self.h, self.f)                                   # N×H×F
        e = (Wh * self.a_dst).sum(-1).unsqueeze(1) + (Wh * self.a_src).sum(-1).unsqueeze(0)  # e[i,j,h]
        e = F.leaky_relu(e, 0.2)
        mask = (A + torch.eye(N)).unsqueeze(-1) > 0
        e = e.masked_fill(~mask, float('-inf'))
        alpha = torch.softmax(e, dim=1)                                          # softmax over the neighbours j
        out = torch.einsum('ijh,jhf->ihf', alpha, Wh)
        return out.reshape(N, -1) if self.concat else out.mean(1)

class GAT(nn.Module):
    def __init__(self, d_in, n_classes=3):
        super().__init__()
        self.l1 = GATLayer(d_in, 8, heads=4)
        self.l2 = GATLayer(32, n_classes, heads=1, concat=False)
    def forward(self, X, A):
        h = F.elu(self.l1(F.dropout(X, 0.3, self.training), A))
        return self.l2(F.dropout(h, 0.3, self.training), A)

# 3) Prepare the data
# A synthetic graph from a stochastic block model (SBM): 3 communities, more links inside than between
N, C, D = 150, 3, 8
y = torch.arange(N) % C
P = torch.where(y[:, None] == y[None, :], 0.12, 0.01)
A = (torch.rand(N, N) < P).float().triu(1); A = A + A.T          # a symmetric adjacency matrix without loops
X = torch.randn(N, D) + 0.6 * F.one_hot(y, D).float()             # weak-signal features; the structure helps
idx = torch.randperm(N); train_idx, test_idx = idx[:45], idx[45:] # only 30% labelled (semi-supervised)

model = GAT(D)

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3, weight_decay=5e-4)

# 5) Training loop (full-batch: the whole graph at each step, with the loss on the labelled nodes only)
for epoch in range(120):
    model.train()
    out = model(X, A)                                   # forward pass for all nodes
    loss = loss_fn(out[train_idx], y[train_idx])
    opt.zero_grad(); loss.backward(); opt.step()
    if epoch % 30 == 0:
        print(f"epoch {epoch:3d}  loss={loss.item():.4f}")

# 6) Evaluation on the unlabelled nodes
model.eval()
with torch.no_grad():
    pred = model(X, A).argmax(1)
print(f"test accuracy = {(pred[test_idx] == y[test_idx]).float().mean().item():.3f}")

# 7) Inference: the class of node 0
print("node 0 predicted community:", pred[0].item(), " true:", y[0].item())

This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 0.962

Trainable parameters in the example: model = 422.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2017

Graph Convolutional Network

Graph Convolutional Network (GCN)

A simple graph layer that takes a normalised average of the representations of a node and its neighbours and then applies a shared linear transformation; it is a first-o…

Graph Neural Networks

Advanced2020

Graph Transformer

Graph Transformer (GT)

Applying self-attention to the nodes of a graph with positional or structural encodings (such as Laplacian eigenvectors) so the model knows the graph's structure, going…

Graph Neural NetworksSpecialized ArchitecturesTransformer Architectures

Basic2015

Attention Mechanism

Attention Mechanism

A mechanism that lets the model compute a weight for each input element according to its relevance to the current step, then take a weighted average, instead of compress…

FoundationsTemporal and Sequential Networks

Research ideas

  • Interpret GAT attention weights in fraud detection and compare them with GNNExplainer explanations.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Over which set are the attention coefficients normalised in GAT?

  2. Comparison

    What is the essential difference between GAT and GCN?

  3. Concept

    What did GATv2 address?

References

  1. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1710.10903

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Brody, S., Alon, U., & Yahav, E. (2022). How attentive are graph attention networks?. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2105.14491

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1609.02907

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20