Graph Attention Network (GAT)
It replaces GCN's fixed weights with learned attention coefficients for each edge, so each node decides how important each neighbour is, with multiple attention heads.
Overview
| Name | Graph Attention Network |
|---|---|
| Year introduced | 2018 |
| Researchers | Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio |
| Original paper | Graph attention networks arXiv |
| Data type | Graphs |
| Domains | Fraud detection, Scientific computing, Social network analysis, Learning on graph data |
| Computational cost | $O(|E|\,F\,K)$ per layer. |
| Data size | Medium to large graphs; with neighbourhood sampling for huge graphs. |
Problem it solves
In GCN neighbours are treated with weights that depend only on degree, while some neighbours may be far more important than others.
Background and evolution
What came before: Graph Convolutional Network, Attention Mechanism
Core idea
For each edge $(i,j)$ a score $e_{ij}$ is computed from the two node representations after a linear transformation, then normalised with Softmax over the neighbours of $i$. The new representation is a weighted average with these coefficients. Several independent heads are used, then concatenated or averaged.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Input | Node features and the graph |
| Linear projection | $Wh$ for each node |
| Attention scores | $a^\top[Wh_i\|Wh_j]$ with LeakyReLU |
| Masked Softmax | Over the neighbours only |
| Multi-head | K independent heads |
| Output | Concatenating or averaging the heads |
Equations
1. Attention coefficients
| $W$ | Projection matrix $F\times d$ |
| $a$ | Attention vector of length $2F$ |
| $\|$ | Concatenation |
2. Multi-head node update
| $K$ | Number of heads |
| $W^k$ | Projection of head k |
How it works, step by step
- Each node's features are projected linearly.
- A score is computed for each edge from its two nodes.
- The scores are normalised over each node's neighbours.
- The neighbours' representations are aggregated with the weights.
- The process repeats for each head, then the results are concatenated.
Strengths and limitations
Strengths
- Learned, flexible neighbour weights.
- Works inductively on new graphs.
- The attention weights can be inspected.
Limitations
- Slower and more memory-hungry than GCN.
- Attention in the original GAT is relatively static (addressed by GATv2).
When to use it, and when not?
Good for
- When the importance of neighbours varies.
- Medium-sized graphs with rich features.
Not suited for
- Very large graphs with limited memory and no sampling.
Applications
| Domain | Use |
|---|---|
| Learning on graph data | Classifying nodes, links and whole graphs. |
| Recommender systems | Recommender systems that represent users and items as a bipartite graph. |
| Scientific computing | Predicting molecular properties in computational chemistry. |
| Fraud detection | Fraud detection in financial transaction networks. |
| Social network analysis | Social network analysis and community detection. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# Graph Attention Network (GAT) — a simplified dense implementation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class GATLayer(nn.Module):
def __init__(self, d_in, d_out, heads=4, concat=True):
super().__init__()
self.h, self.f, self.concat = heads, d_out, concat
self.W = nn.Linear(d_in, heads * d_out, bias=False)
self.a_src = nn.Parameter(torch.randn(heads, d_out) * 0.1)
self.a_dst = nn.Parameter(torch.randn(heads, d_out) * 0.1)
def forward(self, X, A):
N = X.shape[0]
Wh = self.W(X).view(N, self.h, self.f) # N×H×F
e = (Wh * self.a_dst).sum(-1).unsqueeze(1) + (Wh * self.a_src).sum(-1).unsqueeze(0) # e[i,j,h]
e = F.leaky_relu(e, 0.2)
mask = (A + torch.eye(N)).unsqueeze(-1) > 0
e = e.masked_fill(~mask, float('-inf'))
alpha = torch.softmax(e, dim=1) # softmax over the neighbours j
out = torch.einsum('ijh,jhf->ihf', alpha, Wh)
return out.reshape(N, -1) if self.concat else out.mean(1)
class GAT(nn.Module):
def __init__(self, d_in, n_classes=3):
super().__init__()
self.l1 = GATLayer(d_in, 8, heads=4)
self.l2 = GATLayer(32, n_classes, heads=1, concat=False)
def forward(self, X, A):
h = F.elu(self.l1(F.dropout(X, 0.3, self.training), A))
return self.l2(F.dropout(h, 0.3, self.training), A)
# 3) Prepare the data
# A synthetic graph from a stochastic block model (SBM): 3 communities, more links inside than between
N, C, D = 150, 3, 8
y = torch.arange(N) % C
P = torch.where(y[:, None] == y[None, :], 0.12, 0.01)
A = (torch.rand(N, N) < P).float().triu(1); A = A + A.T # a symmetric adjacency matrix without loops
X = torch.randn(N, D) + 0.6 * F.one_hot(y, D).float() # weak-signal features; the structure helps
idx = torch.randperm(N); train_idx, test_idx = idx[:45], idx[45:] # only 30% labelled (semi-supervised)
model = GAT(D)
# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3, weight_decay=5e-4)
# 5) Training loop (full-batch: the whole graph at each step, with the loss on the labelled nodes only)
for epoch in range(120):
model.train()
out = model(X, A) # forward pass for all nodes
loss = loss_fn(out[train_idx], y[train_idx])
opt.zero_grad(); loss.backward(); opt.step()
if epoch % 30 == 0:
print(f"epoch {epoch:3d} loss={loss.item():.4f}")
# 6) Evaluation on the unlabelled nodes
model.eval()
with torch.no_grad():
pred = model(X, A).argmax(1)
print(f"test accuracy = {(pred[test_idx] == y[test_idx]).float().mean().item():.3f}")
# 7) Inference: the class of node 0
print("node 0 predicted community:", pred[0].item(), " true:", y[0].item())
This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test accuracy = 0.962
Trainable parameters in the example: model = 422.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Graph Network Explorer
- AI Model Atlas Another section of the site
- Graph Neural Networks Lab Another section of the site
Related architectures
Graph Convolutional Network
Graph Convolutional Network (GCN)
A simple graph layer that takes a normalised average of the representations of a node and its neighbours and then applies a shared linear transformation; it is a first-o…
Graph Transformer
Graph Transformer (GT)
Applying self-attention to the nodes of a graph with positional or structural encodings (such as Laplacian eigenvectors) so the model knows the graph's structure, going…
Attention Mechanism
Attention Mechanism
A mechanism that lets the model compute a weight for each input element according to its relevance to the current step, then take a weighted average, instead of compress…
No results match the filters.
Research ideas
- Interpret GAT attention weights in fraud detection and compare them with GNNExplainer explanations.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
Over which set are the attention coefficients normalised in GAT?
Why? A neighbourhood-masked Softmax.
What is the essential difference between GAT and GCN?
Why? This is the essence of graph attention.
What did GATv2 address?
Why? Brody and colleagues showed that the ranking of neighbours in the original GAT does not change with the querying node.
References
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1710.10903
Brody, S., Alon, U., & Yahav, E. (2022). How attentive are graph attention networks?. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2105.14491
Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1609.02907
