Attention Mechanism

آلية الانتباه

Definition

A mechanism that lets the model give different weights to parts of the input according to their importance for the element it is processing now.

Definition

An attention mechanism lets a model compute a weighted combination of input representations, where the weights reflect how relevant each input element is to the element currently being processed.

Practical examples

Code example Python
import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    d_k = Q.shape[-1]
    weights = softmax(Q @ K.T / np.sqrt(d_k))
    return weights @ V, weights

rng = np.random.default_rng(0)
Q, K, V = rng.normal(size=(3, 4)), rng.normal(size=(3, 4)), rng.normal(size=(3, 4))
out, w = attention(Q, K, V)
print(out.shape, w.sum(axis=1))

Expected output

(3, 4) [1. 1. 1.]

Mathematical equations

\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V

Scientific sources

References and scientific sources

Learn and review

Suggested terms to learn next