Attention Mechanism
آلية الانتباه
Definition
A mechanism that lets the model give different weights to parts of the input according to their importance for the element it is processing now.
Definition
An attention mechanism lets a model compute a weighted combination of input representations, where the weights reflect how relevant each input element is to the element currently being processed.
Practical examples
import numpy as np
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
def attention(Q, K, V):
d_k = Q.shape[-1]
weights = softmax(Q @ K.T / np.sqrt(d_k))
return weights @ V, weights
rng = np.random.default_rng(0)
Q, K, V = rng.normal(size=(3, 4)), rng.normal(size=(3, 4)), rng.normal(size=(3, 4))
out, w = attention(Q, K, V)
print(out.shape, w.sum(axis=1))Expected output
(3, 4) [1. 1. 1.]
Mathematical equations
\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)VScientific sources
References and scientific sources
-
Research paperNeural Machine Translation by Jointly Learning to Align and Translate One of the first works on the attention mechanism in machine translation.
-
Research paperAttention Is All You Need The paper that introduced the Transformer architecture.
