Positional Encoding

الترميز الموضعي

Definition

Adding token order information to their embeddings, because self-attention does not distinguish positions by itself.

Definition

Positional encoding injects token position information into embeddings, since self-attention is otherwise permutation-invariant.

Mathematical equations

PE_{(pos,2i)}=\sin\!\left(\frac{pos}{10000^{2i/d}}\right),\; PE_{(pos,2i+1)}=\cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Learn and review

Suggested terms to learn next