Positional Encoding
الترميز الموضعي
Definition
Adding token order information to their embeddings, because self-attention does not distinguish positions by itself.
Definition
Positional encoding injects token position information into embeddings, since self-attention is otherwise permutation-invariant.
Mathematical equations
PE_{(pos,2i)}=\sin\!\left(\frac{pos}{10000^{2i/d}}\right),\; PE_{(pos,2i+1)}=\cos\!\left(\frac{pos}{10000^{2i/d}}\right)