Tokenization in modern models
Modern models use fixed subword vocabularies learned once: BERT uses WordPiece, GPT-2 uses byte-level BPE, and T5 and mT5 use SentencePiece (Unigram). The choice of tokenizer affects sequence length and how Arabic is represented.
Sennrich et al. (2016); Kudo (2018)
