NLP LLMs Intermediate

Tokenization

الترميز إلى رموز

Definition

The process of splitting text into units (tokens) such as words, subwords or characters for the model to process.

Definition

Tokenization splits text into discrete units such as words, subwords or characters and maps them to integer ids from a fixed vocabulary for model input.

Practical examples

Code example Python
text = 'learning is fun'
tokens = text.split()
vocab = {t: i for i, t in enumerate(sorted(set(tokens)))}
print([vocab[t] for t in tokens])

Expected output

[2, 1, 0]

Learn and review

Suggested terms to learn next