Tokenization
الترميز إلى رموز
Definition
The process of splitting text into units (tokens) such as words, subwords or characters for the model to process.
Definition
Tokenization splits text into discrete units such as words, subwords or characters and maps them to integer ids from a fixed vocabulary for model input.
Practical examples
text = 'learning is fun'
tokens = text.split()
vocab = {t: i for i, t in enumerate(sorted(set(tokens)))}
print([vocab[t] for t in tokens])Expected output
[2, 1, 0]
