Text tokenization

Tokenization Lab

Split text into words, sentences, characters and subword units, and see the statistics and the difference between traditional tokenization and the tokenization of modern models.

What is this module?

Tokenization is the first step in most language processing systems: turning a string of characters into units that can be counted and processed. Traditional tokenization relies on spaces, punctuation and simple rules, while modern models such as BERT and GPT use Subword Tokenizers that learn their vocabulary from a large text corpus, such as BPE, WordPiece and Unigram. In Arabic the difficulty grows because a single word may carry a conjunction, a preposition, the definite article and an attached pronoun, as in «وبكتابهم» ("and with their book").

Tool: Text tokenization

Runs in your browser (JavaScript) — your text is not sent anywhere
Settings

How does the tool work?

  • Word tokenization: a regular expression that separates Arabic and Latin words, numbers and punctuation.
  • Sentence tokenization: splitting after . ! ? ؟ ؛ … while excluding abbreviations such as “د.” (Dr.) and decimal numbers.
  • Character tokenization: each letter with its diacritics as one unit.
  • Subword tokenization (simulation): training BPE on the input text itself or on the lab corpus, then encoding WordPiece-style (longest match first).
  • Clitic tokenization: rule-based separation of conjunctions, prepositions, the definite article and attached pronouns.

How to read the results?

Each unit is colored by its type (Arabic, English, number, punctuation). The statistics are computed from the resulting units: average word length in letters without diacritics, and the Type-Token Ratio (TTR), which measures vocabulary diversity.

Algorithms and methods

We always distinguish between traditional algorithms, resource-based methods and trained models. The tools on this page are traditional or lexicon-based, while trained models are explained here and run only through an external engine enabled by the site admin.

Traditional algorithm Implemented in the lab Regex word tokenization Rule-based (Regex) Word Tokenization Splitting text into words, numbers and punctuation marks using regular expressions that describe the shape of each kind of unit. Traditional algorithm Implemented in the lab Sentence splitting Sentence Boundary Detection Finding sentence boundaries in a text from punctuation marks, with rules that prevent wrong splits at abbreviations and decimal numbers. Traditional algorithm Implemented in the lab Byte Pair Encoding Byte-Pair Encoding (BPE) A subword tokenization algorithm that learns its vocabulary from a corpus by repeatedly merging the most frequent adjacent pair of symbols, producing units between a character and a word. Traditional algorithm Implemented in the lab WordPiece tokenization WordPiece A subword tokenization method used by BERT models: the vocabulary is built by choosing the merges that raise the likelihood of the training data, and a word is encoded by longest match from the left. Traditional algorithmFor explanation and comparison Unigram language model tokenization Unigram Language Model Tokenization A subword tokenization method that starts with a large vocabulary and gradually removes the least useful units based on a unigram probabilistic model (Unigram), choosing the best segmentation with the Viterbi algorithm. Resource-based Implemented in the lab Arabic clitic segmentation Arabic Clitic Segmentation Splitting an Arabic word into prefixes (conjunctions, prepositions and the definite article), a stem and suffixes (attached pronouns), such as «وبكتابهم» → و + ب + كتاب + هم.

Limitations and accuracy limits

Subword tokenization here is an educational simulation with a small vocabulary trained on a short text, so it does not match the real AraBERT or GPT vocabularies. Rule-based sentence splitting fails with unknown abbreviations and with text that has no punctuation.

This tool's results are educational: they show how the algorithm works step by step, and they are not presented as confirmed scientific results.

From the knowledge base

Concept

Tokenization in modern models

Modern models use fixed subword vocabularies learned once: BERT uses WordPiece, GPT-2 uses byte-level BPE, and T5 and mT5 use SentencePiece (Unigram). The choice of tokenizer affects sequence length and how Arabic is represented.

Sennrich et al. (2016); Kudo (2018)

Frequently asked questions

What is tokenization?

Splitting text into smaller units called tokens, which may be words, parts of words, characters or sentences. It is the first step in almost any language processing system.

Why is splitting on spaces not enough in Arabic?

Because a written Arabic word may carry several units: a conjunction, a preposition, the definite article and a pronoun, such as «وبكتابهم» = و + ب + كتاب + هم. Splitting on spaces counts it as one token, which inflates the vocabulary and weakens the statistics.

What is subword tokenization?

Splitting words into recurring parts that the algorithm learns from the corpus, such as BPE, WordPiece and Unigram. It allows representing words unseen in training through their parts, and it is the basis of modern models such as BERT and GPT.

How does the BPE algorithm work?

It starts by representing each word as separate characters, then counts the most frequent adjacent pair in the corpus and merges it into a new symbol, repeating this a set number of times. Its output is an ordered list of merges that is applied in the same order to new texts.

What is the difference between BPE and WordPiece?

Both build a subword vocabulary. BPE merges the most frequent pair, while WordPiece chooses the merge that most raises the corpus likelihood under a language model, and uses the ## prefix for word-internal pieces when applied with longest match.

Why does BPE in the lab give different results from models such as AraBERT?

Because the lab learns its merge list from the text you enter or from a small teaching corpus, while large models learn their vocabulary from millions of sentences. The algorithm is the same but the data differ, so the resulting pieces differ.

How does the lab split text into sentences?

It relies on sentence-end marks (. ! ؟ ? and optionally the semicolon) with a list of known abbreviations so that it does not split at «د.» or "Dr.", and it ignores the decimal point in numbers. Splitting errors are still possible in unpunctuated text.

What is clitic segmentation?

Separating attached particles and pronouns from the word stem, such as «فسيكتبونها» = ف + س + يكتبون + ها. The lab shows every possible segmentation and marks those that match a list of known words, because the right decision needs context or a full morphological analyser.

Why does the lab show more than one segmentation for a single word?

Because an undiacritized Arabic word may allow more than one correct segmentation. For example, «وجد» may be a single verb or «و + جد». Showing the alternatives makes the ambiguity clear instead of claiming a single answer.

What is the token type shown in the table?

The lab classifies each token as: Arabic word, Latin word, number, punctuation mark, emoji, link, email or hashtag (#). This helps with later cleaning decisions.

What is the difference between the number of words and the number of unique words?

The number of words (Tokens) counts all words with repetition, and the number of unique words (Types) counts the distinct words. Their ratio (Type-Token Ratio) is a simple indicator of vocabulary diversity, and it is affected by the length of the text.

Does the tool support character-level tokenization?

Yes, you can choose tokenization into characters or character n-grams. It is useful for dialects and spelling mistakes because it does not depend on a fixed vocabulary.

Exercises and challenges

Beginner Hands-on in the lab

Counting tokens

Enter the sentence «وبكتابهم وجدنا الحل» in the tokenization tool. How many tokens does whitespace tokenization give? And how many units does prefix and suffix segmentation suggest for the word «وبكتابهم»?

Hint

Look at the "Segmentations" column in the table.

Show solution

By whitespace: 3 tokens. «وبكتابهم» = و + ب + كتاب + هم, that is 4 units.

Beginner Computing by hand

Type-token ratio

Compute the ratio of unique words to the number of words (TTR) in «في البيت وفي الحديقة في المساء» without any normalization.

Hint

«وفي» differs in form from «في».

Show solution

The tokens are 6 (في، البيت، وفي، الحديقة، في، المساء) and the unique ones 5, so TTR = 5/6 ≈ 0.83.

Intermediate Computing by hand

The first BPE merge

A corpus of three words: «كتب كتاب كتابة». What is the first character pair that BPE merges?

Hint

Count all adjacent pairs within the words.

Show solution

The pairs: ك-ت (3), ت-ب (1), ت-ا (2), ا-ب (2), ب-ة (1). The most frequent is ك-ت, so the first merge is «كت».

Intermediate Hands-on in the lab

BPE on a small corpus

Load the example "A small corpus for trying BPE" and run 15 merges. What is the longest subword unit that appeared? And why did it appear before the others?

Hint

Review the table of merges in order.

Show solution

Units such as «كت», then «كتا» or «كتب» appear because these strings are the most frequent in the corpus. The exact order depends on the pair counts after each merge.

Intermediate Conceptual question

BPE versus WordPiece

Explain in three sentences the difference in the merge criterion between BPE and WordPiece.

Hint

Think of "frequency" versus "likelihood".

Show solution

BPE merges the most frequent pair. WordPiece chooses the merge that most increases the corpus likelihood under a language model, which roughly equals the pair's frequency divided by the product of its parts' frequencies. So WordPiece prefers merging rare parts that always appear together.

Advanced Hands-on in the lab

Tricky sentence splitting

Write a text that contains «د.», «3.5» and an Arabic question mark «؟», and check how many sentences the tool gives. Then remove the space after one of the periods and try again.

Hint

The tool relies on an abbreviation list and on the number pattern.

Show solution

The sentence must not be split at «د.» or at 3.5. Removing the space after a sentence-final period may merge two sentences, which is a known limitation of rule-based splitting.

Advanced Research challenge

The effect of segmentation on the vocabulary

Why does separating prefixes and suffixes reduce vocabulary size in Arabic? Estimate the effect on a 1,000-word text using the tool's statistics before and after morphological segmentation.

Hint

Compare the number of unique words.

Show solution

Because forms such as «كتابه، وكتابهم، بكتابها» become the unit «كتاب» with shared affixes, so the number of unique words falls. The exact effect varies with the text.

Beginner Conceptual question

Token types

Classify the following tokens: «#ذكاء_اصطناعي», «😊», «https://a.com», «٣٤», «AI».

Hint

Use the type columns in the tool.

Show solution

Hashtag, emoji, link, number, Latin word.

Related sections of the site