Text preprocessing

Text Preprocessing Lab

Thirteen cleaning and normalization operations for Arabic and English text, which you can enable and order as you like, and see the effect of each step.

What is this module?

Text Preprocessing reduces unhelpful variation in text before it is represented: diacritics, tatweel (elongation), hamza forms, alif maqsura and taa marbuta make the same word appear in many forms. Unifying them increases word matching in search and classification, but it may merge words with different meanings (such as «علم» "knowledge" and «عَلَم» "flag"), so each operation must be chosen according to the task: what helps information retrieval may hurt morphological analysis or text generation.

Tool: Text preprocessing

Runs in your browser (JavaScript) — your text is not sent anywhere
Settings

How does the tool work?

  • The selected operations are applied in the order you set.
  • After each operation the resulting text is saved, and the words that were removed or changed are counted.
  • Hamza policy: "alif only" unifies أ إ آ ٱ into ا, "with carriers" adds ؤ→و and ئ→ي, and "all hamzas" also removes the standalone hamza ء.
  • Stop words are removed using the database list after normalization.

How to read the results?

Compare the number of words and characters before and after processing, and review the list of modified words: if two different words become the same form, the operation may be more aggressive than your task needs.

Algorithms and methods

We always distinguish between traditional algorithms, resource-based methods and trained models. The tools on this page are traditional or lexicon-based, while trained models are explained here and run only through an external engine enabled by the site admin.

Limitations and accuracy limits

The stop-word list is general and may remove a word that matters for a specific task (such as «لا» "not" in sentiment analysis). Converting haa into taa marbuta fails with the attached pronoun. There is no spelling correction operation.

This tool's results are educational: they show how the algorithm works step by step, and they are not presented as confirmed scientific results.

From the knowledge base

Rule

Hamza and its forms

Hamza is written on an alef (أ، إ), a waw (ؤ), a yaa (ئ), on the line (ء), and with madda (آ). Many writers drop the hamza and write «احمد» instead of «أحمد». That is why normalization unifies the hamzated alef forms to «ا» in search systems.

Spelling rules

Rule

Alef maqsura and yaa

Alef maqsura (ى) is written at the end of words such as «على» and «مستشفى», and is often confused with yaa (ي). Common normalization turns it into yaa, so «على» and «علي» match even though they differ.

Spelling rules

Frequently asked questions

What is text preprocessing?

A set of cleaning and unification operations before analysis, such as removing diacritics and tatweel, unifying hamzas, yaa and taa marbuta, and removing links and symbols. Its aim is to reduce surface variation that does not matter for the task.

What is Arabic text normalization?

Unifying the different written forms of the same letter, such as turning أ إ آ into ا, ى into ي and ة into ه. It increases matches between words, but may merge words with different meanings.

Should diacritics always be removed?

Not always. Removing them helps search and classification because most texts are undiacritized, but it deletes important information in tasks such as automatic diacritization, text-to-speech and poetry analysis.

Why might unifying hamzas be harmful?

Because it merges different words, such as «أمل» and «إمل», or «سأل» and «سال». That is why the lab offers several policies: no unification, alef only, carriers, or full unification, so you can compare the effect of each.

What is the difference between turning taa marbuta into haa and the reverse?

Turning ة into ه is common in search systems because writers confuse the two, and the reverse (ه into ة) is riskier because it changes pronouns such as «كتابه». The lab makes the direction an explicit option.

What is tatweel (kashida)?

The character ـ that stretches a word visually, as in «جـــميل». It carries no meaning, so it is usually removed during processing.

What are stopwords?

Highly frequent words with little topical meaning, such as في, من, على and الذي. Removing them helps search and summarization, but harms sentiment analysis because negation particles such as «لا» and «ليس» may be in the list.

Why doesn't the lab remove negation particles along with stopwords by default?

Because removing them reverses the meaning of the sentence in sentiment analysis and classification, as in «لم يعجبني» ("I did not like it"). There is a separate option to keep negation particles when removing stopwords.

What does the repetition-shrinking option do?

It shrinks letters repeated more than twice, such as «حلوووو» to «حلوو» or «حلو» depending on the setting. Common in social media text, but it may change rare correct words.

How does the lab handle Western and Eastern Arabic digits?

They can be left, removed, unified to Western digits (0-9) or Eastern Arabic digits (٠-٩), or replaced with a fixed symbol such as <NUM> so that the vocabulary does not grow.

Why does the lab show the result of each stage?

So you can see the effect of each operation on its own: what changed and what was removed. This helps you choose the right processing for the task instead of applying everything automatically.

Does the order of processing operations matter?

Yes. For example, removing punctuation before detecting links breaks the links, and removing stopwords before normalization may leave non-unified forms of the same words. You can change the order in the "Playground".

Does the lab remove Persian letters?

It does not remove them; it offers an option to convert the Persian and Urdu forms that are mistakenly typed in place of the Arabic letter, such as ک to ك, ی to ي and ہ to ه. It keeps «گ», «چ» and «پ» as they are, because in Iraqi texts they mark sounds that the Arabic letter does not convey: «گ» for the voiced qaf or the Cairene jim, and «چ» for the "ch" sound.

Exercises and challenges

Beginner Hands-on in the lab

Remove diacritics

Apply "Remove diacritics" to the al-Mutanabbi example. How many characters were removed?

Hint

Compare the length of the text before and after in the statistics bar.

Show solution

The number equals the count of vowel marks, shaddas and sukuns in the line, and the tool shows it in the "Changed" line.

Beginner Conceptual question

When do we keep diacritics?

Name two tasks that removing diacritics harms.

Hint

Think of pronunciation and metre.

Show solution

Automatic diacritization, text-to-speech, prosodic analysis of poetry, and teaching reading.

Intermediate Hands-on in the lab

Hamza policies

Apply the four hamza unification policies to «أسئلة سؤال مسائل سأل». Which policy makes the largest number of words identical?

Hint

Look at the stages table.

Show solution

The "full unification" policy removes the most differences, but brings different words together. The "alef only" policy does not change ؤ and ئ.

Intermediate Conceptual question

The risk of ه→ة

Why is turning a word-final haa into taa marbuta risky? Give two examples.

Hint

Think of attached pronouns.

Show solution

Because the haa may be the third-person pronoun, as in «كتابه» and «رأيته»; converting it gives «كتابة» and «رأيتة», which changes the meaning or breaks the word.

Intermediate Hands-on in the lab

Stopwords and negation

Run stopword removal on «لم يعجبني الفيلم» once while keeping negation particles and once without. What happens to the meaning?

Hint

The option is under the stopword settings.

Show solution

Without keeping negation, the sentence becomes «يعجبني الفيلم» ("I liked the film"), so the meaning is reversed. That is why the lab keeps it by default.

Beginner Hands-on in the lab

Social media text

Clean the "A social media text" example so that only Arabic text remains, with no tatweel, repetition or emojis. Which operations did you use?

Hint

Start with cleaning, then tatweel, then repetition, then symbols.

Show solution

Tatweel, shrinking repetition, removing emojis, removing punctuation, unifying or removing digits, then collapsing spaces.

Advanced Hands-on in the lab

The order of operations

In the playground, put "stopwords" before "unify alef" and then after it, on a text containing «إلى» and «أن». Does the result differ? Why?

Hint

The stopword list is written in normalized or original form.

Show solution

It may differ, because matching against the list is affected by the form of the hamza. The lab normalizes the word before matching, so the difference is smaller, but the order of operations still matters for other operations.

Intermediate Conceptual question

Persian letters in Iraqi Arabic

Why might we not want to turn «گ» into «ك» when studying the Iraqi dialect?

Hint

Think of the sound it represents.

Show solution

Because «گ» represents a different sound (the Cairene jim or the voiced qaf), and converting it erases dialect information that may help with dialect identification.

Related sections of the site