Try for yourself how computers understand language: split text, normalize it, extract roots, tag parts of speech and entities, analyze sentiment and dialects, measure similarity, and summarize. All tools run in your browser without a language model and without sending your text anywhere, with special care for Arabic and its dialects.
The lab is not just tools: here is a complete curriculum to read in order, and a guide to models, tasks and data, and every page links you to the tool where you can try the idea.
Each module has an explanation, a working tool, examples, exercises and FAQs. The results come from traditional algorithms and the lab's lexicons, and each result is marked as approximate when it is.
Language processing is traditionally divided into levels: phonology, morphology for word structure, syntax for sentence structure, semantics for meaning, pragmatics for meaning in context, and discourse for the relations between sentences. The lab's tools cover morphology, syntax and part of semantics.
General knowledge in computational linguistics
Concept
The evolution of NLP approaches
Language processing went through three major stages: rule-based systems whose rules were written by experts (until about the 1980s), then statistical methods and machine learning from corpora (the 1990s and 2000s), then deep learning and distributed representations from about 2013, leading to pretrained Transformer models from 2018 and large language models.
Jurafsky & Martin, Speech and Language Processing (3rd ed. draft)
Concept
The linguistic corpus
A corpus is a collection of texts selected for a specific purpose. It may be raw or annotated (with parts of speech, entities or syntax trees). The quality of a statistical model is tied to the corpus's size and to how well it represents the texts the model will work on.
General knowledge
Concept
Numeric representation of text
Algorithms do not work on letters directly but on numbers. The simplest representations are the bag of words and TF-IDF weights, then dense vectors (Word2Vec, GloVe and FastText), then contextual representations from models such as BERT.
General knowledge
Concept
Bag of words
A representation that counts how often each word occurs in a text and ignores their order. Simple and effective in classification, but it does not distinguish «الكلب عض الرجل» ("the dog bit the man") from «الرجل عض الكلب» ("the man bit the dog").
General knowledge
Concept
TF-IDF
A word's weight = its frequency in the document (TF) × the inverse of how common it is across documents (IDF = log(N/df)). It raises the weight of words distinctive of the document and lowers words common to all documents. The lab uses it in summarization, keyword extraction and the assistant.
Salton & Buckley (1988)
Concept
Language models
A language model gives a probability to a sequence of words or predicts the next word. It began with statistical n-gram models, then recurrent neural networks, then the Transformer. Large language models are huge language models trained on vast amounts of text.
General knowledge
Concept
The Transformer architecture
An architecture published in 2017 in the paper "Attention Is All You Need" that relies on self-attention instead of recurrence, allowing parallel processing and representing distant relations between words. It is the basis of BERT, GPT and most modern translation systems.
Vaswani et al. (2017)
Concept
BERT and the Arabic models
BERT is a Transformer model pretrained by predicting masked words. Its Arabic versions include AraBERT (American University of Beirut), CAMeLBERT (New York University Abu Dhabi) and MARBERT (University of British Columbia), trained on tweets that include the dialects. These models are fine-tuned (Fine-tuning) for tasks such as classification and NER.
Devlin et al. (2019); Antoun et al. (2020); Inoue et al. (2021); Abdul-Mageed et al. (2021)
Concept
Fine-tuning
Training a pretrained model on the data of a specific task for a few epochs. It needs far less labelled data than training from scratch, and care must be taken to keep the test data separate and to watch for overfitting.
General knowledge
Concept
Retrieval-augmented generation (RAG)
An approach that retrieves relevant passages from a knowledge base and passes them to a language model to generate an answer based on them. It reduces hallucination and allows citing sources. The LLM mode of the lab's assistant uses this approach on the lab's knowledge base.
Lewis et al. (2020)
Concept
Hallucination in language models
Generating information that seems convincing but is incorrect or absent from the sources. That is why the lab's assistant cites the sources of its answer, says "I don't know" when it lacks enough information, and does not send text to an external model unless the administrator enables it.
General knowledge
Challenges of Arabic processing
Challenge
Missing diacritics
Most Arabic texts are written without vowel marks, so a word may have several readings (such as «كتب»: كَتَبَ، كُتُب، كُتِبَ). Solutions: morphological analysers that give all the possibilities followed by contextual disambiguators, automatic diacritization systems, or contextual models that handle the ambiguity implicitly.
Habash (2010) Introduction to Arabic Natural Language Processing
Challenge
Rich morphology and agglutination
Arabic is a derivational language to which prefixes and suffixes attach, so the forms of a single word multiply. This leads to huge vocabularies and statistical sparsity. Solutions: light stemming, morphological segmentation (such as the ATB scheme), or subwords (BPE/WordPiece).
Habash (2010)
Challenge
Spelling variation
Writers differ in how they write hamzas, yaa and taa marbuta, Persian letters are sometimes used, and common mistakes appear. Normalization solves part of the problem but may merge different words.
General knowledge
Challenge
Dialects and scarce resources
Online texts are full of dialects, which have fewer labelled resources than MSA, and the dialect varies within a single country. Well-known resources include the MADAR corpus, the NADI shared tasks, and the CALIMA analyser for Egyptian in CAMeL Tools.
Bouamor et al. (2018); Abdul-Mageed et al. (2020)
Challenge
Arabizi and mixed writing
Writing Arabic in Latin letters and digits, and mixing Arabic with English or French in the same sentence. It needs transliteration systems and word-level language identification.
General knowledge
Challenge
Evaluating Arabic systems
Many papers evaluate on different datasets or with different splits, which makes comparison hard. It is advisable to use published benchmark datasets with their official splits, state the versions, repeat experiments with different random seeds, and test statistical significance.
General research practice
Research directions
Research direction
Research topics: the Iraqi dialect
Researchable gaps include: labelled corpora for the Iraqi dialect in its varieties (Baghdadi, Moslawi, southern), province-level dialect identification, sentiment analysis of Iraqi comments, morphological segmentation for Iraqi, and translation between Iraqi and MSA. Recent literature must be checked before defining the gap.
A research suggestion (needs a literature review)
Research direction
Research topics: explainability (XAI) in Arabic NLP
Studying how well the explanations of methods such as attention, SHAP, LIME and Integrated Gradients agree with linguists' judgements in Arabic tasks such as sentiment and dialect identification, and the effect of subword tokenization on explainability.
A research suggestion (needs a literature review)
Research direction
Research topics: low-resource settings
Comparing traditional methods (such as Naive Bayes on character n-grams) with pretrained models when data are very scarce, data augmentation techniques for Arabic, and few-shot learning (Few-shot) with large language models.
A research suggestion (needs a literature review)
Research direction
Graduation project topics
Feasible examples: an Arabic news classifier with TF-IDF and SVM, a sentiment analyser for product comments comparing lexical and trained methods, an entity extractor for local news with rules and lists, an extractive summarizer with TextRank and ROUGE evaluation, and an Arabic search tool with light stemming, comparing accuracy with and without normalization.
A teaching suggestion
Research direction
A sound NLP experiment methodology
Define the research question, choose a benchmark dataset, split it before any statistics-based processing, build a simple baseline, compare with suitable metrics (Macro-F1 for imbalanced classes), repeat the experiment with different seeds, report confidence intervals, and analyse the errors by hand.
General research practice
About the lab
About the lab
The engines in the lab
The lab is built on an engine layer: a JavaScript engine in the browser (default), a PHP engine on the server (for the same operations when needed and for the assistant), an optional external Python engine, and an optional external API. Each engine has a list of capabilities, and a function appears only if its engine is enabled and tested.
Lab documentation
About the lab
Privacy in the lab
Tool texts never leave the browser in the default mode. Statistics are daily counts per tool without texts, and visitor counting uses a hashed fingerprint that cannot be reversed and is deleted after the retention period. No data is sent to a third party unless the administrator enables a language model or an external engine.
Lab documentation
About the lab
The difference between the lab's sections
The lab separates four kinds: traditional algorithms that actually run in the browser, language resources (dictionaries, lexicons and lists) from the database, machine learning and deep learning models that are described and run only with an external engine, and optional large language models for the assistant only. Every result is labelled with its kind.
Lab documentation
About the lab
Related libraries
The "Libraries guide" shows real tools such as CAMeL Tools, PyArabic, NLTK, spaCy, Stanza, Farasa, Hugging Face Transformers, the AraBERT, CAMeLBERT and MARBERT models, Arabic WordNet and Universal Dependencies, with each one's licence as it appeared at its official source and the date it was checked.
Lab documentation
How to read the results
Reading the results
How to read the confidence level
"High" means the decision came from a dictionary or a fixed pattern. "Medium" means a form rule or a match after separating prefixes. "Low" means a rule-based guess. Low results need human review and are not suitable for scientific conclusions.
Lab documentation
General questions about the lab and the assistant
What is natural language processing?
Natural language processing (NLP) is a branch of artificial intelligence and computer science concerned with making computers analyse, understand and generate human language. Its tasks include tokenization, morphological and syntactic analysis, entity recognition, sentiment analysis, translation and summarization.
What is the difference between traditional algorithms and trained models in this lab?
Traditional algorithms are well-defined rules and mathematical formulas (such as Light10, TF-IDF and TextRank) that always give the same result and whose steps can be traced. Trained models (such as CRF and BERT) learn from labelled data and need model files. The lab runs the traditional ones in the browser, and only describes the trained models unless the Python engine or an external API is enabled.
Does the lab need the internet or an API key?
No. All the default tools run in the browser with JavaScript algorithms and data from the site's database, and the text is not sent to any external service. The large language model and the Python engine are optional and only an administrator can enable them.
Are the texts I write saved?
Tool texts are never saved, because processing happens in your browser. Only usage counts per tool (a daily number) are recorded, if the administrator enables them. Assistant questions are not logged as text unless the administrator explicitly enables saving questions.
Why does the word "approximate" appear next to some results?
Because some tools are rule-based or lexicon-based with limited coverage, such as rule-based tagging, dialect detection and lexical sentiment. The lab distinguishes a deterministic result (such as a word count) from an estimate, and shows a confidence level and a reason for each decision so that the result is not taken as a confirmed scientific verdict.
What are the biggest challenges in Arabic language processing?
Among the most prominent: ambiguity caused by missing diacritics, rich derivational morphology, prefixes and suffixes attached to the word, the many dialects written without a standard, writing in Latin letters (Arabizi), and the scarcity of labelled resources for some dialects compared with English.
What is the difference between MSA and the dialects from the point of view of automatic processing?
MSA has unified spelling and grammar rules and many resources, while the dialects differ in vocabulary, morphology and pronunciation and are written without a standard spelling. So tools built on MSA lose accuracy when applied to dialect text.
What is a pipeline in NLP?
It is a series of stages the text passes through in order, such as: cleaning, then normalization, then tokenization, then stopword removal, then stemming, then a numeric representation, then a model. The order of the stages affects the result, and you can try this in the "Playground".
Can I use the lab's results in scientific research?
The lab is a teaching tool. It can help you understand and compare methods, but research results must be built on documented tools and resources (such as CAMeL Tools, Farasa or published models) and on benchmark datasets, stating the versions and evaluation settings.
What are the common metrics for evaluating NLP systems?
Accuracy, precision, recall and the F1 score are used for classification and tagging, BLEU and chrF for translation, ROUGE for summarization, LAS and UAS for dependency parsing, and perplexity for language models.
What is the difference between precision and recall?
Precision is the share of correct predictions among everything the system predicted, and recall is the share of all the correct cases that the system caught. The F1 score is their harmonic mean.
What is data leakage in NLP experiments?
It is information from the test data reaching training, such as building the vocabulary or TF-IDF weights from all the data before splitting, or duplicate sentences shared between training and test. This inflates results falsely, and the fix is to build all statistics from the training set only.
What is meant by Arabizi?
Writing Arabic or its dialects in Latin letters and digits, such as 3 for ʿayn, 7 for ḥaa and 2 for hamza. It needs transliteration before Arabic tools can be applied, and this lab does not convert it automatically.
How is this lab related to the Smart Glossary and the AI Academy?
The lab focuses on hands-on language processing. For extended definitions see the Smart Glossary, and for sequenced lessons see the NLP track in the AI Academy. Their links appear on the lab's home page when both plugins are installed.
What is the difference between traditional NLP and large language models (LLMs)?
Traditional NLP builds a separate algorithm or model for each task, with features designed by the researcher. Large language models are pretrained on huge amounts of text and then steered with instructions or fine-tuned for the task. They are stronger at understanding and generation, but cost more and may generate incorrect information (hallucination).
What is the NLP Lab assistant?
An assistant that answers NLP questions from the lab's local knowledge base with traditional retrieval methods (normalization, synonyms, TF-IDF, matching frequently asked questions). It works offline, shows the sources of its answer, and if it finds no answer it says so and suggests related tools.
Does the assistant use ChatGPT or a language model?
Not by default. An administrator can enable a language model provider (OpenAI, Anthropic, Gemini, DeepSeek or an OpenAI-compatible one) with their own key, and then only your question and the retrieved passages are sent to the model, and the answer is labelled with its source.
Why did the assistant say it found no answer?
Because your question does not match the database content closely enough. The assistant does not guess. Try other words, use one of the suggested sections, or see the Smart Glossary.
Does the assistant understand Iraqi Arabic?
It understands common question words such as شنو, شلون, ليش, شكد and وين, turns them into their MSA equivalents before searching, and tolerates some spelling mistakes.
Are my questions to the assistant saved?
By default, only a short log without the question text is kept (the section, the intent, the result and the rating) to improve the database, and it is deleted automatically after the retention period. Saving the question text is an option the administrator must enable explicitly.
What is rating an answer for?
Your rating (helpful or not helpful) is stored with the short log so the administrator knows which questions need improvement in the knowledge base.
Does the assistant answer questions outside NLP?
No. Its scope is natural language processing and the lab's tools. For other questions it suggests the relevant sections of the site.
Assistant Professor of Artificial Intelligence and Deep Learning
University of Al-Qadisiyah
About
Researcher and academic specialising in artificial intelligence (AI), deep learning, computer vision and natural language processing (NLP). The “Deep Learning in Arabic” project aims to enrich Arabic scientific and technical content with sound, free references and software for researchers and learners across the Arab world.
NLP Lab assistant
I am the NLP Lab assistant. Ask me about a concept, an algorithm or a dialect word, and I will answer from the lab's knowledge base, citing the sources.
NLP Lab assistant
Knowledge mode: the lab's knowledge base
It answers from the lab's knowledge base only, without an external language model and without sending your question anywhere.
Smart assistant
Answers from the site’s content
The assistant can make mistakes, so check the attached sources.