Explainable AI Lab

Feature importance, exact SHAP, LIME, partial dependence, counterfactuals and black box vs interpretable models, trained in your browser.

العربية

Module experiments (6)

Concepts and formulas

Shapley value formula

v(S) is the mean prediction with features in S fixed and the rest from the background.

\phi_i=\sum_{S\subseteq F\setminus\{i\}}\frac{|S|!\,(|F|-|S|-1)!}{|F|!}\left[v(S\cup\{i\})-v(S)\right]

Try it in: SHAP values for one decision

SHAP efficiency formula

SHAP values add up to prediction minus average prediction.

f(x)=\mathbb{E}[f(X)]+\sum_{i=1}^{M}\phi_i

Try it in: SHAP values for one decision

LIME objective formula

Fits a simple model g approximating f near x with proximity weights.

\xi(x)=\arg\min_{g\in G}\;\mathcal{L}(f,g,\pi_x)+\Omega(g),\quad \pi_x(z)=\exp\!\left(-\frac{D(x,z)^2}{\sigma^2}\right)

Try it in: LIME explanation and its stability

Explainable AI (XAI)

Methods that make model decisions understandable: inherently interpretable models vs post-hoc explanations, global vs local. Explanations describe the model, not the world, and do not prove causation.

Try it in: Permutation feature importance

SHAP values

Split the difference between a prediction and the average prediction across features using Shapley values (Lundberg & Lee, 2017), with efficiency, symmetry and null-player properties. The lab computes them exactly over all 32 coalitions of five features against a background sample, matching the shap library to 1e-9.

Try it in: SHAP values for one decision

LIME (LIME)

Explains one decision with a simple local linear model: perturb samples near the case, query the original model, weight by proximity (exponential kernel) and fit a weighted regression (Ribeiro et al., 2016). Results depend on sampling and kernel width, so the lab measures stability over seeds.

Try it in: LIME explanation and its stability

Permutation importance

The drop in model performance when one feature's values are shuffled. A large drop means strong reliance. Correlated features can share importance.

Try it in: Permutation feature importance

Partial dependence and ICE

PDP shows the model's average prediction as one feature is set to different values; ICE curves show it per individual, revealing interactions the average hides.

Try it in: Partial dependence and ICE curves

Counterfactual explanation

Answers: what is the smallest change to my inputs that would change the decision? (Wachter et al., 2017). It gives recourse without revealing the model, if restricted to realistic, mutable features.

Try it in: Counterfactual: what would change the decision?

Questions and guides

Do SHAP values mean a feature causes the outcome?

No. SHAP describes how the model uses features, not real-world causation. Correlated features can trade importance, and results depend on the background data.

Try it in: SHAP values for one decision

What is the difference between SHAP and LIME?

Both explain one decision locally. SHAP rests on Shapley values and its contributions add up to the difference from the average; exact computation is costly with many features. LIME fits a weighted linear model around the case; faster and flexible but less stable, depending on sampling and kernel width.

Try it in: LIME explanation and its stability

Module sources

  1. SHAP documentationS. Lundberg et al. (2025) — Tool
  2. LIME — Local Interpretable Model-agnostic Explanations (code)M. T. Ribeiro (2016) — Tool
  3. Interpretable Machine LearningC. Molnar (2025) — Book
  4. scikit-learn — Permutation feature importancescikit-learn developers (2025) — Tool
  5. scikit-learn — Partial dependence and ICE plotsscikit-learn developers (2025) — Tool
  6. A Unified Approach to Interpreting Model Predictions (arXiv:1705.07874)S. Lundberg, S.-I. Lee (2017) — Paper
  7. "Why Should I Trust You?" Explaining the Predictions of Any Classifier (arXiv:1602.04938)M. T. Ribeiro, S. Singh, C. Guestrin (2016) — Paper
  8. Counterfactual Explanations without Opening the Black Box (arXiv:1711.00399)S. Wachter, B. Mittelstadt, C. Russell (2017) — Paper
  9. Peeking Inside the Black Box: ICE plots (arXiv:1309.6392)A. Goldstein, A. Kapelner, J. Bleich, E. Pitkin (2013) — Paper