Explainable AI Lab
Feature importance, exact SHAP, LIME, partial dependence, counterfactuals and black box vs interpretable models, trained in your browser.
العربيةModule experiments (6)
Concepts and formulas
Shapley value formula
v(S) is the mean prediction with features in S fixed and the rest from the background.
\phi_i=\sum_{S\subseteq F\setminus\{i\}}\frac{|S|!\,(|F|-|S|-1)!}{|F|!}\left[v(S\cup\{i\})-v(S)\right]SHAP efficiency formula
SHAP values add up to prediction minus average prediction.
f(x)=\mathbb{E}[f(X)]+\sum_{i=1}^{M}\phi_iLIME objective formula
Fits a simple model g approximating f near x with proximity weights.
\xi(x)=\arg\min_{g\in G}\;\mathcal{L}(f,g,\pi_x)+\Omega(g),\quad \pi_x(z)=\exp\!\left(-\frac{D(x,z)^2}{\sigma^2}\right)Explainable AI (XAI)
Methods that make model decisions understandable: inherently interpretable models vs post-hoc explanations, global vs local. Explanations describe the model, not the world, and do not prove causation.
SHAP values
Split the difference between a prediction and the average prediction across features using Shapley values (Lundberg & Lee, 2017), with efficiency, symmetry and null-player properties. The lab computes them exactly over all 32 coalitions of five features against a background sample, matching the shap library to 1e-9.
LIME (LIME)
Explains one decision with a simple local linear model: perturb samples near the case, query the original model, weight by proximity (exponential kernel) and fit a weighted regression (Ribeiro et al., 2016). Results depend on sampling and kernel width, so the lab measures stability over seeds.
Permutation importance
The drop in model performance when one feature's values are shuffled. A large drop means strong reliance. Correlated features can share importance.
Partial dependence and ICE
PDP shows the model's average prediction as one feature is set to different values; ICE curves show it per individual, revealing interactions the average hides.
Counterfactual explanation
Answers: what is the smallest change to my inputs that would change the decision? (Wachter et al., 2017). It gives recourse without revealing the model, if restricted to realistic, mutable features.
Questions and guides
Do SHAP values mean a feature causes the outcome?
No. SHAP describes how the model uses features, not real-world causation. Correlated features can trade importance, and results depend on the background data.
What is the difference between SHAP and LIME?
Both explain one decision locally. SHAP rests on Shapley values and its contributions add up to the difference from the average; exact computation is costly with many features. LIME fits a weighted linear model around the case; faster and flexible but less stable, depending on sampling and kernel width.
Module sources
- SHAP documentationS. Lundberg et al. (2025) — Tool
- LIME — Local Interpretable Model-agnostic Explanations (code)M. T. Ribeiro (2016) — Tool
- Interpretable Machine LearningC. Molnar (2025) — Book
- scikit-learn — Permutation feature importancescikit-learn developers (2025) — Tool
- scikit-learn — Partial dependence and ICE plotsscikit-learn developers (2025) — Tool
- A Unified Approach to Interpreting Model Predictions (arXiv:1705.07874)S. Lundberg, S.-I. Lee (2017) — Paper
- "Why Should I Trust You?" Explaining the Predictions of Any Classifier (arXiv:1602.04938)M. T. Ribeiro, S. Singh, C. Guestrin (2016) — Paper
- Counterfactual Explanations without Opening the Black Box (arXiv:1711.00399)S. Wachter, B. Mittelstadt, C. Russell (2017) — Paper
- Peeking Inside the Black Box: ICE plots (arXiv:1309.6392)A. Goldstein, A. Kapelner, J. Bleich, E. Pitkin (2013) — Paper
