Uncertainty in Explainable Artificial Intelligence

Artificial intelligence (AI) has become deeply embedded in modern decision-making systems, driving advances across science, industry and society. Yet, as models grow in scale and complexity, they increasingly operate as opaque systems whose internal reasoning remains difficult to understand. Explainable AI (XAI) seeks to remedy this by providing interpretable insights into model behavior, supporting model verification, accountability and calibrated human-AI trust. However, a growing body of work shows that explanations themselves are oftentimes unstable, noisy or sensitive to slight perturbations - a variation that could be attributed to datasets, models or explanation techniques. Without a thorough understanding of where this uncertainty comes from and a reliable way to communicate it, explanations risk being misleading, unreliable and ultimately detrimental to user trust. This threatens to turn XAI from a transparency tool into a further source of ambiguity. In this dissertation, we address these challenges by developing a principled framework for Uncertainty in Explainable AI (UXAI) that conceptualizes and operationalizes uncertainty within the explanation process. First, we introduce a mathematical formulation of explainers as functions $e_{\theta}(x, f)$, enabling a propagation-based view of how uncertainty originating in the data, model and explainer hyperparameters flows into the final explanation. Building on this formulation, we derive a test for checking how reliably uncertainty is propagated through the explanation pipeline, comparing first-order uncertainty approximations against empirical perturbation-based estimates. We then demonstrate how uncertainty quantification can enhance transparency in practice by (i) constructing a post-hoc stochastic concept-based explainer and (ii) conformalizing a multimodal, multitask model used in real-world scientific workflows. As far as the human dimension is concerned, we develop a human-in-the-loop experimental framework to study how explanations communicated together with model uncertainty influence human-AI collaboration, user performance and learning in unfamiliar tasks. Finally, we conclude by discussing what is still missing in UXAI and future directions that could advance the field further.