Uncertainty in large language models

Large language models (LLMs) are increasingly serving as a primary interface for accessing information and interacting with digital systems. Nonetheless, LLMs can generate text that includes incorrect or unsupported claims while being fluent, coherent, and presented as factual. This creates a fundamental reliability issue: the generated text alone often does not reveal whether it can be trusted. Such unsupported factual claims are commonly referred to as LLM hallucinations. Detecting hallucinations is essential for deciding when a generated text can be trusted and when the system should instead abstain from answering, retrieve further evidence, or defer to a human. A principled approach to hallucination detection is uncertainty quantification, as elevated uncertainty is associated with a higher risk of hallucinations. However, estimating uncertainty in LLMs is challenging because uncertainty depends on the model’s predictive distribution over possible output sequences. Since LLMs stochastically generate output sequences token by token, this predictive distribution is high-dimensional and generally intractable to compute exactly. Thus, uncertainty estimation must rely on tractable approximations of this high-dimensional distribution. Developing effective approximations is a key enabler for the reliable deployment of LLM-based systems. In this cumulative thesis, we address the challenges posed by high-dimensional distributions to advance uncertainty estimation. We first establish foundational principles in the context of classical classification: our method QUAM demonstrates that uncertainty estimation benefits from targeted exploration, explicitly searching for samples that most influence the estimate. We then translate this insight to LLMs through SDLG, a method that improves uncertainty estimation by sampling output sequences that are most informative about the underlying distribution. Furthermore, we show that competitive uncertainty estimates can also be obtained without sampling multiple output sequences by leveraging a single-sequence measure: the negative log-likelihood of the most likely output sequence, approximated through our method G-NLL. This provides a highly efficient and practical alternative to sampling-based uncertainty estimation in LLMs. While this thesis primarily focuses on the internal reliability of LLMs through uncertainty estimation, it also investigates the external reliability of LLM-based systems deployed as multimodal agents that observe and act within their environment. To this end, we introduce Malicious Image Patches (MIPs), demonstrating that multimodal agents can be compromised into executing malicious actions when such patches appear in their visual observations. By exposing this vulnerability, this work highlights the need for safeguards against external failure modes and broadens the thesis from the internal to the external reliability of systems built around LLMs. Overall, the core contributions of this thesis advance uncertainty estimation in LLMs and thereby improve uncertainty-based hallucination detection. Complemented by insights into malicious failure modes, these contributions provide a principled foundation for improving the internal and external reliability of LLM-based systems.