Understanding and the Black Box Problem in Artificial Intelligence

Artificial intelligence (AI) systems developed with automated machine learning procedures are used throughout industry, science, and society. However, the opacity of these systems threatens to undermine the satisfaction of critical desiderata including safety, trust, and fairness. This worry is known as the black box problem (BBP). In this dissertation, I develop a novel conceptual framework for analysing the BBP and, through five interrelated papers, address three key research questions about its nature and our prospects for solving it. First, I consider the central epistemic construct of understanding, arguing that an abilities-based view is best-suited to the requirements of BBP analysis. Conceptualising understanding as abilities, I claim, appropriately balances theoretical justification with practical utility and can help coordinate interdisciplinary efforts in explainable AI (XAI). Second, I address the potential of XAI, a research field which attempts to render AI systems comprehensible to human stakeholders. I focus on mechanistic interpretability, an approach which employs functional decomposition to characterise the internal structure of AI systems. Mechanistic interpretability can deliver detailed explanations, but at significant costs in time and labour. Additionally, I present a framework for evaluating explainability approaches, using it to show how performance-explainability trade-offs can be effectively analysed. Third, I address the solubility of the BBP in particular contexts. In neuroscience, I argue against critics that artificial neural network models, in conjunction with XAI, can enhance our understanding of the brain. In psychiatry, I argue that even if the opacity of AI systems is overcome, data-driven approaches are unlikely to deliver transformative improvements to patient outcomes. Overall, this dissertation demonstrates how an abilities-based view of understanding and a considered assessment of XAI’s potential can be combined to yield new insights into the BBP and inform its resolution across diverse contexts.