Learning semantic representations of language, vision and code for visualization-oriented dialogue
Abstract
Data visualization is essential for navigating the vast volume of data that defines our time, letting humans use perception to make sense of complex datasets. Yet creating visualizations requires mastery of specialized programming languages, producing a gap between the analytical intention in the user's mind and its implementation in executable code. This raises the cognitive load on experts, who cannot concentrate fully on visual exploration while also writing the visualization program, and it creates a barrier for non-experts, hindering widespread access to insights from data. Bridging this gap is a central challenge for Visualization-oriented Natural Language Interfaces (V-NLIs), which translate human intent into executable code. This work decomposes the problem into two interrelated challenges: understanding, which builds rich, contextual representations of user intent, and generation, which reliably synthesizes visualization programs under strict real-time constraints. The thesis addresses both through five interconnected contributions. It begins with a survey of 119 papers, revealing that while the community focuses on sophisticated interactive exploration, the underlying technology relies on classical, rule-based methods, motivating a turn to deep learning. KeywordScape, a visual analytics system, makes the contextualized representations of transformer models visible and interactive, showing they can themselves resolve ambiguity and enable fine-grained exploration. Paparazzi extends this to multimodal representations, revealing that large vision-language models such as CLIP exhibit surprising deficiencies in 3D viewpoint understanding, and introduces a data-efficient contrastive fine-tuning method to improve geometric grounding. Finally, VIST5, a practical V-NLI built on a retrieval-augmented architecture around a small language model, shows that complex visualization code can be synthesized in real time while ensuring data privacy and modular extensibility.
Keywords
Not available
Specialisations
Similar theses
- Improving Human Behavior Understanding with Semi-supervised Learning Bruno Manuel Degardin · 2026
- AI-Generated Image Detection Using Multimodal Deep Learning Mamadou Keïta · 2026
- A Study on Robustness, Adaptation, and Generalization for Visual and Multimodal Learning Xiasi Wang · 2026
- Addressing data heterogeneity, scarcity, and training efficiency in privacy-preserving federated learning Chen, Haokun · Ludwig-Maximilians-Universität München · 2025
- A Novel Explainable Artificial Intelligence Framework with Improved Learning Mechanisms Kabir, Sami · Luleå University of Technology · 2025
- Machine learning for pediatric healthcare Hamed Fayyaz · University of Delaware · 2025
