Two-Stream Transformer Architecture With Discrete Attention for Better Interpretrability and Separation of Model Concerns
Abstract
The transformer has become a central model for natural language processing tasks ranging from translation to classification to representation learning. Its success demonstrates the effectiveness of stacked attention as a replacement for recurrence for many tasks. Attention is broadly interpreted as selectively attending to different parts on an input. So, in theory, attention offers more insights into the model’s internal decisions; however, in practice, when stacked, it quickly becomes nearly as fully-connected, making it hard to disentangle final decision dependencies. In this work, we propose an alternative transformer architecture, discrete transformer, with the goal of improving model interpretability. We use discrete latent variable attention to ensure that decision steps only depend on a limited context. We separate out attention decisions from representation modeling by using a separate stream for each. Empirically, on both classification and translation tasks, this approach maintains similar levels of performance on several datasets as the standard transformer, while obtaining quantitatively better attention interpretability and separating out syntactic features in the learned representations. Finally, our two-stream formulation can be used to transfer knowledge in a multiview arithmetic evaluation task.
Keywords
Not available
Specialisations
Similar theses
- Explainable Transformers in Natural Language Processing George Mihaila · 2026
- Automating news classification with large language models: Exploring fine-tuning, dataset size, and architecture Yesilyurt, Burcu · 2025
- Using Bidirectional Encoder Representations from Transformers for Conversational Machine Comprehension Evangelina Gogoulou · KTH Royal Institute of Technology · 2019
- A Comparative Forecasting Performance Analysis of Symbolic Representation Techniques for Time Series Using Transformer Models and Multi-Head Attention Mechanisms Rami Alalwan · University of Klagenfurt · 2026
- Counter-Rotating Aperture Resolution: A Formal Stream-Processing Model for SHA-256 Recurrence, Root-Cover Coupling, and Transformer Language Attention Dean Kulik · 2026
- AI-Generated Image Detection Using Multimodal Deep Learning Mamadou Keïta · 2026
