CONTRASTIVE ATTENTION TRANSFORMER FOR CLASSIFICATION
Abstract
Most traditional classification methods process each data point in isolation, ignoring the structural information embedded in the relationships among data points. This dissertation develops the contrastive attention transformer, a novel classification architecture that attends a given point separately to each candidate class, producing one representation per class, and predicts the label by contrasting these representations through a shared classifier. We develop the framework for both binary and multiclass classification, proposing three loss functions under two complementary approaches. The comparative approach enforces the relative ordering of confidences across representations through the probability difference and proportional losses. The aggregation approach maximizes joint correctness through the conditional probability loss, which we show to coincide with a forecast aggregation model from decision analysis and to be equivalent to classification on the sum of the representations, motivating a flexible concatenation architecture. A key-value caching scheme further reduces the computational cost of attending to each class separately by 50 to 60 percent for large numbers of classes. We evaluate the framework on five binary classification tasks, four multiclass classification tasks, and a controlled CIFAR-100 experiment, spanning image, text, and tabular data. The best contrastive variant outperforms a feedforward network baseline and a standard transformer baseline with unrestricted attention on every binary and multiclass task. The CIFAR-100 experiment identifies the distinguishability of the classes, rather than the number of classes, as the primary driver of the contrastive method's advantage over unrestricted attention.
Keywords
Not available
Specialisations
Similar theses
- Transformers and other Attention-Based Algorithms for Biomedicine Marco Hernández Pérez · University of Salamanca · 2025
- Transformers and contrastive semi-supervised learning for medical image segmentation Qianying Liu · The University of Glasgow · 2026
- Understanding Human-Model Alignment in Visual Social Attention with Video Transformers Sabrina M. Lopez · 2026
- Hybrid Deep Learning Model: LLM-Enhanced Linear Self-Attention Transformer-CNN for Stock Price Prediction Yilong Chen · Auckland University of Technology · 2025
- Higher Order Transformers With Kronecker-Structured Attention Soroush Omranpour · 2025
- Sparse Attention Mechanisms in Vision Transformers: from Spatiotemporal Transformers to Instance-Dependent Attention Patterns Brendan William Duke · University of Toronto · 2023
