Adapting the transformer attention mechanism for efficient and effective information retrieval

Transformer-based language models have become the dominant architecture in natural language processing because they are effective across a wide range of tasks. However, an architecture that works for many tasks is not necessarily the best architecture for a specific task. In this dissertation, we investigate how we can adapt the attention mechanism inside a transformer layer to improve the effectiveness and efficiency of transformer-based language models for the specific task of informational retrieval. We introduce three attention adaptations. The first improves the efficiency of bi-encoders by pooling neighboring vectors inside the attention mechanism. The second improves the efficiency of cross-encoders by deactivating the attention between various sub-sequences, such as between query and document tokens. The third improves the effectiveness of cross-encoders by allowing multiple documents to attend to one another via an additional interaction token. Our analyses show that our attention adaptations lead to substantial effectiveness and efficiency improvements. In addition, we introduce two resources that simplify fine-tuning and evaluating bi-encoders and cross-encoders. The first is LightningIR, a framework that unifies the implementation of various model variants, and the second is Rank-DistiLLM, a set of high-quality distillation datasets for fine-tuning. We demonstrate that our resources can produce state-of-the-art bi-encoders and cross-encoders with minimal manual effort. We conclude that our attention adaptations demonstrate the potential of task-specific attention modifications, but the space of possible adaptations is not yet exhausted. We hope that our work inspires future research on task-specific adaptations for transformer-based models for information retrieval and other tasks.