Long Text Understanding with Transformer-based Models
الملخص
Transformer-based models have achieved remarkable success across a wide range of natural language processing tasks. However, their performance degrades substantially when processing long inputs, including multi-turn dialogues, lengthy documents, and sequences beyond the training context window. These limitations arise not only from computational constraints but also from structural misalignment, where local predictions drift from global semantics, document coherence is fragmented, and positional extrapolation disrupts pretrained attention patterns. This thesis investigates long text understanding in Transformer-based models through the perspective of preserving structural alignment under constrained context regimes. Three complementary frameworks are proposed. First, for multi-turn dialogue understanding, a multi-level semantic distillation framework aligns intent, slot, and domain knowledge across utterance-, token-, and dialogue-level representations using heterogeneous teacher supervision to improve semantic consistency and robustness. Second, for long document understanding, a semantic-preserving chunk-level representation method compresses documents into meaningful chunks through unsupervised keyphrase extraction while retaining fine-grained token information. Third, for length extrapolation beyond the training context window, a training-free inference-time method stabilizes positional generalization by reusing pretrained positional intervals and locally interpolating attention logits. Experiments on multi-turn dialogue benchmarks, long document understanding tasks, and long-context evaluation suites demonstrate the effectiveness and generalizability of the proposed methods. Collectively, these contributions establish a unified framework for robust long text understanding by preserving semantic coherence, structural consistency, and positional stability under context constraints.
الكلمات المفتاحية
غير متوفر
التخصصات
رسائل وأطاريح مشابهة
- Adaptive Deep Learning for Vision and Multimodal Understanding with Heterogeneous Data Youwei Liang · 2026
- Hybrid Deep Learning Model: LLM-Enhanced Linear Self-Attention Transformer-CNN for Stock Price Prediction Yilong Chen · Auckland University of Technology · 2025
- Leveraging deep learning for visual understanding of videos Clement Xian Ren Tan · Nanyang Technological University · 2024
- Sparse Attention Mechanisms in Vision Transformers: from Spatiotemporal Transformers to Instance-Dependent Attention Patterns Brendan William Duke · University of Toronto · 2023
- Understanding Human-Model Alignment in Visual Social Attention with Video Transformers Sabrina M. Lopez · 2026
- Efficient fine-tuning of Transformer Models Under Hardware Constraints : An Empirical Study of LoRA and Knowledge Distillation Sai Mounika Karasu · Dalarna University Library · 2026
