Efficient algorithms for scaling large language models

Recent advances in machine learning have transformed it from a specialized predictive tool into a general-purpose technology, exemplified by foundation models that power modern artificial intelligence systems. Much of this progress has been driven by scaling along two key dimensions: model size and context length. In the text domain, larger models enable stronger reasoning and generation capabilities, while longer contexts allow models to process complex inputs such as documents, interaction histories, and tool outputs, enabling emerging applications including AI assistants and autonomous agents. However, continued scaling along these dimensions introduces substantial challenges for current hardware systems during both training and inference stages. This dissertation studies how to improve the efficiency and adaptability of large language models (LLMs) along these two scaling axes. The central goal is to enable large models to be trained and deployed efficiently while improving their ability to process and reason over long contexts. To achieve this goal, the dissertation develops methods that address key bottlenecks in both two scaling axes. The first part of this dissertation focuses on efficient scaling of model capacity across the entire pipeline, from training to deployment. Scaling model size introduces three practical challenges: the large memory overhead required for training, the high latency and system cost of serving large models during inference, and the need to deploy models across heterogeneous devices with different hardware budgets. To address these issues, this work introduces memory-efficient optimization methods that reduce training overhead through compressed gradient representations, adaptive inference mechanisms that selectively activate or load critical model parameters to improve serving efficiency, and elastic deployment strategies that adjust model capacity to the available hardware resources. Together, these techniques enable large models to scale more efficiently and flexibly in real-world systems. The second part of this dissertation focuses on scaling context length for LLMs. Long-context modeling introduces several challenges, including the efficiency bottleneck of inference caused by KV caches whose memory footprint grows linearly with sequence length, the difficulty for models to effectively utilize information distributed across long contexts, and the lack of mechanisms to control generation during long reasoning processes. To address these challenges, this work develops efficient long-context inference methods that reduce KV-cache overhead through selective context retention, improved positional encoding techniques that enhance long-range dependency modeling, and inference-time steering mechanisms that guide reasoning behavior during long-context generation. These approaches improve both the efficiency of long-context inference and the effectiveness of reasoning over extended contexts. Together, the contributions of this dissertation advance the foundations of scalable large-model systems. By improving training efficiency, enabling adaptive model deployment, and enhancing long-context reasoning, this work helps bridge the gap between the rapidly increasing capabilities of large language models and the practical constraints of real-world hardware systems.