Value estimation in Large Language Models
الملخص
Large language models (LLMs) are commonly post-trained with reinforcement learning to improve reasoning performance. A central design choice in this process is whether to estimate advantages using a learned value function or directly from Monte Carlo returns. Value-based estimators can reduce variance and improve credit assignment but require training an additional critic, whereas Monte Carlo estima- tors are simpler and avoid this computational cost. This thesis studies the utility of value models during both training and inference. We first conduct controlled com- parisons between value-based and Monte Carlo estimators across models, tasks, and optimization settings. No estimator is uniformly superior: Monte Carlo methods can achieve competitive peak performance, while PPO often provides stronger average performance or lower sensitivity to configuration selection. Because surrounding op- timization choices can have a larger effect than the estimator, learning a critic does not consistently yield a better final policy. We then introduce value decoding, a test-time generation method that reuses the trained critic as a process supervisor. At each decoding step, the actor generates multiple candidate sub-trajectories, the critic scores them, and the highest-valued candidate is selected to continue the response. Across our experiments, value de- coding consistently improves upon the actor’s pass@1 performance. To understand 4 what determines its effectiveness, we systematically study the influence of additional critic training, critic size, training objective, and value-target discretization. Longer critic training does not reliably improve value decoding, whereas larger critics provide the most consistent gains. Cross-entropy training is competitive with MSE and can reduce variability, while performance is largely insensitive to the number of discretiza- tion bins. Experiments on MATH500 with models ranging from 1.5B to 7B param- eters demonstrate that trained critics can improve test-time reasoning, although the magnitude of these gains depends strongly on critic quality.
الكلمات المفتاحية
غير متوفر
التخصصات
رسائل وأطاريح مشابهة
- Cognitive Behaviours of Large Language Models Bryan Wilie · 2026
- Generating game ideas with large language models Uneeb Kari · Aalto University · 2025
- Improving Code Quality Using Fine-tuning Large Language Models Quang Hieu Nguyen · Tampere University · 2024
- Learning to adapt: From structured domain adaptation to efficient in-context learning for large language models Zihao Xu · Rutgers University · 2026
- On the Safety Alignment of Large Language Models Mengxuan Hu · 2026
- Documenting and Improving Prompts for Large Language Models Amandine M. Caut · Dalarna University Library · 2026
