Not stated in source 2026 Needs review

Value estimation in Large Language Models

Visit original source

Abstract

Large language models (LLMs) are commonly post-trained with reinforcement learning to improve reasoning performance. A central design choice in this process is whether to estimate advantages using a learned value function or directly from Monte Carlo returns. Value-based estimators can reduce variance and improve credit assignment but require training an additional critic, whereas Monte Carlo estima- tors are simpler and avoid this computational cost. This thesis studies the utility of value models during both training and inference. We first conduct controlled com- parisons between value-based and Monte Carlo estimators across models, tasks, and optimization settings. No estimator is uniformly superior: Monte Carlo methods can achieve competitive peak performance, while PPO often provides stronger average performance or lower sensitivity to configuration selection. Because surrounding op- timization choices can have a larger effect than the estimator, learning a critic does not consistently yield a better final policy. We then introduce value decoding, a test-time generation method that reuses the trained critic as a process supervisor. At each decoding step, the actor generates multiple candidate sub-trajectories, the critic scores them, and the highest-valued candidate is selected to continue the response. Across our experiments, value de- coding consistently improves upon the actor’s pass@1 performance. To understand 4 what determines its effectiveness, we systematically study the influence of additional critic training, critic size, training objective, and value-target discretization. Longer critic training does not reliably improve value decoding, whereas larger critics provide the most consistent gains. Cross-entropy training is competitive with MSE and can reduce variability, while performance is largely insensitive to the number of discretiza- tion bins. Experiments on MATH500 with models ranging from 1.5B to 7B param- eters demonstrate that trained critics can improve test-time reasoning, although the magnitude of these gains depends strongly on critic quality.

Keywords

Not available

Specialisations

Similar theses