Contributions to reinforcement learning with function approximators

Reinforcement Learning tackles sequential decision making problems under uncertainty. It has proven to be especially relevant for problems involving hard to model dynamics or large state-action spaces, where classical approaches from the dynamic programming literature fail to scale. A key component to scale reinforcement learning to such complex problems is to use function approximators such as neural networks. However, adding function approximators to reinforcement learning can introduce instabilities to an already complex training process, due to errors in policy evaluation or update. In this manuscript, we present several approaches leveraging entropy regularization to average out errors in the policy evaluation step and discuss their theoretical and empirical merits. Because entropy regularization promotes stochastic behaviors, using it as is for data gathering can be ill-suited for applications such as robotics. As such, we round off the manuscript with a discussion on episodic exploration and Bayesian policy search, proposing alternative data gathering policies that are in some cases more appropriate for real world applications.