Curiosity-based Reinforcement Learning

The thesis initially focused on the study of the main Reinforcement Learning methods, highlighting their principal limitations and their temporal evolution. Subsequently, the sparse reward problem was analyzed in detail, which is associated with the lack of updates of the current policy due to the low frequency of feedback provided by the environment. To address this issue, several curiosity-based methods proposed in the literature were examined. Among these, two different methodologies were implemented and experimentally compared with the selected Reinforcement Learning algorithm in applications characterized by sparse rewards and increasing levels of complexity. Three different types of tasks were therefore considered: the control of an inverted pendulum, a manipulation task based on visual observations only, and a navigation task in the presence of obstacles, which combines visual observations with other types of sensory information. In the first experiment, it was observed that the integration of curiosity modules does not provide significant benefits, while incurring a higher computational cost. In the second experiment, instead, the contribution of the modules appears to be more evident. It was therefore concluded that curiosity-based methods are determinant for achieving a performance improvement in the specific application scenario. In the final experiment, it was observed that the introduction of multimodal observations led to a marked deterioration in the performance of one of the two methods, mainly due to the higher sensitivity of the module to visual observations. This behavior can be attributed to a component fixed at the beginning of the training process, which is unable to adapt the relative importance of the received observations. Conversely, the second method updates the internal feature extractor of the model, thereby avoiding this limitation and ensuring, also in this case, superior performance compared to the other evaluated methods.