Achieving Robust Autonomous Driving through Multi-modality Deep Reinforcement Learning

Autonomous driving systems rely heavily on sensor inputs to perceive and navigate their environment.However, individual sensors such as cameras or LiDAR often experience performance degradation in adverse weather conditions, which can lead to unreliable perception and unsafe control decisions.This limitation presents a critical challenge for deep reinforcement learning (DRL)-based driving agents, which typically depend on a single sensor modality.To address this issue, we propose a multi-modal DRL framework that integrates camera and LiDAR observations to achieve robust autonomous driving across varying weather conditions.The proposed method trains two independent DRL agents in the CARLA simulator: one that uses camera images and another that uses LiDAR point-cloud data.A shared reward mechanism encourages cooperative learning, while a fuzzy-inspired Rule-Based Action Fusion model combines the actions of both agents using weatherconditioned weighting.The fusion ratios are tuned for different weather conditions to improve reliability.We evaluate the proposed framework under three conditions -sunny, rainy, and foggy-and compare it with camera-only and LiDAR-only baselines trained under the same settings.Experimental results demonstrate that the multi-modal fusion approach significantly outperforms single sensor agents across all weather scenarios.In sunny conditions, the fusion model achieves an average reward of 11.99, compared with 2.07 and 8.25 for the camera and LiDAR agents, respectively.In rain and fog, where sensor degradation is more pronounced, the fusion model maintains strong performance with rewards of 10.64 and 11.51, while single-modality agents show substantial declines.These findings demonstrate that combining complementary sensor modalities can improve the stability, safety, and reliability of autonomous driving systems.This research contributes to the development of robust multi-sensor DRL methods capable of operating under diverse real-world conditions.