Multimodal and explainable Learning using Fusion Architecture Through Exploration (FATE) approach

The way we experience the world is multimodal, each of our five senses perceives the world differently, yet the brain can explore this information at once with ease. This work explores the potential of mimicking this exceptional cognitive capability in machines. This thesis lies in this research area, called Multimodal Deep Learning, where we spotlight the importance of robust fusion methods and effective modality representations in real-world scenarios. This study concludes with a proposed light, easy-to-use, and extendable fusion method called MultConcat, that was evaluated for multiple data source combinations: image and text, RGB and depthmap frames (video flows), as well as an MRI and tabular data in a medical context. We then propose FATE (Fusion Architecture Through Exploration), a methodology that treats a given input multimodal problem by solving real-world research questions related to the fusion method, the appropriate fusion timing, and the best encoder for each modality. Furthermore, we propose a concept-based explainability approach for multimodal settings based on segmentations using Foundation models to ensure human interpretability. Finally, we explore the difference in model design we might have between ready-to-use Vision-Language Models, Multimodal Large Language Models, and configurable models that can be built using our suggested FATE methodology.