Multimodal Integration for Natural Language Classification and Generation
الملخص
Multimodal integration is a framework for building models that can accept information from different types of modalities. Due to the recent success in the Transformer model and Pre-training Fine-tuning Techniques, Vision-and-Language Pre-training Models have been heavily investigated and they achieved State-of-the-Art in various of Vision-and-Language downstream tasks, such as Visual Question Answering, Image Text Matching and Image Captioning. However, most of the previous studies focus on improving the performance of the models and only provide accessible code for research purposes. There are several existing open-source libraries such as Natural Language Toolkit, OpenCV and HuggingFace, which combine and standardise the available models and tools for easy access, but applying these libraries still requires expertise in both Deep Learning and programming. Moreover, there has been no recent research aimed at establishing user-friendly multimodal question-answering platforms for non-deep-learning users. Therefore, the question of how State-Of-The-Art multimodal models can be easily applied by professionals in other domains remains open. Apart from the first challenge, there exists another challenge in the less-common domain. Since general multimodal domains such as street view, landscape, and indoor scenes have been extensively studied with current VL-PMs, while specific domains like medicine, geography, and esports have garnered less attention. Due to the difficulties in data collection, there aren't many publicly available multimodal datasets, and those that exist tend to be small. This scarcity poses challenges for model training. Consequently, the question of how to collect a comprehensive multimodal dataset in the esports domain and how to improve domain-specific multimodal models remains open. Therefore, the main focus of this thesis is integrating multimodal information for natural language classification and generation tasks by addressing the two challenges.
الكلمات المفتاحية
غير متوفر
التخصصات
رسائل وأطاريح مشابهة
- Adaptive Deep Learning for Vision and Multimodal Understanding with Heterogeneous Data Youwei Liang · 2026
- Learning semantic representations of language, vision and code for visualization-oriented dialogue Henrik Voigt · 2026
- Deep learning and natural language processing for computer aided diagnosis Sadam Hussain · Tecnológico de Monterrey · 2025
- Inductive and Transductive Learning with Vision-Language Models in Low-Shot Regimes Maxime Zanella · 2026
- AI-Generated Image Detection Using Multimodal Deep Learning Mamadou Keïta · 2026
- A Study on Robustness, Adaptation, and Generalization for Visual and Multimodal Learning Xiasi Wang · 2026
