Multimodal Deep Learning for Driver Monitoring: Integrating EEG and Vision for Robust Drowsiness Detection and Safety Enhancement
Abstract
Road accidents remain a major global concern, with driver drowsiness and delayed reaction times recognized as key contributing factors. This thesis advances driver-monitoring research by developing multimodal approaches that integrate electroencephalography (EEG) and vision data to predict reaction times and drowsiness. The investigation first demonstrates that pre-stimulus EEG signals—specifically spectral power in the alpha and theta bands—contain rich information for estimating reaction times to critical road events. Using subject-independent machine-learning pipelines, short EEG windows recorded before event onset effectively differentiate between fast and slow responses. The work then explores the benefits of incorporating vision data as a second modality by fusing EEG signals with camera-based observations of the driver. One branch converts EEG power-spectral-density features into image-like representations for analysis with deep convolutional neural networks and transformer models. Another branch directly integrates raw EEG signals with synchronised video frames through end-toend multimodal transformer architectures. Results indicate that transformers equipped with cross-modal attention capture complex interdependencies between neural and visual cues, yielding significant improvements in driver-drowsiness detection over unimodal approaches. Real-time deployment is addressed by designing and optimising a lightweight pipeline for edge-based processing. This resource-efficient model enables rapid analysis of facial cues under diverse driving conditions, ensuring operation on embedded platforms such as smartphones and automotive edge devices. Extensive evaluations on large-scale simulated datasets confirm the generalisability of the proposed approaches across varied driving scenarios. Experiments reveal that transformer-based fusion significantly enhances predictive performance by effectively combining complementary neural and visual cues. Moreover, the lightweight pipeline maintains high accuracy under stringent computational constraints, enabling real-time, ondevice deployment.
Keywords
Not available
Specialisations
Similar theses
- Advancing heat stress detection in dairy cows through machine learning and computer vision Shu, Hang · University of Liège · 2024
- A Real-Time and Automatic Ultrasound-Enhanced Multimodal Second Language Training System: A Deep Learning Approach Mozaffari Maaref · University of Ottawa · 2020
- Adaptive Deep Learning for Vision and Multimodal Understanding with Heterogeneous Data Youwei Liang · 2026
- Transformers and contrastive semi-supervised learning for medical image segmentation Qianying Liu · The University of Glasgow · 2026
- A Novel Explainable Artificial Intelligence Framework with Improved Learning Mechanisms Kabir, Sami · Luleå University of Technology · 2025
- An efficient deep learning approach to detect plant diseases using image data Ahmad Anis ; Babli Saha ; Maisa Tarannum Srizee · BRAC University · 2025
