Multimodal Deep Learning for Zero-Shot Action Recognition

Advancements in machine learning have improved the performance of machine learning (ML) models.However, larger training datasets present challenges, such as prolonged labeling and training times.One potential solution is zero-shot learning, which enables models to identify unfamiliar entities without extensive training data.This not only saves time and effort but also allows easy integration of new categories.However, its efficacy requires enhancement for broad real-world applications.This paper presents three methods to optimize the accuracy of zero-shot action recognition (ZSAR).1. Three-stream graph convolutional networks (TS-GCN):Current models often handle only one data type, limiting their information leverage.Our proposed two-stream network processes both RGB image data and optical flow data, which captures object movement.Each branch calculates video classification separately, and a weighted average determines the final classification.2. Improving ZSAR using human instruction with text description: The video-text matching model (TVMM) model gauges the matching degrees between text features and video content.We annotated text features for each action, and the TVMM model used these for action recognition.The results showed improved action recognition accuracy.3. Zero-shot action recognition with ChatGPT-based instruction: Manual text feature annotation improved accuracy but at a labor cost.To alleviate this, we used ChatGPT for automated text feature generation.This eliminated the need for human annotation.While achieving high recognition accuracy with reduced labor costs, it was slightly lower than previous methods but still suitable for various real-life applications.We propose three innovative methods that optimize zero-shot action recognition.The experiments demonstrate the effectiveness of our approach.