Attribute-based analysis of zero-shot learners
الملخص
Vision-language models trained on large, randomly collected data had a significant impact in many areas since they appeared. But although they show great performance in various fields, such as image-text-retrieval, their inner workings are still not fully understood. An interesting application of those models is the zero-shot scenario that is analyzed in this work. A new benchmark consisting of five experiment setups is proposed to evaluate how zero-shot learners handle class labels and attributes. The experiments are divided into two sections, the classification performance with class label containing the first three experiment setups and the attribute based classification performance containing the last two experimental setups. The suggested experiments rely on attribute-based datasets with attribute annotations per image. Since only a few datasets with image attribute annotations exist, this work proposes a method to generate image attribute annotations. For a dataset that contains attribute annotations for classes, image attribute annotations are created with the help of a vision-language (VL) model. To validate if current zero-shot learners are really capable of zero-shot, an occurrence analysis is suggested. The recently released LAION-400M data corpus and the publicly available OpenCLIP model trained on LAION-400M are leveraged and ImageNet, AWA2, and CUB are used as evaluation datasets. For each class of an evaluation dataset, the training samples corresponding to the evaluation class are counted and the correlation between the model performance and class occurrences in the training data is provided. Using the assigned image attributes for the AWA2 and the CUB dataset, the proposed benchmark is applied to three publicly available VL models (OpenCLIP, CLIP, FLAVA). With a different prompt generation technique, the five experiment setups are repeated for AWA2 which leads to similar results and indicates their robustness. The provided analysis shows that: • Most of the classes in popular zero-shot benchmarks are observed (a lot) during pretraining. • Zero-shot performance mainly comes out of models’ capability of recognizing class labels whenever they are present in the text, and a significantly lower performing capability of attribute-based zero-shot learning is only observed when class labels are not used. • The number of attributes used can have a significant effect on performance, and can easily cause a significant performance decrease.
الكلمات المفتاحية
غير متوفر
التخصصات
رسائل وأطاريح مشابهة
- Vision–Language Models for Zero- and Few-Shot Anomaly Detection in Chest X-Rays Md Ashrafuzzaman · University of Jyväskylä · 2026
- Inductive and Transductive Learning with Vision-Language Models in Low-Shot Regimes Maxime Zanella · 2026
- Adaptive Deep Learning for Vision and Multimodal Understanding with Heterogeneous Data Youwei Liang · 2026
- AI-Generated Image Detection Using Multimodal Deep Learning Mamadou Keïta · 2026
- Addressing data heterogeneity, scarcity, and training efficiency in privacy-preserving federated learning Chen, Haokun · Ludwig-Maximilians-Universität München · 2025
- Advancing Computational Pathology with Multimodal Deep Learning Md. Mamunur Rahaman · 2025
