Not stated in source 2025 Needs review

Attribute-based analysis of zero-shot learners

Visit original source

Abstract

Vision-language models trained on large, randomly collected data had a significant impact in many areas since they appeared. But although they show great performance in various fields, such as image-text-retrieval, their inner workings are still not fully understood. An interesting application of those models is the zero-shot scenario that is analyzed in this work. A new benchmark consisting of five experiment setups is proposed to evaluate how zero-shot learners handle class labels and attributes. The experiments are divided into two sections, the classification performance with class label containing the first three experiment setups and the attribute based classification performance containing the last two experimental setups. The suggested experiments rely on attribute-based datasets with attribute annotations per image. Since only a few datasets with image attribute annotations exist, this work proposes a method to generate image attribute annotations. For a dataset that contains attribute annotations for classes, image attribute annotations are created with the help of a vision-language (VL) model. To validate if current zero-shot learners are really capable of zero-shot, an occurrence analysis is suggested. The recently released LAION-400M data corpus and the publicly available OpenCLIP model trained on LAION-400M are leveraged and ImageNet, AWA2, and CUB are used as evaluation datasets. For each class of an evaluation dataset, the training samples corresponding to the evaluation class are counted and the correlation between the model performance and class occurrences in the training data is provided. Using the assigned image attributes for the AWA2 and the CUB dataset, the proposed benchmark is applied to three publicly available VL models (OpenCLIP, CLIP, FLAVA). With a different prompt generation technique, the five experiment setups are repeated for AWA2 which leads to similar results and indicates their robustness. The provided analysis shows that: • Most of the classes in popular zero-shot benchmarks are observed (a lot) during pretraining. • Zero-shot performance mainly comes out of models’ capability of recognizing class labels whenever they are present in the text, and a significantly lower performing capability of attribute-based zero-shot learning is only observed when class labels are not used. • The number of attributes used can have a significant effect on performance, and can easily cause a significant performance decrease.

Keywords

Not available

Specialisations

Similar theses