Low-shot Learning based on Pre-trained Model

In the field of computer vision, deep learning has demonstrated remarkable accomplishments. However, the dependence on extensive annotated datasets poses significant challenges, particularly when such data is scarce, costly, or ethically sensitive. This thesis addresses the crucial issue of low-shot learning by aiming to equip learning models with the capability to generalize effectively from limited examples. It explores various low-supervision methodologies, including self-supervised, unsupervised, semi-supervised, and active learning, with the goal of reducing the reliance on labeled data. One potential solution to this challenge is leveraging pre-trained models trained on large-scale datasets and transferring them to downstream tasks and unknown classes. Nevertheless, directly applying pre-trained models in a frozen state often fails to meet the requirements of specific tasks. Fine-tuning these models on base classes with limited training samples and then transferring them to novel domains presents obstacles, as it may disrupt the general knowledge encoded in the model, leading to overfitting on the base classes and suboptimal performance on unseen categories. The central focus of this research is to fully exploit the knowledge encoded in large pre-trained models and enhance their capacity to transfer to novel classes. This thesis introduces a systematic approach to balance generalization and task-specific adaptation, ensuring that fine-tuning preserves pre-trained knowledge while effectively adapting to new data. These models encapsulate extensive knowledge from diverse data sources, enabling swift adaptation to new tasks and classes with minimal additional training. The downstream tasks investigated in this doctoral work range from the low-shot problem for few-shot image classification to more intricate endeavors such as zero-shot and few-shot pixel-level semantic segmentation. The exploration encompasses various pre-trained model architectures, including convolutional and transformer-based models, spanning both pure vision and cross-modal vision-language paradigms. By harnessing the representational capabilities of state-of-the-art pre-trained models, this research aims to push the boundaries of what is achievable with limited data. The novel methodologies proposed strive to maximize performance, maintain generalization across domains, and improve inference efficiency, paving the way for broader adoption in low-data scenarios.