Adversarial signals against discriminative and generative models for data protections

In recent years, discriminative and generative models have been widely adopted across a broad range of applications—including classification, detection, image synthesis, and editing—significantly enhancing productivity and creativity in various domains. However, their proliferation also introduces critical societal challenges such as copyright infringement, misinformation, and privacy violations. Protecting visual data from unauthorized manipulation and model training has thus become an urgent and essential task. Although adversarial attacks have been explored as protection strategies, achieving transferable and generalizable attacks remains a major open challenge due to the diversity of model architectures and task-specific training procedures. In response, this thesis systematically analyzes the mechanisms underlying discriminative and generative models and proposes a series of attack and defense strategies aimed at either protecting visual data or exposing the vulnerabilities of existing methods to inform future research. For discriminative models, data availability attacks (DAAs) have been widely adopted to protect copyrighted data against unauthorized training, by poisoning training data to degrade the performance of downstream models. However, existing DAAs often suffer from limited robustness or poor image quality. To address this, we introduce Imperfect Restoration Poisoning (IRP), a novel poisoning technique designed for supervised learning (SL), which achieves strong attack performance while preserving high image fidelity. Through extensive comparisons of IRP with eight baselines, coupled with evaluations alongside five representative defense methods, we showcase the superiority of IRP. In addition to proposing improved DAAs, we investigate the defense side, especially how to mitigate the impact of data poisoning. While Self-Supervised Learning (SSL) has been considered a promising paradigm to mitigate poisoning due to its objective mismatch with supervised attacks, our experiments across multiple poison types and datasets (CIFAR-10 and ImageNet-100) reveal that SSL often performs poorly when trained on poisoned data. To address this and further expose the limit of existing DAAs, we introduce VESPR (Vulnerability Exploitation of Supervised Poisoning for Robust SSL), which introduce adversarial training on SL to obfuscate poison features and guide robust feature learning for SSL. VESPR outperforms six existing defenses across seven poisoning strategies, revealing a research gap in understanding and improving poisoning robustness in discriminative models. Beyond classification models, this thesis also investigates visual data protection against unauthorized training in generative models, particularly with diffusion models (DMs) fine-tuned via techniques such as Textual Inversion (TI). We identify critical vulnerabilities in existing generative DAAs through a series of analyses. First, we propose Semantic Sensitivity Maps (SSM) to visualize the influence of poisoning on token-level text embeddings during TI training. Second, we observe that DMs exhibit non-uniform learning behavior across timesteps, with DAAs primarily injecting poison signals at early (low-noise) stages—leaving high-timestep stages insufficiently defended. Third, we show that poisoned signals shift attention away from semantically relevant regions, impairing embedding quality. Based on these insights, we introduce Safe-Zone Training (SZT), a novel training strategy that mitigates poisoning in DMs. SZT comprises 3 key components: (1) JPEG compression to weaken high-frequency poison signals, (2) restriction to higher timesteps during textual inversion training to avoid adversarial signal at lower timesteps, and (3) loss masking to constrain learning to target regions. Our results demonstrate that SZT effectively neutralizes several state-of-the-art DAAs, revealing a significant research gap for poisoning on generative models in the future. Apart from training-time protection, we extend the study to inference-time adversarial attacks to prevent AI-driven visual manipulations. In some tasks, such as visual watermarking removal, some prior adversarial attack methods are typically constrained to specific types of target models and operate in white-box settings, limiting their applicability. Within the context of visual watermarking removal, multiple studies have revealed that watermarks are vulnerable to both inpainting-based removers and blind watermark removers. In the thesis, a novel black-box method is proposed, named Adversarial Watermark Defender with Attribution-Guided Perturbation (AWD-AGP), which is the first attack defending against both types of watermark removers. The key innovation for superior transferability lies in leveraging the mechanistic differences between inpainting-based and blind removal models.