Logo image
Back
Prompt-Conditioned Vision-Language Models for Detecting Unseen Backdoor Images
Conference proceeding   Peer reviewed

Prompt-Conditioned Vision-Language Models for Detecting Unseen Backdoor Images

Kyle Stein, Andrew Arash Mahyari, Guillermo Francia and Eman El-Sheikh
Conference record - Asilomar Conference on Signals, Systems, & Computers, pp.351-355
Asilomar Conference on Signals, Systems, and Computers, 59th (Pacific Grove, California, USA, 10/26/2025–10/29/2025)
10/26/2025
DOI: https://doi.org/10.1109/IEEECONF67917.2025.11443859

Metrics

1 Record Views

Abstract

Backdoor attacks pose a critical threat to computer vision by embedding hidden triggers into inputs, causing models to misclassify them into target labels. While extensive research has focused on mitigating these attacks in object recognition models through model cleansing, much less attention has been given to detecting backdoored samples directly. Given the vast datasets used in training, manual inspection for backdoor triggers is impractical, and even state-of-the-art defense mechanisms fail to fully neutralize their impact. To address this gap, we introduce a novel method to detect unseen backdoored image types during both training and inference. Leveraging the transformative success of conditional prompt tuning in Vision Language Models (VLMs), our approach trains learnable text prompt prefixes to differentiate clean images from those with hidden backdoor triggers. Furthermore, we shift the learned prefix based on the image features for each sample through a lightweight, image-conditioned network. Experiments demonstrate the exceptional efficacy of this method, achieving an impressive average accuracy of 84% across two renowned datasets for detecting unseen backdoor triggers, establishing a new standard in backdoor defense.

Details

Logo image