Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification
Shows prompt reformulation destabilizes VLM safety classifier scores
Single-prompt first-token probabilities from zero-shot vision-language model (VLM) safety classifiers are treated as decision scores, but we show they are unreliable under semantically equivalent prompt reformulation: even when the binary label is constrained to a fixed output position, equivalent prompts can induce materially different unsafe probabilities for the sam…
