Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation

Gender bias in vision-language foundation models (VLMs) raises concerns about their safe deployment and is typically evaluated using benchmarks with gender annotations on real-world images. However, as these benchmarks often contain spurious correlations between gender and non-gender features, such as objects and backgrounds, we identify a critical oversight in gender bias evaluation: Do confounding features distort gender bias evaluation? To address this question, we systematically perturb non-gender features across four widely used benchmarks (COCO-gender, FACET, MIAP, and PHASE) and various VLMs to quantify their impact on bias measurements. Our findings reveal that even minimal perturbations, such as masking just 10% of objects or weakly blurring backgrounds, can dramatically alter bias scores, shifting metrics by up to 175% in generative VLMs and 43% in CLIP variants. This suggests that current bias evaluations often reflect model responses to confounders rather than true gender bias, undermining their reliability. Since creating confounder-free benchmarks is fundamentally challenging, we recommend reporting bias metrics alongside confounder-sensitivity measurements to enable a more reliable assessment of gender bias in VLMs.