Bias scores shift with context: Contextual StereoSet
Models that seem fair in the lab can slip in the wild. Contextual StereoSet shows how measured bias swings when you change the framing—no adversarial prompting required.
- Same stereotypes, new frames: Hold content constant, vary time, place, or audience.
- Striking shifts: Anchoring to 1990 (vs. 2030) raised stereotype choices in all tested models (p<0.05). Gossip framing raised them in 5/6 models. Out-group observer framing shifted rates by up to 13 percentage points.
- Across domains: Effects replicate in hiring, lending, and help-seeking vignettes.
- Quick or deep: A 360-context diagnostic grid for deep dives, and a budgeted protocol covering 4,229 items for production screening.
- CSF profiles: Context Sensitivity Fingerprints summarize how a model’s bias score disperses across contexts, with bootstrap CIs and FDR-corrected contrasts.
The takeaway: stop asking Is this model biased? Start asking Under what conditions does bias appear? It’s a robustness stress test, not a claim about ground-truth bias rates. Code, benchmark, and results: https://arxiv.org/abs/2601.10460v1
Paper: https://arxiv.org/abs/2601.10460v1
Register: https://www.AiFeta.com
#AI #MachineLearning #LLM #NLP #AIEthics #ResponsibleAI #Bias #Evaluation
