Bias scores shift with context: Contextual StereoSet

Kari Jaaskelainen

16 Jan 2026 — 1 min read

Models that seem fair in the lab can slip in the wild. Contextual StereoSet shows how measured bias swings when you change the framing—no adversarial prompting required.

Same stereotypes, new frames: Hold content constant, vary time, place, or audience.
Striking shifts: Anchoring to 1990 (vs. 2030) raised stereotype choices in all tested models (p<0.05). Gossip framing raised them in 5/6 models. Out-group observer framing shifted rates by up to 13 percentage points.
Across domains: Effects replicate in hiring, lending, and help-seeking vignettes.
Quick or deep: A 360-context diagnostic grid for deep dives, and a budgeted protocol covering 4,229 items for production screening.
CSF profiles: Context Sensitivity Fingerprints summarize how a model’s bias score disperses across contexts, with bootstrap CIs and FDR-corrected contrasts.

The takeaway: stop asking Is this model biased? Start asking Under what conditions does bias appear? It’s a robustness stress test, not a claim about ground-truth bias rates. Code, benchmark, and results: https://arxiv.org/abs/2601.10460v1

Paper: https://arxiv.org/abs/2601.10460v1

Register: https://www.AiFeta.com

#AI #MachineLearning #LLM #NLP #AIEthics #ResponsibleAI #Bias #Evaluation

Meet GRACE: a moral governor for safer, more transparent AI

AI agents are getting powerful—so how do we make sure they do the right thing, not just the effective thing? Meet GRACE, a reason-based moral governor that keeps AI behavior aligned with human norms by separating moral reasoning from goal-driven decision-making. * Moral Module: uses deontic logic and explicit reasons

Why some fine-tuned LLMs miss phishing—and how to fix it

Not all fine-tuned LLMs spot phishing equally. A new study tests Llama 3.1 8B, Gemma 2 9B, and Mistral on high-stakes phishing detection—and uses SHAP and mechanistic interpretability to reveal why models do (or don’t) generalize. * Architecture × data diversity matters: Gemma 2 9B hits state-of-the-art performance (F1

AI that explains itself—by following the science

AI that explains itself—by following the science Black-box AI can be powerful, but it often can’t tell us why it made a decision. Concept Bottleneck Models (CBMs) try to fix that by predicting human-understandable concepts first, then the final answer. The catch: standard CBMs ignore domain-specific cause-and-effect and

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A

TL;DR: Finding reliable answers in genomics is hard. GenomAgent turns one big AI into a coordinating team of specialists—and beats the current leader by 12% on a key benchmark. Genomic facts live across many databases. Standard chatbots struggle because they can’t flexibly query those sources. GeneGPT added

Read more

Meet GRACE: a moral governor for safer, more transparent AI

Why some fine-tuned LLMs miss phishing—and how to fix it

AI that explains itself—by following the science

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A