When Leaderboards Mislead: Annotation Errors in Text-to-SQL

Kari Jaaskelainen

14 Jan 2026 — 1 min read

Leaderboards drive Text-to-SQL progress—but what if the test sets are wrong? This study audits two popular benchmarks and finds widespread annotation errors that can flip who looks best.

Error rates: 52.8% in BIRD Mini-Dev, 62.8% in Spider 2.0-Snow.
After correcting a BIRD Dev subset, open-source agents saw relative performance shifts from -7% to +31% and rank changes of -9 to +9.
Rankings on the uncorrected subset correlate strongly with the full Dev set (Spearman r_s=0.85), but weakly after correction (r_s=0.32)—evidence that errors distort results.

Why it matters: Companies and researchers use these leaderboards to pick systems for real deployments. If annotations are wrong, we may reward shortcuts and overlook robust methods.

Takeaway: Treat Text-to-SQL leaderboards with caution. Audit and correct annotations, report uncertainty, and prefer evaluations with verified labels.

Paper: https://arxiv.org/abs/2601.08778v1 • Code/data: https://github.com/uiuc-kang-lab/text_to_sql_benchmarks

Paper: https://arxiv.org/abs/2601.08778v1

Register: https://www.AiFeta.com

AI NLP TextToSQL Databases MachineLearning Benchmarking Reproducibility Leaderboards

Meet GRACE: a moral governor for safer, more transparent AI

AI agents are getting powerful—so how do we make sure they do the right thing, not just the effective thing? Meet GRACE, a reason-based moral governor that keeps AI behavior aligned with human norms by separating moral reasoning from goal-driven decision-making. * Moral Module: uses deontic logic and explicit reasons

Why some fine-tuned LLMs miss phishing—and how to fix it

Not all fine-tuned LLMs spot phishing equally. A new study tests Llama 3.1 8B, Gemma 2 9B, and Mistral on high-stakes phishing detection—and uses SHAP and mechanistic interpretability to reveal why models do (or don’t) generalize. * Architecture × data diversity matters: Gemma 2 9B hits state-of-the-art performance (F1

AI that explains itself—by following the science

AI that explains itself—by following the science Black-box AI can be powerful, but it often can’t tell us why it made a decision. Concept Bottleneck Models (CBMs) try to fix that by predicting human-understandable concepts first, then the final answer. The catch: standard CBMs ignore domain-specific cause-and-effect and

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A

TL;DR: Finding reliable answers in genomics is hard. GenomAgent turns one big AI into a coordinating team of specialists—and beats the current leader by 12% on a key benchmark. Genomic facts live across many databases. Standard chatbots struggle because they can’t flexibly query those sources. GeneGPT added

Read more

Meet GRACE: a moral governor for safer, more transparent AI

Why some fine-tuned LLMs miss phishing—and how to fix it

AI that explains itself—by following the science

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A