Meet SWE-Compass: A Real-World Benchmark for Coding AIs

Kari Jaaskelainen

10 Nov 2025 — 1 min read

How good are today’s coding AIs at real developer work—not just toy puzzles? Meet SWE-Compass, a new benchmark that puts large language models (LLMs) through realistic, production-style tasks.

What’s new

Broad coverage: 8 task types across 8 development scenarios and 10 programming languages.
Real data: 2,000 carefully vetted cases sourced from authentic GitHub pull requests.
Agent workflows: Tested under two popular agent frameworks—SWE-Agent and Claude Code.
Wide model sweep: Results across 10 leading LLMs.
Actionable insights: Reveals a clear difficulty ladder by task, language, and scenario.

Why it matters: Many benchmarks overfit to algorithms or Python-only bug fixes. SWE-Compass aligns evaluation with real developer workflows, offering a rigorous, reproducible way to diagnose where coding AIs excel—or stumble—across languages and contexts.

For researchers and teams building AI dev tools, SWE-Compass is a compass for progress. Paper: http://arxiv.org/abs/2511.05459v1

Paper: http://arxiv.org/abs/2511.05459v1

Register: https://www.AiFeta.com

#AI #LLM #SoftwareEngineering #DevTools #Benchmark #OpenSource #GitHub #AgenticAI

Automating GDPR Compliance: A Roadmap for Companies and Law Firms

GDPR compliance is more than checkboxes. A new roadmap from the Privatech project shows how automation and machine learning can help companies and law firms assess—and even generate—privacy compliance. * Shift the focus to data processors’ real workflows: drafting policies, mapping data uses, documenting decisions. * Break compliance into machine-ready

FPGAs for Faster, Leaner Deep Learning: A Review of CNN Accelerators

Deep learning drives image search, robots, and medical scans. Most systems lean on CPUs and GPUs. This review asks: what if we run convolutional neural networks (CNNs) on FPGAs—reconfigurable chips you can tailor to the model? * Why FPGAs: custom dataflows, low latency, and strong energy efficiency—great for cameras,

Dynamic-K: Recommendations That Know When to Stop

Most apps show a fixed number of “top” items—say 10 movies or 20 products—assuming there are always enough good options. But that’s not always true: sometimes there are few relevant items, or some users are extra picky. The result? Filler recommendations. Dynamic-K flips the script. Instead of

Teaching chatbots to stop contradicting themselves (DECODE)

Teaching chatbots to stop contradicting themselves Ever had a bot say one thing, then the opposite a few turns later? This study introduces DECODE—a new task and dataset for spotting contradictions in everyday conversations, drawn from both human-human and human-bot chats. * New data beats existing natural language inference (NLI)

What’s new

Read more

Automating GDPR Compliance: A Roadmap for Companies and Law Firms

FPGAs for Faster, Leaner Deep Learning: A Review of CNN Accelerators

Dynamic-K: Recommendations That Know When to Stop

Teaching chatbots to stop contradicting themselves (DECODE)