AI
How do we know if an AI model is truly aligned with human preferences—helpful, honest, safe, and instruction-following? This paper explores a surprisingly effective shortcut: judge the judges. Instead of grading a model’s open-ended answers (which needs lots of human effort or very strong AI judges), the authors
AI
LLMs can write tidy code that passes unit tests—but can they build money-making agents that plan, bid, and deliver under pressure? This study pitted 40 LLM-coded agents (prompted with methods including “vibe coding,” i.e., high-level guidance) against 17 human-coded agents from grad CS students in a real-world-style challenge:
AI
Ever notice how some AIs agree with a confident (but wrong) authority? That’s sycophancy. PARROT is a new benchmark that tests how much language models bend under authority and persuasion. * How it works: The same question is asked two ways—neutral vs. with an authoritative (but wrong) cue—scored
AI
Imagine telling your phone what you want—and an AI taps, types, and swipes to get it done. This paper introduces DigiData, a large, diverse, multi‑modal dataset designed to train general‑purpose mobile control agents. Instead of relying on unstructured user logs, the team systematically explores app features to
AI
Can AI really make trading decisions in the messy, fast-changing real world? LiveTradeBench puts large language models to the test with live market data. * Live streams: Real-time prices and news—no offline backtests or leakage. * Portfolio view: Multi-asset allocation with built-in risk management. * Multi-market: U.S. stocks and Polymarket prediction
AI
AI agents promise to speed up discovery by doing the messy parts of research—forming hypotheses, writing code, running experiments, and analyzing results. But do they actually handle end-to-end projects? InnovatorBench is a new benchmark+platform that tests agents on realistic Large Language Model (LLM) research workflows. * 20 tasks across