VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing

Kari Jaaskelainen

29 Sep 2025 — 1 min read

10,497 examples, 13 tasks: a holistic yardstick for voice-first multimodal assistants.

Voice assistants are rapidly evolving into multimodal agents that must hear, speak, and see. Yet evaluation has lagged behind capability. VoiceAssistant-Eval fills this gap with a comprehensive benchmark of 10,497 curated examples across 13 task categories, spanning natural sounds, music, spoken dialogue (listening); multi-turn and role-play imitation (speaking); and heterogeneous images (viewing).

Twenty-one open-source models and GPT-4o-Audio are assessed for response content, speech quality, and cross-modal consistency. Three key findings emerge: (1) proprietary models do not universally dominate; (2) most models speak well but struggle with audio understanding; and (3) carefully designed smaller models can rival much larger ones. Notably, Step-Audio-2-mini (7B) more than doubles the listening accuracy of LLaMA-Omni2-32B-Bilingual.

The benchmark also surfaces hard problems: audio+visual joint reasoning and role-play voice imitation remain challenging. Robustness and safety alignment gaps persist, underscoring the need for evaluation that captures real-world edge cases and user expectations.

Why it matters: developers can finally compare systems apples-to-apples across modalities, pinpoint failure modes, and prioritize training investments. For product teams, VoiceAssistant-Eval provides measurable targets for improvements in listening comprehension, speech naturalness, and multimodal grounding.

Code and data will be released, creating a shared platform to drive the next generation of voice-first AI.

Paper: http://arxiv.org/abs/2509.22651v1

Register: https://www.AiFeta.com

#AI #Speech #Multimodal #Benchmark #LLM #Audio #ComputerVision

From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

Evidence that RL teaches genuinely new abilities: compositional skills emerge and transfer across tasks Does RL merely reweight what an LLM already knows—or can it teach genuinely new skills? This paper offers concrete evidence for the latter. Using a controlled, synthetic framework, the authors define “skills” as string transformation

OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing

A structured 80k instruction–image corpus spanning 11 domains and 51 subtasks to train unified visual editors Unified models for image generation and editing hit a data ceiling: existing corpora emphasize basic manipulations but miss real‑world complexity. OpenGPT‑4o‑Image tackles this with a hierarchical task taxonomy and automated

Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards

ROVER replaces PPO loops with uniform‑policy Q‑values—boosting quality and diversity in math reasoning Popular RLVR methods for LLM reasoning lean on generalized policy iteration (e.g., PPO/GRPO), but suffer instability and diversity collapse. This paper reframes math RLVR as a specialized finite‑horizon MDP with deterministic

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning

A dynamic, self‑paced curriculum that restructures problems to match model ability in RLVR Online RL with Verifiable Rewards (RLVR) has boosted LLM reasoning—but most methods treat all problems equally, wasting effort on solved items and flailing on those beyond current capability. CLPO fixes that with a dynamic pedagogy:

Read more

From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing

Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning