The Era of Real-World Human Interaction: RL from User Conversations

Kari Jaaskelainen

30 Sep 2025 — 1 min read

RLHI taps in-the-wild conversations to align models to real user preferences and personas.

Most conversational models are aligned with expert-annotated feedback—effective yet limited in scale and personalization. This work proposes Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from organic user conversations, connecting long-term personas to moment-by-moment preferences.

Two complementary methods form RLHI:

User-Guided Rewrites: The model revises unsatisfying outputs using natural-language follow-ups from users, converting free-form feedback into actionable improvements.
User-Based Rewards: A reward model conditions on a user’s long-term interaction history (persona) to score responses, enabling persona-conditioned preference optimization.

Trained on WildChat-derived conversations, both RLHI variants outperform strong baselines on personalization and instruction following. Notably, the same feedback signals also improve reasoning benchmarks—suggesting that organic interaction provides rich, scalable supervision for multifaceted alignment.

Why it matters: Personalization isn’t just a feature—it’s essential for trust and utility. RLHI turns real-world dialogue into a continuous learning loop, allowing systems to adapt to the nuanced, evolving preferences of individuals. By linking personas and turn-level choices, RLHI moves beyond static alignment toward dynamic, human-centered models.

Bottom line: Instead of relying solely on curated labels, RLHI unlocks the value of everyday conversations—at scale—while keeping alignment grounded in what users actually want.

Paper: arXiv: RL from User Conversations
Register: https://www.AiFeta.com

#RLHI #Personalization #Alignment #ConversationalAI #UserModeling #RLHF #NLP #LLM

From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

Evidence that RL teaches genuinely new abilities: compositional skills emerge and transfer across tasks Does RL merely reweight what an LLM already knows—or can it teach genuinely new skills? This paper offers concrete evidence for the latter. Using a controlled, synthetic framework, the authors define “skills” as string transformation

OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing

A structured 80k instruction–image corpus spanning 11 domains and 51 subtasks to train unified visual editors Unified models for image generation and editing hit a data ceiling: existing corpora emphasize basic manipulations but miss real‑world complexity. OpenGPT‑4o‑Image tackles this with a hierarchical task taxonomy and automated

Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards

ROVER replaces PPO loops with uniform‑policy Q‑values—boosting quality and diversity in math reasoning Popular RLVR methods for LLM reasoning lean on generalized policy iteration (e.g., PPO/GRPO), but suffer instability and diversity collapse. This paper reframes math RLVR as a specialized finite‑horizon MDP with deterministic

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning

A dynamic, self‑paced curriculum that restructures problems to match model ability in RLVR Online RL with Verifiable Rewards (RLVR) has boosted LLM reasoning—but most methods treat all problems equally, wasting effort on solved items and flailing on those beyond current capability. CLPO fixes that with a dynamic pedagogy:

Read more

From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing

Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning