LLM

Smarter LLM Pretraining: Beyond URLs

Kari Jaaskelainen

27 Nov 2025 — 1 min read

What’s new

Adding the right metadata can make LLM pretraining faster and more effective—and it’s not just about URLs.

Fine‑grained signals work: Prepending detailed quality indicators helps models learn quicker.
Append-and-predict: Appending metadata and training the model to predict it as an auxiliary task boosts efficiency.
Learnable meta‑tokens: Special tokens trained with masked loss recover part of the speedup by inducing quality‑aware structure.
Common thread: The most useful metadata encode information at finer granularity.
Why it works: Probing shows metadata guides internal representations during learning.

Takeaways

Prefer fine‑grained, trustworthy metadata over coarse labels.
Either prepend metadata or use append‑and‑predict—both can help.
Keep metadata compact to avoid token bloat.

Paper by Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, and Martin Jaggi. Read: https://arxiv.org/abs/2511.21613v1

Paper: https://arxiv.org/abs/2511.21613v1

Register: https://www.AiFeta.com

LLM AIResearch MachineLearning NLP Pretraining Metadata Efficiency

Beyond URLs: Metadata That Makes LLMs Train Faster

Smarter LLMs, Faster—thanks to metadata What if training a large language model didn't just rely on text, but on the context around it? This study shows that adding fine-grained metadata—not just URLs—can meaningfully speed up pretraining and improve quality. * Beyond URLs: detailed quality signals (e.

Escaping the Verifier: Learning to Reason via Demonstrations

LLMs can learn to reason—without task verifiers Many real-world problems don’t have automatic checkers to grade answers, even though we have lots of expert solutions. RARO (Relativistic Adversarial Reasoning Optimization) shows how to train reasoning skills from those examples alone. How it works: * A policy (the model) tries

VacuumVLA: A Two-in-One Robot Hand That Grips and Suctions

Robots guided by Vision-Language-Action (VLA) AI are getting better at everyday tasks—but most still use simple two-finger grippers. That limits them on smooth, flat, or handleless objects. VacuumVLA is a low-cost robot hand that merges a standard two-finger gripper with a vacuum suction cup. The robot can switch between

ChatDRex: No‑Code, Conversational AI for Drug Repurposing

Meet ChatDRex Finding new uses for approved drugs can save years and millions. But making those predictions usually takes teams of specialists and a tangle of tools. ChatDRex changes that. It’s a conversation-based, no‑code, multi‑agent system that lets clinicians and researchers ask complex bioinformatics questions in plain