Making LLMs Faster: Fix Memory and Interconnect, Not Just Compute

Kari Jaaskelainen

10 Jan 2026 — 1 min read

Why running LLMs is hard (and how hardware can help)

Large language models don't just need fast math—they need to fetch and share enormous amounts of data, one token at a time. In inference, the autoregressive "decode" phase dominates, making memory and interconnect, not compute, the true bottlenecks.

Trend headwinds: Bigger models, longer context windows, and limited batching all amplify memory pressure and communication costs.
What could fix it:
High-Bandwidth Flash: ~10× more memory capacity with HBM-like bandwidth to keep models close to the chips.
Processing-Near-Memory and 3D memory–logic stacking: bring simple operations to the data and boost on-package bandwidth.
Low-latency interconnects: faster links across accelerators and servers to reduce waiting during decode.

The paper focuses on datacenters, with lessons that can extend to mobile devices as on‑device AI grows.

Bottom line: To make LLMs faster, cheaper, and greener, prioritize memory and networking innovations over more compute.

Paper by Xiaoyu Ma and David Patterson: https://arxiv.org/abs/2601.05047

Paper: https://arxiv.org/abs/2601.05047v1

Register: https://www.AiFeta.com

#AI #LLM #Hardware #Datacenter #Semiconductors #Memory #Interconnect #EdgeAI

Meet GRACE: a moral governor for safer, more transparent AI

AI agents are getting powerful—so how do we make sure they do the right thing, not just the effective thing? Meet GRACE, a reason-based moral governor that keeps AI behavior aligned with human norms by separating moral reasoning from goal-driven decision-making. * Moral Module: uses deontic logic and explicit reasons

Why some fine-tuned LLMs miss phishing—and how to fix it

Not all fine-tuned LLMs spot phishing equally. A new study tests Llama 3.1 8B, Gemma 2 9B, and Mistral on high-stakes phishing detection—and uses SHAP and mechanistic interpretability to reveal why models do (or don’t) generalize. * Architecture × data diversity matters: Gemma 2 9B hits state-of-the-art performance (F1

AI that explains itself—by following the science

AI that explains itself—by following the science Black-box AI can be powerful, but it often can’t tell us why it made a decision. Concept Bottleneck Models (CBMs) try to fix that by predicting human-understandable concepts first, then the final answer. The catch: standard CBMs ignore domain-specific cause-and-effect and

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A

TL;DR: Finding reliable answers in genomics is hard. GenomAgent turns one big AI into a coordinating team of specialists—and beats the current leader by 12% on a key benchmark. Genomic facts live across many databases. Standard chatbots struggle because they can’t flexibly query those sources. GeneGPT added

Why running LLMs is hard (and how hardware can help)

Read more

Meet GRACE: a moral governor for safer, more transparent AI

Why some fine-tuned LLMs miss phishing—and how to fix it

AI that explains itself—by following the science

Meet GenomAgent: A Team of AI Specialists for Smarter Genomics Q&A