Jet-Nemotron: Searching for the Best Attention Architecture
DeltaNet + Hardware-aware Search
RWKV, Mamba-2, DeltaNet, and others replace quadratic-time attention with linear-time mechanisms. This unlocks million-token contexts without blowing up memory.
However, purely “linear-attention” models often underperform strong, full self-attention baselines at scale. The emerging remedy is hybrid designs that mix linear and full attention. Jamba, Nemotron-H, and, more recently, Qwen3-Next are notable successes.
But which hybrid works best? Designing these models raises several hard questions:
Which linear blocks? Mamba, DeltaNet, RWKV, …?
How often to insert full attention? Each self-attention layer improves quality but hurts efficiency (latency, max context, memory).
What hyperparameters fit your hardware? Heads, dimensions, sequence scheduling, etc.
NVIDIA’s Jet-Nemotron framework automatically searches over hybrid architectures and hyperparameters to find high-performing, hardware-aware designs. Compared to prior work, it’s cheaper to run and discovers models that can outperform full self-attention peers while being more efficient.
What’s in this article:
A quick refresher on the most popular alternatives to full self-attention.
A deep dive into Jet-Nemotron: how the search works, and why it’s cost-effective.
Results and caveats (this is NAS, after all): when the approach shines, and where to be cautious.