The Salt - Curated AI

The Salt - Curated AI

Home
Notes
AI Notebooks
AI Repositories
Related Articles
deep dive
Archive
About

deep dive

Gemma 4: A Technical Look at the Architecture and Training
An efficient global-local attention architecture
 • Benjamin Marie
DeepSeek-V4: The Interesting Part Is the Attention Architecture
CSA, HCA, shared KV, mHC, ... How to make a good and efficient model with 1 million tokens in context
 • Benjamin Marie
A Review of GLM-5: 744B-Parameter LLM Built for 200K Context and Agentic Training
With Sparse and Multi-Head Latent Attention
 • Benjamin Marie
Qwen3-VL: DeepStack Fusion, Interleaved-MRoPE, and a Native 256K Interleaved Context Window
Understanding why Qwen3-VL are the best VLMs
 • Benjamin Marie
Jet-Nemotron: Searching for the Best Attention Architecture
DeltaNet + Hardware-aware Search
 • Benjamin Marie
Magistral: Advancing Reasoning with Efficient GRPO Training
No More KL Penalty, No Need for a Reference Model
 • Benjamin Marie
Qwen3 Technical Report: Reasoning in Pre-Training and Post-Training
Plus a Brief Look at the Limitations of the Multilingual Evaluation
 • Benjamin Marie
Qwen2.5-VL: High-Resolution Vision Encoding with Efficient Windowed Attention
Also impressive in language generation tasks!
 • Benjamin Marie
TÜLU 3: The Post-Training Recipe
SFT + DPO + RLVR
 • Benjamin Marie
TÜLU 3's High-Quality Synthetic Datasets for Post-Training LLMs
Made by GPT-4o
 • Benjamin Marie
Go Zero-Shot for Cheaper LLM Evaluations
Unless you use a generative benchmark
 • Benjamin Marie
Evaluating AdEMAMix: A New Optimizer for Faster, More Efficient LLM Training
But with hyperparameter values not easy to find!
 • Benjamin Marie
© 2026 Benjamin Marie · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture