The Salt - Curated AI

Vocabulary Parallelism for More Efficient LLMs

Memory-Efficient Inference with 4-bit Activations for 1-bit LLMs

Go Zero-Shot for Cheaper LLM Evaluations

Your Prompts Are Not Safe with Mixture of Experts Models

The Effective Context Length of LLMs and Semi-Supervised Fine-Tuning

Mixture-of-Experts: Mixture-of-Head Attention and Embedding Model

Cancelling Attention Noise with Differential Transformer

Evaluating AdEMAMix: A New Optimizer for Faster, More Efficient LLM Training

Cross Capabilities of LLMs and Contextual Document Embeddings

LLMs Can Follow Instructions Without Instruction Tuning

Qwen2-VL: How Does It Work?

SCoRe: Teach LLMs to Self-Correct