Topic
Batching & Continuous Batching
6 posts on batching & continuous batching — part of glossary on the n4n AI blog.
What is batching in LLM inference?
A precise technical definition of batching in LLM inference, covering static vs continuous batching, KV cache mechanics, throughput-latency tradeoffs, and common misconceptions engineers encounter in production.
Static batching vs continuous batching, compared
Compare static batching and continuous batching for LLM inference — latency, throughput, memory, and when to use each.
How continuous batching improves GPU utilization
Continuous batching keeps GPUs saturated by interleaving prefill and decode, but it introduces scheduler complexity and memory pressure that static batching avoids.
How batch size affects LLM latency and throughput
How batch size creates a fundamental tradeoff between latency and throughput in LLM inference, with concrete guidance for production systems.
Continuous batching explained: how it beats static batching
Continuous batching vs static batching for LLM serving — how iteration-level scheduling beats request-level batching for throughput and latency.
Batching trade-offs: throughput gains vs per-request latency
Deep dive on batching strategies for LLM serving — how static and continuous batching affect throughput and latency, with concrete scheduling tradeoffs.
More topics in glossary
- Structured Outputs & JSON Mode19
- AI Agents Fundamentals12
- Hallucination in LLMs11
- Sampling Parameters: Top-p, Top-k & Penalties11
- Context Window & Context Length10
- Fine-Tuning Fundamentals9
- Foundation Models: Base vs Instruct vs Chat9
- Model Families & Naming Conventions: GPT-5, Claude, Gemini 3, Llama 4, Mistral, DeepSeek, Qwen, Grok9
- Grounding & Fact-Checking in AI8
- LLM Benchmarks: MMLU, HumanEval, SWE-bench & GPQA8
- Max Tokens, Stop Sequences & Output Truncation8
- Quantization Formats: GGUF, GPTQ, AWQ & INT4/INT88