Topic
Batch Inference Throughput Benchmarks
12 posts on batch inference throughput benchmarks — part of benchmarks & performance on the n4n AI blog.
Why larger batches don't always mean higher throughput
Explains why increasing batch size eventually stops improving LLM inference throughput, with concrete tradeoffs between latency and hardware limits.
Tokens per second at batch size 1 vs batch size 64
A practical tokens per second batch size comparison: batch size 1 versus 64 across latency, cost, ergonomics, and limits for LLM inference.
Throughput vs latency: the batch size tradeoff explained
Practitioner guide to the throughput latency batch size tradeoff: measure baseline, sweep batches, find the knee, and implement adaptive batching.
Throughput benchmark: single-GPU vs multi-GPU inference
Practical comparison of single-GPU vs multi-GPU throughput for LLM inference: scaling efficiency, cost, latency, and which setup to pick for your workload.
Measuring throughput ceiling on 8x H100 clusters
Practical analysis of measuring the real 8x H100 cluster throughput ceiling for batch inference: parallelism, saturation testing, and tradeoffs.
How continuous batching improves LLM throughput
Continuous batching LLM throughput gains come from dynamic scheduling. This guide shows how to measure, implement, and tune it for production inference serving.
Batch throughput benchmark: Mixtral vs dense 70B models
Practical batch throughput comparison of Mixtral 8x7B MoE against dense 70B models across cost, latency, and ergonomics for LLM inference pipelines.
Batch size tuning for maximum LLM serving throughput
A practical, measurement-driven guide to batch size tuning throughput for LLM serving: step-by-step sweeps, memory limits, and validation under load.
Batch inference throughput: vLLM vs TensorRT-LLM
A practitioner's head-to-head comparison of vLLM vs TensorRT-LLM throughput for batch inference across capabilities, cost, latency, ergonomics, ecosystem, and hard limits.
Batch inference throughput for Llama 3.1 70B at scale
Analysis of Llama 3.1 70B throughput at scale: continuous batching, KV cache limits, and parallel strategies that decide batch inference cost and latency.
Batch inference throughput benchmark on H100 vs A100
Direct H100 vs A100 throughput benchmark for batch LLM inference across capabilities, cost, latency, and ergonomics, with a head-to-head table and verdict.
Batch inference throughput benchmark for DeepSeek V3
Analysis of DeepSeek V3 throughput benchmark: MoE batching behavior, KV cache limits, and how to measure real-world inference performance.
More topics in benchmarks & performance
- Agentic Workflow Performance Benchmarks14
- Benchmark Methodology and Measurement14
- Code Generation Latency for Dev Tools14
- Flagship Model Speed Showdown14
- Llama 4 Inference Speed by Provider14
- Price-Performance Rankings14
- Provider Uptime and Reliability Benchmarks14
- Reasoning Model Latency Overhead14
- Customer Support Chatbot Latency13
- DeepSeek Performance Benchmarks13
- GPU Inference Benchmarks13
- Long-Context Latency Benchmarks13