n4nAI
Open console →
Benchmarks & performanceAnalysisBatch Inference Throughput Benchmarks

Batch inference throughput for Llama 3.1 70B at scale

Analysis of Llama 3.1 70B throughput at scale: continuous batching, KV cache limits, and parallel strategies that decide batch inference cost and latency.

n4n Team5 min read1,072 words

Audio narration

Coming soon — every post will get a voice note here.

Llama 3.1 70B throughput at scale is governed less by raw FLOPS and more by how efficiently you pack sequences into the GPU memory hierarchy. If you treat batch inference as a simple queue of requests, you will leave 40-60% of your expensive accelerators idle while paying for context windows you never use.

The thesis: batching is a scheduler, not a flag

The decisive factor for Llama 3.1 70B throughput at scale is scheduling policy, not model architecture. Continuous batching with paged attention is necessary but not sufficient; you also need to align batch composition with the tensor-parallel degree and the decode-phase memory bandwidth ceiling.

Naive static batching pads sequences to the longest in the batch, wasting VRAM and compute. Continuous batching interleaves incoming requests as earlier ones finish, keeping the GPU fed. But if your batch size grows without bounds, KV cache allocation stalls and tail latency explodes. The engineer’s job is to tune the maximum number of sequences in flight against the 128K context limit of Llama 3.1 70B and the physical memory of the host.

A good scheduler separates prefill from decode, prioritizes short sequences when latency matters, and backs off when KV cache pressure crosses a threshold. That logic is where throughput is won or lost.

What actually bounds throughput for 70B

Memory bandwidth vs compute

A 70B model in fp16 weighs ~140GB. On 2× A100 80GB with tensor parallelism, each decode step moves weights from HBM to compute at roughly the memory bandwidth limit (about 2TB/s per card), not the math limit. Throughput scales with tokens generated per second per GPU, which is inversely related to batch size until you saturate bandwidth.

Compute-bound prefill (processing prompts) benefits from large batches. Decode-bound generation benefits from moderate batches that hide memory latency. Mixing them requires a scheduler that separates prefill and decode queues.

KV cache fragmentation

Llama 3.1 70B at 128K context allocates a massive KV cache per sequence. Without paging (e.g., vLLM’s PagedAttention), fragmentation forces you to reserve contiguous blocks, capping concurrent sequences far below theoretical limits. Paging recovers that headroom but adds bookkeeping overhead that grows with batch size. At 128K context, a single sequence can consume tens of GB just for KV; paging is mandatory for any serious batch workload.

Continuous batching in practice

Launch a serving stack that implements continuous batching. vLLM is the reference open-source choice:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70b-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9

This exposes an OpenAI-compatible endpoint. Client-side batching should issue concurrent requests and let the server coalesce them:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://your-endpoint/v1", api_key="key")

async def gen(prompt: str):
    resp = await client.chat.completions.create(
        model="meta-llama/Llama-3.1-70b-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
        temperature=0.0,
    )
    return resp.choices[0].message.content

async def main(prompts):
    return await asyncio.gather(*[gen(p) for p in prompts])

prompts = [f"Summarize doc {i}" for i in range(512)]
results = asyncio.run(main(prompts))

The server’s scheduler decides how many of those 512 are actually in-flight. Your client concurrency merely keeps the queue non-empty.

Measuring throughput without lying

To report Llama 3.1 70B throughput at scale you must separate prefill tokens from decode tokens. A batch that ingests 1M prompt tokens and emits 10K generated tokens is prefill-dominated; quoting aggregate tokens/sec hides the real bottleneck.

Use the usage field returned by OpenAI-compatible APIs:

resp = await client.chat.completions.create(...)
# resp.usage.prompt_tokens, resp.usage.completion_tokens

Track wall-clock per batch. Compute:

prefill_tps = total_prompt_tokens / prefill_time
decode_tps = total_completion_tokens / decode_time

Only compare numbers with similar sequence-length distributions. A 2K-token average batch and a 32K-token average batch are different machines qualitatively.

Sequence length distribution matters

Homogeneous batches (all prompts ~2K, all outputs ~256) let the scheduler pack efficiently. Skewed distributions cause head-of-line blocking: a few 100K-token requests starve the rest.

Bucketing mitigates this. Route requests into length buckets (e.g., 0-2K, 2-8K, 8-32K) and batch within bucket. This keeps padding low and KV cache predictable. For Llama 3.1 70B at scale, a 10x spread in prompt length can cut effective throughput in half if unbucketed.

Quantization: the silent lever

FP16 is the default but not the only option. FP8 on H100 or INT4/AWQ on A100 shrinks the weight footprint, freeing VRAM for larger batches and reducing memory-bandwidth pressure per token. The tradeoff is slight quality loss on nuanced tasks; for classification or extraction it is often negligible.

Quantized 70B fits on a single 80GB card with careful KV limits, turning a 2-node problem into 1-node, halving interconnect overhead. That architectural simplification often yields more throughput gain than any scheduler tweak.

Scaling across nodes: tensor parallel and pipeline parallel

Single-node 2× A100 is the floor, not the ceiling. To push Llama 3.1 70B throughput at scale, you add nodes with pipeline parallelism (PP) on top of tensor parallelism (TP). TP within a node uses fast NVLink; PP across nodes pays InfiniBand latency.

A 4× A100 (2 nodes, TP=2, PP=2) setup can double prefill throughput but may reduce decode efficiency if microbatch boundaries stall. Empirical rule: keep PP stages balanced in layer count (70B has 80 layers; 40+40 splits cleanly) and size microbatches to fill each stage’s compute.

Tradeoffs: latency vs throughput

Maximum throughput sacrifices latency. A batch of 256 sequences with 2K-token prompts and 256-token generations will show p50 latency of seconds, but tokens/sec/GPU climbs. If your product needs interactive latency (<500ms), you must cap batch size and accept lower utilization.

Weigh the cost: idle GPU hours are the dominant expense. For offline extraction, summarization, or eval runs, crank batch size until KV cache hits 90% utilization. For serving, set a max-num-seqs that bounds tail latency.

Gateway-level routing and metering

When batches span multiple providers or regions, routing matters. A gateway such as n4n.ai that honors client routing directives lets you pin a large batch to a single region, avoiding cross-zone KV cache cold starts and giving predictable per-token metering for cost analysis. Automatic fallback is useful for spot capacity, but for a planned batch job you want deterministic placement.

Cache-control hints forwarded to providers can reuse prefill across repeated system prompts—a real win when you batch homogeneous tasks like classification with a fixed instruction.

Failure modes at scale

OOM from KV cache is the most common outage. Set --gpu-memory-utilization conservatively and monitor resident sequences. Starvation occurs when a few long requests monopolize scheduler slots; use priority queues. Provider degradation (throttling, packet loss) silently drops throughput; if you run through a gateway with automatic fallback, ensure it does not reshuffle your batch mid-flight and invalidate cache.

Decisive takeaway

Treat Llama 3.1 70B throughput at scale as a memory-bandwidth and scheduling problem. Deploy continuous batching with paged KV cache, set tensor-parallel size to your NVLink domain, and tune max concurrent sequences against your latency budget. For offline jobs, maximize batch size until GPU memory utilization peaks; for online, cap it. Use bucketing and quantization to recover another 30-50% before buying more hardware. Pin routing to avoid cache misses. That discipline, not bigger clusters, is what separates a $5 batch from a $50 one.

Tagsllama-3-1-70bthroughputbatch-inferencescale

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceBatch inference throughput benchmark for DeepSeek V3Analysis of DeepSeek V3 throughput benchmark: MoE batching behavior, KV cache limits, and how to measure real-world inference performance.Benchmarks & performanceBatch inference throughput benchmark on H100 vs A100Direct H100 vs A100 throughput benchmark for batch LLM inference across capabilities, cost, latency, and ergonomics, with a head-to-head table and verdict.Benchmarks & performanceBatch inference throughput: vLLM vs TensorRT-LLMA practitioner's head-to-head comparison of vLLM vs TensorRT-LLM throughput for batch inference across capabilities, cost, latency, ergonomics, ecosystem, and hard limits.Benchmarks & performanceHow continuous batching improves LLM throughputContinuous batching LLM throughput gains come from dynamic scheduling. This guide shows how to measure, implement, and tune it for production inference serving.Benchmarks & performanceMeasuring throughput ceiling on 8x H100 clustersPractical analysis of measuring the real 8x H100 cluster throughput ceiling for batch inference: parallelism, saturation testing, and tradeoffs.Benchmarks & performanceThroughput benchmark: single-GPU vs multi-GPU inferencePractical comparison of single-GPU vs multi-GPU throughput for LLM inference: scaling efficiency, cost, latency, and which setup to pick for your workload.

All batch inference throughput benchmarks posts →