Tokens per second at batch size 1 vs batch size 64

When you move from processing a single prompt to firing 64 prompts at once, the tokens per second batch size comparison stops being theoretical and starts dictating your infrastructure bill. At batch size 1 the GPU spends most of its cycles waiting on memory; at batch size 64 it finally gets enough parallel work to saturate compute—but per-request latency and memory pressure shift the tradeoff hard. This article breaks down the two operating points across the dimensions that matter when you ship.

What batch size actually changes

Batch size in LLM inference is the number of sequences you forward through the model in the same CUDA kernel launch. At BS=1, the matrix multiplies are tiny: a 7B model with float16 weights still needs ~14GB of memory reads per token, but the math is only a few GFLOPs. The accelerator is memory-bandwidth bound and idle most of the time.

At BS=64, those same weight reads are reused across 64 sequences. Arithmetic intensity climbs, and you approach the GPU’s peak FLOPS. You generate more total tokens per second, but each sequence waits behind the others in the batch for its turn in the attention and MLP layers.

# Static batching with Hugging Face, BS=64
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, time

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16, device_map="auto"
)
prompts = ["Summarize: batch inference matters."] * 64
inputs = tok(prompts, return_tensors="pt", padding=True, truncation=True).to("cuda")
start = time.time()
out = model.generate(**inputs, max_new_tokens=128)
elapsed = time.time() - start
generated = (out != tok.pad_token_id).sum().item() - inputs.input_ids.numel()
print(f"Aggregate: {generated/elapsed:.0f} tok/s, per-seq: {elapsed/64:.2f}s")

The same loop with prompts = ["..."] * 1 gives you the BS=1 baseline.

Head-to-head: batch size 1 vs batch size 64

The following table summarizes the operating characteristics. Numbers are qualitative; exact figures depend on model size, GPU, and serving stack.

Capabilities

BS=1 handles one stream. You can use speculative decoding, tight timeouts, and per-request sampling params without coordination. BS=64 requires either static padding (wasting compute on pad tokens) or a ragged/batch-aware kernel. Most production servers avoid explicit batching by using continuous batching, but if you run offline generation you still choose how many prompts to pack.

Price and cost model

Cloud APIs charge per output token regardless of how many concurrent requests you open. The tokens per second batch size comparison matters because higher batch throughput on your side reduces the wall-clock during which you hold connections and retries. Self-hosting, the math is direct: a fixed GPU hour yields N tokens at BS=1 and ~10N at BS=64, so cost per million tokens drops proportionally.

Latency and throughput

This is the core of the tokens per second batch size comparison. BS=1 gives you, say, 30–50 tok/s on a 7B model on one A100, with time-to-first-token under 50 ms. BS=64 might push aggregate decode to 1500+ tok/s, but each user in the batch sees TTFT in the hundreds of ms and per-token latency that depends on batch position. For synchronous UX, that is unacceptable; for a nightly corpus rewrite, it is ideal.

Ergonomics

A BS=1 call is a single POST /v1/completions with one prompt. BS=64 either means 64 parallel requests (handled by a continuous-batching server) or one request with an array of prompts if the server supports it. You must manage padding, attention masks, and uneven generation lengths. In practice, engineers lean on vLLM or TGI to hide this, but the mental model still shifts from “one answer” to “a queue drained together.”

# 64 concurrent requests against an OpenAI-compatible endpoint
for i in $(seq 1 64); do
  curl -s localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"local","prompt":"batch test","max_tokens":128}' &
done
wait

Ecosystem and tooling

Nearly every high-performance inference server (vLLM, TensorRT-LLM, Hugging Face TGI) implements continuous batching, so the client rarely sets batch size explicitly. For offline workloads, PyTorch DataLoader with batch_size=64 and generate() is standard. If you aggregate throughput across providers via a gateway such as n4n.ai, per-token metering and automatic fallback let you sustain large batch jobs without hand-rolling retry and rate-limit logic.

Hard limits

VRAM is the wall. KV cache size = 2 * batch * seq_len * n_layers * n_heads * head_dim * dtype_bytes. At BS=64 with 4k context on a 13B model, that cache alone can exceed 20GB, leaving little for weights. You also hit scheduler queues: some endpoints cap concurrent requests. Finally, long-tail latency appears when one sequence in the batch requests 2k tokens while others finish at 32—the batch stays alive until the longest completes unless the server supports chunked eviction.

Benchmark methodology that doesn’t lie

Synthetic “hello world” prompts produce misleading tokens per second batch size comparison results because they exit early. Use real-length prompts and a fixed max_new_tokens. Measure wall-clock from first byte to last, then compute:

aggregate_tok_s = total_generated_tokens / elapsed_seconds
per_seq_latency = elapsed_seconds / batch_size   # naive, ignores overlap

Better: log per-request completion timestamps from the client side when using concurrent calls. That captures the true queuing penalty at BS=64.

Continuous batching erases the binary

The BS=1 vs BS=64 framing is clean for benchmarks, but production servers don’t wait for a full batch. They admit new requests as old ones finish, holding GPU occupancy near 90% even at low traffic. So the practical tokens per second batch size comparison is really “static batched offline” vs “dynamic batched online.” If you control the serving layer, tune --max-num-seqs (vLLM) rather than forcing a fixed batch.

Which to choose

Interactive applications — chatbots, autocomplete, agent tool calls. Use BS=1 semantics (or continuous batching with low concurrency caps). Prioritize TTFT and stable inter-token delay. Pay the higher per-token cost; users notice latency before they notice your GPU bill.

Offline transformation — document summarization, training-data generation, embedding backfills. Pack to BS=64 or the maximum your VRAM allows. Throughput per dollar is the only metric; latency of minutes is fine.

Evaluation and benchmarking — accuracy suites over 10k prompts. Batch aggressively, but shard across multiple workers to avoid a single stuck sequence. Use the aggregate tok/s number to size your fleet.

Hybrid RAG pipelines — if you retrieve 64 chunks and need a single synthesized answer, that’s not BS=64; it’s one long context. Keep the generation call at BS=1 and parallelize only the embedding or retrieval steps.

Pick the batch size that matches the latency budget, not the headline throughput. The tokens per second batch size comparison is a tool for capacity planning, not a setting you ship to users.