Token counting streaming responses before the generation completes is a problem every LLM app faces once it tries to show live cost or progress. You cannot run the provider’s billing tokenizer on text that does not exist yet, yet product teams still demand a running total. This analysis argues that precise token counting streaming responses is impossible client-side, but a hybrid estimation strategy gives you enough confidence for UI and guardrails without blocking the stream.
The core impossibility
Tokenization is a deterministic function from string to token list for a given model. If you have the full string, you can count exactly. The trouble is that a streaming response delivers the string in fragments over time, and the final length is unknown until the stream ends.
Worse, byte-pair encoding (BPE) tokenizers merge adjacent characters based on global frequency stats. Tokenizing a prefix in isolation can yield different token boundaries than tokenizing that same prefix as part of a longer string. You are not just missing future tokens; you may be miscounting present ones if you treat each chunk independently.
What you can actually measure
As chunks arrive, you can maintain an accumulating buffer and run the exact tokenizer on the whole buffer after each delta. For a 1k-token response, that is ~1k tokenizer calls on growing input—fine for tiktoken, heavy for remote tokenizers.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
buffer = ""
total_tokens = 0
def on_chunk(delta: str):
global buffer, total_tokens
buffer += delta
# exact count on accumulated text
total_tokens = len(enc.encode(buffer))
return total_tokensThis gives exact counts for everything received so far. It says nothing about the remaining unseen text. For token counting streaming responses, this is the only ground truth you get before the finish.
Why naive delta counting drifts
A common mistake is to tokenize each delta separately and sum. Because BPE merges cross chunk boundaries, the sum overestimates or underestimates.
# Wrong: tokenize deltas independently
wrong = 0
for delta in ["Hel", "lo wor", "ld"]:
wrong += len(enc.encode(delta))
# Right: encode full buffer
right = len(enc.encode("Hello world"))For "Hello world", cl100k_base yields 2 tokens. The naive sum might yield 3 or 4 depending on splits. Over a long stream, the drift is usually under 5%, but it is systematic, not noise.
Estimation strategies
You need a prediction of total tokens before the stream ends. Three approaches, each with tradeoffs.
Character heuristics
Roughly 4 characters per token for English prose, 2–3 for code, 1.5 for Chinese. Multiply streamed chars by ratio and extrapolate from elapsed time or a prior length guess.
function estimateTokens(text: string, charsPerToken = 4): number {
return Math.ceil(text.length / charsPerToken);
}Cheap, no tokenizer dependency, but blind to model-specific vocab. Good for a progress bar, not for billing.
Stateful streaming tokenizer
Some tokenizer libs expose a stateful encoder that ingests bytes and emits tokens only when a boundary is stable. This avoids re-encoding the whole buffer.
// Pseudocode using a hypothetical streaming BPE interface
const tok = createStreamingEncoder("gpt-4");
tok.push(chunk);
const newTokens = tok.flushStable(); // tokens that won't changeIf you lack such a lib, re-encoding the buffer every N chunks is a pragmatic compromise.
Provider usage hints
OpenAI’s Chat Completions stream sends a final chunk with usage. Anthropic returns usage in the response. Use that as the authoritative total and reconcile your running estimate.
{
"choices": [
{
"delta": {},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 134,
"total_tokens": 146
}
}For token counting streaming responses, treat the usage field as the correction signal, not the live signal.
Implementing a hybrid counter
The pattern I ship: start with character heuristic for immediate feedback, switch to exact buffer encoding once the buffer crosses a threshold, and overwrite with provider usage at the end.
class StreamCost {
private buf = "";
private exactTokens = 0;
private charEstimate = 0;
private usedExact = false;
constructor(private charsPerToken = 4) {}
add(chunk: string) {
this.buf += chunk;
this.charEstimate = Math.ceil(this.buf.length / this.charsPerToken);
if (this.buf.length > 200 && !this.usedExact) {
// lazy-load exact tokenizer in real app
this.exactTokens = exactEncode(this.buf);
this.usedExact = true;
} else if (this.usedExact) {
this.exactTokens = exactEncode(this.buf);
}
}
get displayTokens(): number {
return this.usedExact ? this.exactTokens : this.charEstimate;
}
finalize(usage: { completion_tokens: number }) {
// authoritative from provider
return usage.completion_tokens;
}
}The heuristic covers the first 50ms; exact covers the middle; provider covers the end. No single method is sufficient, but the seams are invisible to the user.
Cost guardrails, not just UX
Live token counts are not only for UI. If you cap spend at 10k tokens, you must halt the stream when estimated completion approaches the limit. Because estimates lag, set the hard cutoff at 95% of budget using exact counts when available, and 90% using heuristics.
# Example guard config
export MAX_COMPLETION_TOKENS=10000
export SOFT_STOP_ESTIMATE=9000
export HARD_STOP_EXACT=9500A gateway that performs per-token usage metering, such as n4n.ai, will bill the actuals precisely, but your client still owns the responsibility of terminating early based on its own estimates.
Tradeoffs weighed
Exact buffer encoding is accurate but O(n²) if done naively on every chunk; mitigate with batching or streaming tokenizer. Character heuristics are fast but model-blind; acceptable for short streams. Provider usage is perfect but arrives too late to act. The hybrid adds code complexity, yet removes the worst failure modes of each.
If you skip the hybrid and ship only heuristics, you will overcharge users in your UI or falsely trigger guardrails. If you skip heuristics and wait for exact, your first 500ms shows zero cost, which product managers reject.
Decisive takeaway
Build token counting streaming responses as a three-stage pipeline: cheap extrapolation from characters, exact counting on accumulated text once the buffer is nontrivial, and reconciliation with provider usage at termination. Use the estimate to drive UX and soft limits; use the exact count for hard limits when you have it; use the provider’s final usage for billing truth. Anything less either lies to your users or reacts too late. Implement the hybrid once in a shared client module and never trust a single method for live token visibility.