A LlamaIndex context window overflow fix is required the moment you see ValueError: Context window overflow or a 400 from your model provider about exceeding max tokens. The root cause is almost always that the serialized prompt—system message, retrieved nodes, chat history, and the query—exceeds the model’s max_input_size. This guide walks through reproducing the failure, then applying concrete configuration and retrieval changes to keep your prompts bounded.
Step 1: Reproduce the overflow and capture token counts
Don’t guess where the tokens go. Enable LlamaIndex debug logging and wrap your query call so you can inspect the exact exception and the prompt size that triggered it.
import logging
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
logging.basicConfig(level=logging.DEBUG)
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
try:
response = query_engine.query("Summarize the quarterly report")
except ValueError as e:
# LlamaIndex raises ValueError from PromptHelper on overflow
logging.error(f"Overflow: {e}")If you’re on a recent version, the error originates in PromptHelper.repack. To see the numbers, instantiate the helper directly:
from llama_index.core.llms import ChatMessage
from llama_index.core.prompts import PromptHelper
helper = PromptHelper(
context_window=128000,
num_output=256,
chunk_overlap_ratio=0.1,
tokenizer_fn=lambda x: len(x.split()), # replace with real tokenizer
)Swap the dummy tokenizer for tiktoken before trusting the output. The point is to confirm the overflow is real and not a misconfigured max_input_size.
Step 2: Audit chunk size and node parsing
The default chunk_size in LlamaIndex is 1024 tokens, but if you ingest large PDFs with SimpleNodeParser and never set chunk_overlap, you can still pull huge nodes into context. In v0.10+ configure globally via Settings:
from llama_index.core import Settings
from llama_index.core.node_parser import SimpleNodeParser
Settings.chunk_size = 512
Settings.chunk_overlap = 64
Settings.node_parser = SimpleNodeParser.from_defaults(
chunk_size=512, chunk_overlap=64
)If you’re still on ServiceContext, the equivalent is:
from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(
chunk_size=512, chunk_overlap=64
)Smaller chunks reduce the chance that a single retrieved node blows your budget. But chunking alone won’t save you if similarity_top_k is set to 20 and each chunk is 512 tokens—that’s 10k tokens before the query is added.
Step 3: Set the LLM max input size explicitly
LlamaIndex attempts to infer max_input_size from the LLM object. For OpenAI models it usually works; for local models or OpenAI-compatible proxies it often defaults to 4096, which is wrong and causes premature overflow errors. Set it manually:
from llama_index.llms.openai import OpenAI
llm = OpenAI(
model="gpt-4o-mini",
max_input_size=128000,
max_new_tokens=256,
)
Settings.llm = llmIf you route through an OpenAI-compatible gateway such as n4n.ai, which fronts 240+ models behind one endpoint, you still must set max_input_size to match the underlying model you select—the gateway won’t auto-expand context. A 32k model behind the endpoint will still reject a 40k prompt.
Step 4: Trim retrieved nodes with postprocessors
The fastest LlamaIndex context window overflow fix after chunking is to cut the number of nodes that reach the prompt. Use a SimilarityPostprocessor to drop weak matches:
from llama_index.core.postprocessor import SimilarityPostprocessor
query_engine = index.as_query_engine(
similarity_top_k=10,
node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.78)],
)For time-sensitive data, FixedRecencyPostprocessor keeps only the most recent nodes. For keyword-bounded queries, KeywordNodePostprocessor filters by required terms:
from llama_index.core.postprocessor import KeywordNodePostprocessor
post = KeywordNodePostprocessor(
required_keywords=["revenue", "2024"],
exclude_keywords=["draft"],
)These run after retrieval but before prompt assembly, so they directly reduce token count.
Step 5: Compress context with reranking or LongLLMLingua
Trimming by similarity still leaves full-text chunks. Reranking keeps the top‑n most relevant and drops the rest:
from llama_index.core.postprocessor import SentenceTransformerRerank
rerank = SentenceTransformerRerank(
top_n=3, model="BAAI/bge-reranker-base"
)
query_engine = index.as_query_engine(
similarity_top_k=12,
node_postprocessors=[rerank],
)For aggressive compression, LongLLMLinguaPostprocessor summarizes and prunes nodes using a small language model:
from llama_index.core.postprocessor import LongLLMLinguaPostprocessor
llm_lingua = LongLLMLinguaPostprocessor(
instruction_str="Given the context, answer the question",
target_token=1000,
rank_method="longllmlingua",
)
query_engine = index.as_query_engine(
node_postprocessors=[llm_lingua],
)This drops token usage dramatically but adds latency and a dependency. Use it only when retrieval volume is unpredictable.
Step 6: Consider a larger context model—but fix retrieval first
Swapping to a 128k model masks the problem and costs more per call. If you must, change the model name and max_input_size as in Step 3. With a gateway that aggregates providers, you can flip models via a header without code changes, but you’re still paying for redundant context transfer. The durable LlamaIndex context window overflow fix is bounding what you retrieve, not relying on a bigger window.
Step 7: Verify with a token-count integration test
Success means your production query path never raises overflow and stays under a hard token ceiling. Write a test that assembles the prompt the same way LlamaIndex does and asserts the count:
import tiktoken
from llama_index.core import Settings
enc = tiktoken.encoding_for_model("gpt-4o-mini")
def estimate_query_tokens(query: str, nodes: list) -> int:
text = query + "\n".join(n.get_content() for n in nodes)
return len(enc.encode(text))
def test_prompt_fits_within_budget():
nodes = retrieve_nodes("Summarize the quarterly report")
used = estimate_query_tokens("Summarize the quarterly report", nodes)
assert used < Settings.llm.max_input_size - 256 # reserve for outputRun this in CI against a fixed corpus. Additionally, set a smoke test that executes query_engine.query on the largest expected document and checks no ValueError is raised. If both pass, the overflow is fixed.
Verification checklist
- Debug log shows prompt token count below
max_input_size - num_output. similarity_top_kpluschunk_sizemath yields headroom for query and system prompt.- Postprocessors reduce node count without dropping answer quality (spot‑check responses).
- Integration test fails loudly if a document grows beyond budget.
Following these steps gives you a repeatable LlamaIndex context window overflow fix rather than a one‑off patch. The pattern is always: measure, shrink chunks, cap retrieval, compress, then verify with real token counts.