Building a reliable system to prompt AI legal agent clause extraction from dense contracts is harder than calling a chat endpoint. You need a strict output schema, explicit instructions, and a verification loop to catch missed or hallucinated clauses before they reach a legal workflow.
Step 1: Define a strict clause schema
Vague definitions produce vague JSON. Before writing any prompt, decide exactly what a clause is for your domain. A good schema forces the model to return verbatim text, a location, and a normalized type. Avoid free-form “summary” fields; they invite hallucination.
{
"type": "object",
"properties": {
"clauses": {
"type": "array",
"items": {
"type": "object",
"properties": {
"clause_type": {
"type": "string",
"enum": ["termination", "liability", "confidentiality", "payment", "ip_assignment", "other"]
},
"text": {"type": "string"},
"section": {"type": "string"},
"parties": {"type": "array", "items": {"type": "string"}}
},
"required": ["clause_type", "text", "section"]
}
}
},
"required": ["clauses"]
}The other enum value is not laziness—it lets the agent flag unknown clause kinds without dropping them or forcing a misfit. Back this with Pydantic so validation is one import away.
from pydantic import BaseModel, Field
from typing import List, Optional
class Clause(BaseModel):
clause_type: str
text: str
section: str
parties: Optional[List[str]] = None
class ExtractionResult(BaseModel):
clauses: List[Clause]Why schema-first matters
If you prompt AI legal agent clause extraction without a fixed schema, you will spend weeks writing parsers for slightly different key names. Lock the contract now.
Step 2: Write a system prompt that constrains the agent
A legal agent will improvise if you let it. Tell it to return only clauses that match the schema, never summarize, and emit an empty array when nothing fits. Repeat the constraints; models weight later instructions but repetition reduces drift.
You are a contract analysis agent. Extract every clause that matches one of these types: termination, liability, confidentiality, payment, ip_assignment, other.
For each clause, return the verbatim text from the contract, the section heading or number, and the obligated parties.
Do not paraphrase. Do not add commentary. If a type is not present in a chunk, return an empty clauses array.
Always output valid JSON conforming to the provided schema.The key to prompt AI legal agent clause extraction successfully is forbidding inference. Lawyers want the source language, not your model’s rewrite.
Few-shot is optional, not required
With JSON schema mode, a clear system prompt outperforms fragile few-shot examples that bloat context. Save few-shot for edge cases like cross-referenced sections.
Step 3: Chunk the contract and preserve metadata
Most agreements exceed 32k tokens. Split on page breaks or numbered sections, but keep a document ID and chunk index in each call. This lets you merge later without losing provenance.
import re
def chunk_contract(text: str, max_chars: int = 12000) -> List[dict]:
# split on section markers like "1.", "2.1", or ALL CAPS headings
rough = re.split(r'(?=\n\d+\.|\n[A-Z][A-Z ]+\n)', text)
chunks = []
buf = ""
idx = 0
for part in rough:
if len(buf) + len(part) > max_chars:
chunks.append({"doc_id": "contract_123", "chunk": idx, "content": buf})
idx += 1
buf = part
else:
buf += part
if buf:
chunks.append({"doc_id": "contract_123", "chunk": idx, "content": buf})
return chunksOverlap chunks by 200 characters to avoid cutting a clause at the boundary. Store the offset so you can remap sections later.
Step 4: Call the model with structured outputs
Use an OpenAI-compatible client. Point it at your gateway; if you use n4n.ai, it forwards provider cache-control hints, so mark the static schema as cached to avoid reprocessing it each chunk. Its per-token usage metering also lets you attribute cost per document without building your own middleware.
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1", # OpenAI-compatible endpoint
api_key="YOUR_KEY"
)
schema = {...} # from Step 1
SYSTEM_PROMPT = "..." # from Step 2
def extract_chunk(chunk: dict) -> ExtractionResult:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Document {chunk['doc_id']} chunk {chunk['chunk']}:\n{chunk['content']}"}
],
response_format={"type": "json_schema", "schema": schema},
temperature=0.0,
extra_headers={"cache-control": "schema"}
)
return ExtractionResult.model_validate_json(resp.choices[0].message.content)The prompt AI legal agent clause extraction loop runs this per chunk. Use the smallest model that hits your accuracy bar; clause extraction is pattern matching, not reasoning.
Step 5: Merge and deduplicate across chunks
Clause text may span chunk boundaries or be detected twice. Hash the normalized text and keep the first occurrence with its earliest section tag.
import hashlib
def merge_results(results: List[ExtractionResult]) -> ExtractionResult:
seen = set()
merged = []
for res in results:
for clause in res.clauses:
norm = re.sub(r'\s+', ' ', clause.text.strip())
h = hashlib.sha256(norm.encode()).hexdigest()
if h not in seen:
seen.add(h)
merged.append(clause)
return ExtractionResult(clauses=merged)Handling split clauses
If a clause is truncated at a chunk edge, the merge step will keep the partial from chunk N and the continuation from chunk N+1 as two entries. Add a post-merge check: if section matches and texts are prefixes, concatenate.
Step 6: Verify extraction coverage
Verification is not optional. Take a labeled sample of 20 contracts with known clause counts. Run your pipeline and compute precision/recall against the labels. For unlabeled production traffic, use a second cheap model to check that each extracted text appears verbatim in the source.
def verify_citations(result: ExtractionResult, source: str) -> float:
hits = 0
for c in result.clauses:
norm = re.sub(r'\s+', ' ', c.text.strip())
if norm in re.sub(r'\s+', ' ', source):
hits += 1
return hits / max(len(result.clauses), 1)
# Success criterion: citation match > 0.98 and zero schema violations.If citation match drops below 0.95, your prompt is leaking summaries. Tighten the system prompt and re-run.
Human-in-the-loop threshold
For high-stakes contracts, route any clause_type: other or any clause where parties is empty to a human reviewer. The agent is a first-pass filter, not a notary.
Step 7: Handle provider degradation
Rate limits and 529s happen. Wrap the call in a retry with exponential backoff. If you route through a gateway with automatic fallback when a provider is rate-limited or degraded, the same code works without branching. Otherwise, implement fallback across two model IDs manually.
import time, random
def extract_with_retry(chunk, attempts=3):
for i in range(attempts):
try:
return extract_chunk(chunk)
except Exception as e:
if i == attempts - 1:
raise
time.sleep(2 ** i + random.random())To prompt AI legal agent clause extraction at scale, treat the model as a flaky microservice, not an oracle. Log the chunk ID and response latency on every call.
Verify success end to end
Success means: (1) every output validates against the Pydantic model, (2) ≥98% of clause texts are verbatim present in the input, (3) on a held-out labeled set, recall exceeds 90% for termination and liability clauses. Run the verification script in CI on a fixed fixture so regressions surface before deploy.
Build the schema, constrain the prompt, chunk with metadata, call structured, merge, verify, harden. Anything less ships hallucinations into a legal workflow where they cost real money.