Building a compliance-checking agent for legal contracts

A compliance checking agent legal contracts needs to do more than naive keyword matching—it must understand clause semantics and map them to regulatory obligations. This tutorial builds a focused agent that extracts key clauses from a contract and validates them against a configurable rule set using a structured LLM call. By the end you’ll have a runnable Python script that prints a severity-ranked compliance report.

Prerequisites

• Python 3.11 or newer
• openai, pydantic, python-dotenv installed
• An API key for an OpenAI-compatible inference gateway (n4n.ai provides one endpoint covering 240+ models with automatic fallback, which we’ll use)
• A sample contract saved as contract.txt (plain text, under ~12k characters for this demo)

pip install openai pydantic python-dotenv

Create a .env file:

LLM_API_KEY=sk-your-key-here

Step 1: Model compliance rules as data

Hardcoding prompts inside loop logic creates unmaintainable review code. Define rules with Pydantic so they can be loaded from YAML or a database later.

from pydantic import BaseModel

class ComplianceRule(BaseModel):
    id: str
    severity: str  # "high" | "medium" | "low"
    check_prompt: str  # instruction for LLM to assess extracted clauses

RULES = [
    ComplianceRule(
        id="GDPR-01",
        description="Contract must specify data processing purpose",
        severity="high",
        check_prompt=(
            "Given the clauses, does the contract explicitly state the purpose "
            "for which personal data is processed? Respond JSON: "
            "{'pass': bool, 'evidence': str}"
        )
    ),
    ComplianceRule(
        id="LIB-01",
        description="Limitation of liability cap must be present",
        severity="medium",
        check_prompt=(
            "Is there a monetary cap on either party's liability? Respond JSON: "
            "{'pass': bool, 'evidence': str}"
        )
    ),
]

Step 2: Extract clauses with structured output

We ask the model to return only clause types relevant to our rules. Using response_format={"type": "json_object"} forces valid JSON.

import os, json
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible gateway
    api_key=os.getenv("LLM_API_KEY")
)

def extract_clauses(contract_text: str) -> list[dict]:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": (
                "You are a contract analysis engine. Extract clauses related to "
                "data protection, liability, termination, or confidentiality. "
                "Return JSON: {'clauses': [{'type': str, 'text': str}]}"
            )},
            {"role": "user", "content": contract_text[:12000]}
        ],
        response_format={"type": "json_object"}
    )
    data = json.loads(resp.choices[0].message.content)
    return data["clauses"]

Run a checkpoint:

if __name__ == "__main__":
    with open("contract.txt") as f:
        text = f.read()
    clauses = extract_clauses(text)
    print(json.dumps(clauses, indent=2))

Expected output (truncated):

[
  {
    "type": "data_protection",
    "text": "The Processor shall process personal data only for the purpose of providing the Service as set out in Schedule 1."
  },
  {
    "type": "liability",
    "text": "Each party's total liability shall not exceed the fees paid in the preceding 12 months."
  }
]

Step 3: Evaluate rules against clauses

Each rule gets its own LLM call. Keeping calls atomic makes failures debuggable and lets you cache results per rule.

def check_rule(rule: ComplianceRule, clauses: list[dict]) -> dict:
    context = "\n".join(f"[{c['type']}] {c['text']}" for c in clauses)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": rule.check_prompt},
            {"role": "user", "content": f"Clauses:\n{context}"}
        ],
        response_format={"type": "json_object"}
    )
    result = json.loads(resp.choices[0].message.content)
    return {
        "rule_id": rule.id,
        "severity": rule.severity,
        "description": rule.description,
        "pass": result.get("pass", False),
        "evidence": result.get("evidence", "")
    }

If the gateway supports forwarding provider cache-control hints, add extra_headers={"cache-control": "max-age=3600"} to repeated rule checks for the same clause set.

Step 4: Assemble and print the report

def run_agent(contract_path: str) -> list[dict]:
    with open(contract_path) as f:
        text = f.read()
    clauses = extract_clauses(text)
    return [check_rule(r, clauses) for r in RULES]

if __name__ == "__main__":
    findings = run_agent("contract.txt")
    for f in findings:
        status = "PASS" if f["pass"] else "FAIL"
        print(f"{f['rule_id']} [{f['severity']}] {status}: {f['evidence']}")

Expected output:

GDPR-01 [high] PASS: Clause states purpose: "providing the Service as set out in Schedule 1."
LIB-01 [medium] PASS: Cap present: "fees paid in the preceding 12 months."

If a clause is missing, the agent returns FAIL with an evidence string explaining the gap.

Handling scale and degradation

Running this on thousands of contracts exposes you to provider rate limits. A gateway that honors client routing directives and automatically falls back when a provider is degraded keeps the same code operational under load. The n4n.ai endpoint we used earlier does exactly that, so no application change is needed when a backing model returns 429s.

For production, wrap check_rule in retries with exponential backoff and log resp.usage for per-token metering:

usage = resp.usage
print(f"rule {rule.id} used {usage.prompt_tokens} prompt / {usage.completion_tokens} completion")

Extending the agent

The core loop—extract, check, report—is deliberately small. Real systems add:

• Vector retrieval for long contracts: chunk and embed, retrieve top-k per rule instead of stuffing full text.
• Human review queue: persist findings with a reviewed_by field; fail-closed on high severity.
• Rule versioning: stamp each finding with the rule hash so audit logs show which policy was applied.
• Multi-model voting: call two models and require consensus on high severity rules to reduce hallucination.

A compliance checking agent legal contracts is only as trustworthy as its evidence trail. Keep the LLM output structured, log the raw clauses, and never let a silent failure mask a missing clause.

Final notes on prompt design

The check_prompt strings are the contract between you and the model. Specify the JSON shape explicitly, and avoid open-ended questions. If you need citation, demand quote extraction as we did with evidence. When the contract domain shifts (healthcare HIPAA instead of GDPR), you swap RULES—not the agent code. That separation is what makes the approach maintainable across the legal-tech stack.