How to build an AI agent for compliance document checks

Building an AI agent compliance document checks pipeline that holds up in a regulated environment means treating extraction as a verifiable process, not a black box. You need explicit schemas, provider redundancy, and an audit trail that survives scrutiny. This guide walks through a concrete implementation you can ship.

Step 1: Define the compliance schema and acceptance criteria

The first task for any AI agent compliance document checks build is to pin down what “compliant” means as a typed contract. Legal reviewers think in clauses; your agent should think in fields. Use a strict schema so downstream code can fail loudly.

We’ll model a vendor agreement review. The agent must confirm a data processing agreement (DPA), capture any liability cap, identify governing law, and list missing required clauses.

from pydantic import BaseModel, Field, validator

class ComplianceCheck(BaseModel):
    has_data_processing_agreement: bool
    liability_cap_usd: float | None = None
    governing_law: str | None = None
    missing_clauses: list[str] = Field(default_factory=list)
    confidence: float = Field(ge=0.0, le=1.0)

    @validator("governing_law")
    def check_law(cls, v):
        if v and len(v) < 2:
            raise ValueError("governing law must be a valid jurisdiction")
        return v

Schema validation catches malformed model output before it reaches a database. Treat confidence as a first-class signal; anything below a threshold goes to human review.

Step 2: Ingest and chunk documents without losing context

PDFs from legal teams are messy: multi-column layouts, scanned images, footnotes. Don’t trust a generic “text splitter” that breaks mid-clause. Extract per page, then chunk on section boundaries where possible.

import pdfplumber

def extract_pages(path: str) -> list[str]:
    pages = []
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages:
            pages.append(page.extract_text() or "")
    return pages

For a 40-page contract, page-level chunks are often coherent enough. If a clause spans pages, keep a sliding window with 200-character overlap. The goal is to give the model self-contained text so it doesn’t hallucinate missing clauses.

Step 3: Wire a model gateway with fallback

When deploying an AI agent compliance document checks system, model availability is a compliance risk. A single provider outage during an overnight batch means missed SLAs. Point your client at a gateway that aggregates providers.

A gateway such as n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and performs automatic fallback when a provider is rate-limited or degraded, which matters when you process batches of legal PDFs at night. Its per-token usage metering also feeds the audit table directly.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible, 240+ models, fallback
    api_key="YOUR_KEY",
)

def call_model(system_prompt: str, user_prompt: str, model="anthropic/claude-3.5-sonnet"):
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_object"},
        temperature=0.0,
    )
    return resp.choices[0].message.content, resp.usage

Set temperature=0.0 for deterministic extraction. Use response_format to force JSON when the model supports it; otherwise parse defensively.

Step 4: Implement the extraction and validation agent loop

The agent loop is simple: extract per chunk, validate, aggregate. If confidence is low, retry with the neighboring chunk concatenated. Never silently swallow validation errors.

import json

SYSTEM_PROMPT = """You are a compliance extraction agent. Given document text, return JSON matching the schema:
{
  "has_data_processing_agreement": bool,
  "liability_cap_usd": float|null,
  "governing_law": string|null,
  "missing_clauses": string[],
  "confidence": float between 0 and 1
}
Only output JSON."""

def check_chunk(text: str) -> tuple[ComplianceCheck, object]:
    content, usage = call_model(SYSTEM_PROMPT, text)
    data = json.loads(content)
    check = ComplianceCheck(**data)
    if check.confidence < 0.7:
        raise ValueError(f"Low confidence: {check.confidence}")
    return check, usage

Aggregation merges booleans with AND, takes the max liability cap, and unions missing clauses. Keep the highest confidence score per field for traceability.

def aggregate(results: list[ComplianceCheck]) -> ComplianceCheck:
    merged = ComplianceCheck(
        has_data_processing_agreement=any(r.has_data_processing_agreement for r in results),
        liability_cap_usd=max((r.liability_cap_usd for r in results if r.liability_cap_usd), default=None),
        governing_law=next((r.governing_law for r in results if r.governing_law), None),
        missing_clauses=sorted({c for r in results for c in r.missing_clauses}),
        confidence=min(r.confidence for r in results),
    )
    return merged

This loop is the core of your AI agent compliance document checks logic. It is deliberately boring—exceptions route to a review queue, not to a wrong database row.

Step 5: Add human-in-the-loop and audit logging

Audit trails are non-negotiable for an AI agent compliance document checks workflow. Regulators want to know which model version inspected which text snippet and how many tokens it cost. Store every call.

import hashlib, datetime, sqlite3

def log_check(doc_id: str, chunk: str, check: ComplianceCheck, usage, model: str):
    conn = sqlite3.connect("audit.db")
    c = conn.cursor()
    c.execute("""CREATE TABLE IF NOT EXISTS checks
                 (ts TEXT, doc_id TEXT, chunk_hash TEXT, model TEXT,
                  prompt_tokens INT, completion_tokens INT,
                  has_dpa INT, liability_cap REAL, law TEXT,
                  missing TEXT, confidence REAL)""")
    c.execute("INSERT INTO checks VALUES (?,?,?,?,?,?,?,?,?,?,?)",
              (datetime.datetime.utcnow().isoformat(), doc_id,
               hashlib.sha256(chunk.encode()).hexdigest()[:16], model,
               usage.prompt_tokens, usage.completion_tokens,
               int(check.has_data_processing_agreement),
               check.liability_cap_usd, check.governing_law,
               ",".join(check.missing_clauses), check.confidence))
    conn.commit(); conn.close()

When missing_clauses is non-empty or confidence is low, open a task in your review tool. The human decision should also be logged with a reviewer ID. This closes the loop and gives you labeled data for eval.

Step 6: Deploy and verify success

Package the steps into a CLI or worker. For verification, build a golden set of ten documents with known compliance states. Run the agent and assert field-level accuracy.

def test_sample_contract():
    pages = extract_pages("sample.pdf")
    chunk = " ".join(pages[:2])
    result, _ = check_chunk(chunk)
    assert result.has_data_processing_agreement is True
    assert result.confidence >= 0.7

Run a batch over the golden set and compute precision/recall on missing_clauses against manual labels. A reasonable bar before production: 95% precision on required-clause detection. If you fall short, adjust chunking or prompt, not the schema.

Success means the agent flags the same gaps a junior associate would, and every decision is reproducible from the audit log. That is what makes an AI agent compliance document checks system defensible in an audit.