How AI agents summarize clinical notes for physicians

AI agents clinical notes summarization has moved from research demo to deployable backend service, but the gap between a prompt and a trustworthy clinical tool is wide. This guide lays out a concrete pipeline you can stand up in an afternoon and harden for production: ingest, chunk, summarize with a model loop, validate, and deliver. Every step includes runnable code and the operational caveats we learned shipping similar systems.

Step 1: Define the summary contract

Physicians do not want a wall of prose. They want fixed sections: chief complaint, history of present illness (HPI), assessment, plan, active medications, and pending follow-ups. Lock this schema before writing any LLM code. A stable contract lets you render the same summary in an EHR widget, a pager message, or a PDF without re-prompting.

Use a JSON schema the model must fill. This makes downstream rendering trivial and lets you validate completeness programmatically.

{
  "type": "object",
  "properties": {
    "chief_complaint": {"type": "string"},
    "hpi": {"type": "string"},
    "assessment": {"type": "string"},
    "plan": {"type": "array", "items": {"type": "string"}},
    "active_medications": {"type": "array", "items": {"type": "string"}},
    "followups": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["chief_complaint", "hpi", "assessment", "plan"]
}

Store this as summary_schema.json. Your agent will pass it via response_format (structured outputs) or function calling depending on the model. Do not let the model free-form text; physicians will not parse inconsistent keys.

Step 2: Ingest and chunk clinical notes safely

Pull the raw note from your EHR. In most US systems that means a FHIR DocumentReference or a ClinicalNote resource. If you are outside a BAA-covered environment, strip direct identifiers (MRN, name, DOB, address) at the edge. In a covered deployment, still encrypt in transit and log access.

Clinical notes often exceed 8k tokens. Naive truncation loses the plan section because it sits at the end. Chunk by section headers first, then by character count as fallback.

import re

def chunk_note(text: str, max_chars: int = 6000) -> list[str]:
    # Split on common clinical headers like "Chief Complaint:" or "Assessment:"
    sections = re.split(r'(?m)^(?=[\w ]+:)', text)
    chunks = []
    current = ""
    for sec in sections:
        if len(current) + len(sec) <= max_chars:
            current += sec
        else:
            if current:
                chunks.append(current)
            if len(sec) > max_chars:
                for i in range(0, len(sec), max_chars):
                    chunks.append(sec[i:i+max_chars])
                current = ""
            else:
                current = sec
    if current:
        chunks.append(current)
    return chunks

Run this on a sample note. You should get 1–4 chunks for a typical encounter. If you see more than six, your max_chars is too small or the note is unusually long—escalate to a human.

Step 3: Build the summarization agent loop

The agent calls an LLM per chunk, then merges. We route through an OpenAI-compatible endpoint to avoid vendor lock and get fallback. For example, n4n.ai exposes one endpoint that fronts 240+ models and automatically fails over when a provider is rate-limited, which matters when a single clinic bursts 500 notes at end of day.

Point the OpenAI client at the gateway, pass the schema, and keep temperature at zero.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY"
)

SYSTEM_PROMPT = (
    "You are a clinical summarization agent. Extract the required fields "
    "from the note chunk. Output strict JSON matching the schema. "
    "Do not invent findings not present."
)

def summarize_chunk(chunk: str, schema: dict) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": chunk}
        ],
        response_format={"type": "json_schema", "schema": schema},
        temperature=0.0
    )
    return json.loads(resp.choices[0].message.content)

Merge chunk outputs by concatenation for free-text fields and union for arrays. For a small practice this is enough; for specialists you may need a second pass to deduplicate near-identical plan items.

def merge_summaries(parts: list[dict], schema: dict) -> dict:
    merged = {k: "" for k in ["chief_complaint", "hpi", "assessment"]}
    merged["plan"] = []
    merged["active_medications"] = []
    merged["followups"] = []
    for p in parts:
        for k in merged:
            if isinstance(merged[k], list):
                merged[k].extend(p.get(k, []))
            else:
                merged[k] += "\n" + p.get(k, "")
    for k in ["plan", "active_medications", "followups"]:
        merged[k] = list(dict.fromkeys(merged[k]))
    return merged

Add a retry wrapper: on APIError or empty content, sleep and switch model name via the same endpoint. The gateway handles provider fallback, but you should still catch parse errors.

Step 4: Validate and self-critique

A summary that drops the antibiotic plan is dangerous. Run a critique pass: feed the original note and the merged summary to a model with a strict rubric. Use a different model family than the summarizer to reduce shared blind spots.

CRITIQUE_PROMPT = """
Compare the source note and the summary. Flag any of:
1. Missing required section
2. Fact not supported by source
3. PHI leak (name, MRN, address)
Return JSON: {"ok": bool, "issues": [str]}
"""

def critique(note: str, summary: dict) -> dict:
    resp = client.chat.completions.create(
        model="claude-3-5-sonnet",
        messages=[
            {"role": "system", "content": CRITIQUE_PROMPT},
            {"role": "user", "content": f"NOTE:\n{note}\n\nSUMMARY:\n{json.dumps(summary)}"}
        ],
        response_format={"type": "json_object"},
        temperature=0.0
    )
    return json.loads(resp.choices[0].message.content)

If ok is false, route to a human queue or re-run with a stronger model. Do not auto-deliver on failure. Keep the critique issues in your audit log; they are the fastest way to find prompt drift.

Step 5: Render and deliver to the physician

The merged, validated summary goes back to the EHR or a secure inbox. A minimal TypeScript snippet for a web dashboard:

async function postSummary(encounterId: string, summary: object) {
  const res = await fetch(`/api/encounters/${encounterId}/summary`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(summary)
  });
  if (!res.ok) throw new Error('summary delivery failed');
  return res.json();
}

Keep the UI read-only with a “verify” button. Physicians trust summaries only after they sign off. Store the signed summary as a new FHIR Composition so it is part of the legal record.

Step 6: Verify success

You cannot ship AI agents clinical notes summarization on vibes. Use this verification plan before any real patient data touches the system:

1. Synthetic gold standard: Write 20 fake notes with known sections. Generate summaries and diff against hand-written targets. Expect section coverage of 100% on chief complaint and assessment; partial credit on HPI.
2. PHI leak scan: Run the critique pass on all outputs. Zero leaks is the only acceptable count.
3. Latency check: Measure p95 time per note on your hardware. A 4-chunk note should summarize in under 20 seconds with small models.
4. Clinician review: Have two physicians rate 50 real summaries on a 1–5 accuracy scale. Target median 4+.

Automate steps 1–3 in CI:

# run synthetic eval
python eval_summaries.py --notes synthetic/ --out report.json
# fail build if leaks > 0
grep -q '"ok": false' report.json && exit 1

If any gate fails, adjust chunk size or model routing before expanding to more clinics.

Operational notes

Cache control matters. Forward provider cache hints so repeated section headers are not re-tokenized. The gateway we used honors client routing directives and forwards cache-control hints, trimming cost on high-volume days.

Per-token metering is not optional. You need line-item cost attribution per clinic to avoid a surprise invoice. Use the usage field from each response and ship it to your billing pipeline:

usage = resp.usage
print(usage.prompt_tokens, usage.completion_tokens)

AI agents clinical notes summarization is a solvable engineering problem. The hard parts are chunking without losing context, validating without a human in every loop, and delivering where the physician already works. Build the pipeline above, measure against the verification plan, and iterate on the critique rubric.