Building a production-grade voice AI agent appointment scheduling system is less about ML wizardry and more about wiring speech I/O to a reliable tool-calling loop. This tutorial walks through a working implementation using Twilio for telephony, a SQLite backend, and an OpenAI-compatible LLM endpoint with function calling. By the end you’ll have a phone number that can book medical appointments through natural speech.
Prerequisites
- Python 3.11 or newer
- A Twilio account with a provisioned phone number
- ngrok for exposing your local Flask app to the internet
- An API key for an OpenAI-compatible LLM endpoint (we use n4n.ai’s gateway)
- Install dependencies:
pip install flask twilio openaiYou should also have a basic understanding of REST webhooks and TwiML. No prior speech model training is required; Twilio handles transcription and text-to-speech.
Architecture
The flow is simple:
- Twilio receives a call and sends a webhook to
/voice. - Your server responds with TwiML that uses
<Gather input="speech">to capture the caller’s utterance. - The transcribed text posts to
/process, where an LLM interprets intent and emits tool calls. - Local functions book or query appointments in SQLite.
- The LLM synthesizes a spoken confirmation, and Twilio reads it back.
This keeps the voice AI agent appointment scheduling logic stateless except for the database writes.
1. Database layer
We start with a minimal SQLite store. It is enough for a clinic demo and easy to swap for Postgres later.
import sqlite3
def init_db():
conn = sqlite3.connect("appointments.db")
conn.execute("""
CREATE TABLE IF NOT EXISTS appointments (
id INTEGER PRIMARY KEY,
patient_name TEXT,
phone TEXT,
start_time TEXT,
reason TEXT
)
""")
conn.commit()
conn.close()
def book_appointment(patient_name: str, phone: str, start_time: str, reason: str):
conn = sqlite3.connect("appointments.db")
cur = conn.execute(
"INSERT INTO appointments (patient_name, phone, start_time, reason) VALUES (?, ?, ?, ?)",
(patient_name, phone, start_time, reason),
)
conn.commit()
conn.close()
return {"status": "booked", "id": cur.lastrowid}
def get_available_slots(date_str: str):
# Stub: real systems check practitioner calendars
return [
f"{date_str}T09:00:00",
f"{date_str}T10:00:00",
f"{date_str}T11:00:00",
]Run init_db() once at startup.
2. LLM client and tools
We define two tools the model can call. The base URL points to the gateway, which fronts 240+ models and provides automatic fallback if a provider is degraded.
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key="YOUR_N4N_KEY",
)
tools = [
{
"type": "function",
"function": {
"name": "book_appointment",
"description": "Book a medical appointment for a patient",
"parameters": {
"type": "object",
"properties": {
"patient_name": {"type": "string"},
"start_time": {"type": "string", "description": "ISO 8601 datetime"},
"reason": {"type": "string"},
},
"required": ["patient_name", "start_time", "reason"],
},
},
},
{
"type": "function",
"function": {
"name": "get_available_slots",
"description": "List available appointment slots for a date",
"parameters": {
"type": "object",
"properties": {"date_str": {"type": "string", "description": "YYYY-MM-DD"}},
"required": ["date_str"],
},
},
},
]
def llm_turn(messages):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
tool_choice="auto",
)
return resp.choices[0].messageNote we omit phone from the tool schema; we inject the caller ID server-side to prevent spoofing.
3. Twilio webhook handlers
Create a Flask app with two routes. The first greets and gathers speech; the second processes it.
from flask import Flask, request, Response
from twilio.twiml.voice_response import VoiceResponse, Gather
app = Flask(__name__)
@app.route("/voice", methods=["POST"])
def voice():
resp = VoiceResponse()
gather = Gather(input="speech", action="/process", timeout=5, speech_timeout="auto")
gather.say("Hello, this is the scheduling assistant. What date and time would you like to come in?")
resp.append(gather)
resp.say("We didn't hear you. Goodbye.")
return Response(str(resp), mimetype="application/xml")Expected TwiML output when curling locally:
<Response>
<Gather input="speech" action="/process" timeout="5" speechTimeout="auto">
<Say>Hello, this is the scheduling assistant. What date and time would you like to come in?</Say>
</Gather>
<Say>We didn't hear you. Goodbye.</Say>
</Response>Now the processing route:
@app.route("/process", methods=["POST"])
def process():
speech = request.form.get("SpeechResult", "")
caller = request.form.get("From", "")
messages = [
{"role": "system", "content": "You are a clinic scheduling agent. Use tools to book or check slots."},
{"role": "user", "content": speech},
]
msg = llm_turn(messages)
if msg.tool_calls:
for call in msg.tool_calls:
if call.function.name == "book_appointment":
args = json.loads(call.function.arguments)
args["phone"] = caller
result = book_appointment(**args)
messages.append(msg)
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
final = llm_turn(messages)
reply = final.content
else:
reply = msg.content
resp = VoiceResponse()
resp.say(reply)
resp.redirect("/voice") # allow follow-up turns
return Response(str(resp), mimetype="application/xml")4. Local run and tunnel
Start the server:
python app.pyExpose it:
ngrok http 5000Point your Twilio number’s voice webhook to https://<your-ngrok>.ngrok-free.app/voice.
Sample call flow
Caller: “I need to see Dr. Smith on tomorrow at 10 AM for a checkup.”
Twilio transcribes and posts to /process.
LLM emits book_appointment with patient_name inferred from conversation or prompted earlier, start_time = next day 10:00, reason = checkup.
Database row inserted.
LLM replies: “You’re booked for tomorrow at 10:00 AM for a checkup. Anything else?”
TwiML loops back to /voice.
Checkpoint: query the DB.
sqlite3 appointments.db "SELECT * FROM appointments;"Output similar to:
1|Jane Doe|+15551234567|2025-02-21T10:00:00|checkup5. Hardening the voice AI agent appointment scheduling
The naive loop works, but production needs guardrails.
Timezone handling. Store UTC and convert using the caller’s area code or explicit ask. Don’t let the LLM emit local naive times without qualification.
Confirmation step. Require the model to read back the exact ISO time and ask “Should I confirm?” before writing. This reduces double-bookings.
Fallback. The n4n.ai endpoint automatically falls back when a provider is rate-limited or degraded, so the voice AI agent appointment scheduling stays responsive during spikes. Per-token metering lets you attribute cost per call.
Concurrency. SQLite will lock under parallel writes. Use a connection pool or move to Postgres before handling more than one call per second.
Privacy. Healthcare contexts imply HIPAA. Twilio has a HIPAA-eligible product; ensure your LLM processor signs a BAA or use a self-hosted model through the same OpenAI-compatible interface.
6. Extending to multi-step dialogues
For a real clinic, you’ll want to collect patient name, DOB, and insurance before booking. Implement a small state machine in the session (Twilio passes CallSid; store intermediate slots in Redis). Feed the accumulated slots back to the LLM as a system note:
messages.append({
"role": "system",
"content": f"Current collected data: {json.dumps(state)}"
})The model will only call book_appointment once all required fields are present. This keeps the voice AI agent appointment scheduling robust without custom intent parsers.
Wrap-up
You now have a callable phone number that talks to an LLM, books into a database, and confirms via speech. The pattern—speech capture, tool-calling loop, server-side side-effects—generalizes to any voice AI agent appointment scheduling use case beyond healthcare. Swap the SQLite layer for your EHR API and you’re most of the way to production.