Building a livekit agents whisper real-time voice app means assembling a pipeline where audio transport, speech-to-text, and language modeling operate under tight latency budgets. This guide walks through a runnable Python agent that uses LiveKit for WebRTC transport, Whisper for transcription, and an OpenAI-compatible LLM for responses, then shows how to verify it end to end.
Step 1: Provision LiveKit and install dependencies
You need a LiveKit server reachable via WebSocket. For local dev, the official Docker image is fastest:
docker run -d -p 7880:7880 \
-e LIVEKIT_KEYS="devkey: secret" \
livekit/livekit-serverThat gives you wss://localhost:7880 with API key devkey and secret secret. For production, use LiveKit Cloud or a hardened self-hosted deployment behind a TLS terminator.
Create a Python virtual environment and install the agent framework plus the Whisper and OpenAI plugins:
python -m venv .venv && source .venv/bin/activate
pip install livekit-agents livekit-plugins-whisper livekit-plugins-openai livekit-plugins-silero python-dotenvThe livekit-plugins-silero package provides a local voice activity detector (VAD) that segments audio into utterances—critical because the Whisper API is batch-oriented, not streaming. Without VAD, you would have to guess utterance boundaries yourself.
Store credentials in .env:
LIVEKIT_URL=wss://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret
OPENAI_API_KEY=sk-your-key
# Optional: point LLM at n4n.ai's OpenAI-compatible gateway
LLM_BASE_URL=https://api.n4n.ai/v1
N4N_API_KEY=your-gateway-keyThe livekit agents whisper real-time voice app stack relies on these env vars being present at worker startup; the SDK reads them automatically for LiveKit connection, but we will load them explicitly for the LLM.
Step 2: Understand the Whisper STT constraint
OpenAI’s Whisper API transcribes a complete audio clip. It is not a streaming endpoint. In a livekit agents whisper real-time voice app, you cannot ship 20ms chunks to Whisper and expect low latency. Instead, the whisper.STT plugin relies on a VAD to detect speech boundaries, then sends the buffered utterance (typically 1–4 seconds) to Whisper. This adds a fixed delay equal to the silence padding after each speaker turn.
If you need sub-second STT, use a streaming provider like Deepgram or Whisper.cpp with a local model. But Whisper’s accuracy on noisy channels is still compelling, and the batch approach is fine for conversational agents where a 500ms–1s lag is acceptable.
Initialize the STT object:
from livekit.plugins import whisper
stt = whisper.STT(model="whisper-1") # uses OPENAI_API_KEY by defaultThe plugin handles WAV encoding and chunk size. You can pass temperature or language kwargs if you know the call is monolingual; otherwise let Whisper auto-detect.
Step 3: Wire the LLM and TTS
The language model drives the dialogue. The openai.LLM plugin speaks the OpenAI Chat Completions protocol. If you want automatic fallback when a provider is rate-limited, point base_url at an OpenAI-compatible gateway. For example, n4n.ai exposes one endpoint that fronts 240+ models and honors client routing directives; swap your LLM_BASE_URL and key and the rest of the code stays identical.
from livekit.plugins import openai
import os
llm = openai.LLM(
model="gpt-4o-mini",
base_url=os.environ.get("LLM_BASE_URL"), # None falls back to OpenAI
api_key=os.environ.get("N4N_API_KEY") or os.environ["OPENAI_API_KEY"],
)
tts = openai.TTS(model="tts-1", voice="alloy")openai.TTS streams audio back to the room using the standard OpenAI TTS endpoint. For lower latency, consider tts-1-hd only if your users are on high-bandwidth links; the base model is snappy enough. The LLM streams tokens to the TTS engine as they arrive, so the user starts hearing the answer before the full sentence is generated.
Step 4: Assemble the agent session
LiveKit Agents abstracts the real-time loop into an AgentSession. You hand it the STT, LLM, TTS, and VAD; the session manages room events, interruption, and turn-taking.
import os
from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import whisper, openai, silero
load_dotenv()
async def entrypoint(ctx: JobContext):
await ctx.connect()
session = AgentSession(
stt=whisper.STT(model="whisper-1"),
llm=openai.LLM(
model="gpt-4o-mini",
base_url=os.environ.get("LLM_BASE_URL"),
api_key=os.environ.get("N4N_API_KEY") or os.environ["OPENAI_API_KEY"],
),
tts=openai.TTS(model="tts-1", voice="alloy"),
vad=silero.VAD.load(),
)
agent = Agent(instructions="You are a concise voice assistant. Keep answers under 30 words.")
await session.start(agent, room=ctx.room)
await session.say("Hello, I'm listening.")
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))This is the entire core of a livekit agents whisper real-time voice app. The Agent object holds system instructions; session.say triggers an immediate spoken greeting. The session automatically subscribes to microphone tracks from other participants and runs the STT→LLM→TTS cycle.
Step 5: Run the worker and connect a client
Start the worker:
python agent.py startThe process registers with your LiveKit server and waits for room jobs. To simulate a user, use the LiveKit CLI or the hosted sandbox:
livekit-cli join-room \
--url wss://localhost:7880 \
--api-key devkey --api-secret secret \
--room test \
--identity user1Or embed the room URL in a minimal web client using the LiveKit JS SDK. When the user joins, the agent automatically connects (because ctx.connect() happens in the entrypoint) and speaks its greeting.
Verify success: Watch the worker logs. You should see room joined, then the agent’s greeting audio appears in the client. Speak a sentence like “What time is it?” After a short pause (VAD tail), the log prints transcription: What time is it? and the LLM response is spoken. If you see both, the pipeline works. If you only see the greeting but no transcription, check that the participant’s microphone permission is granted and that Silero loaded without CUDA errors.
Step 6: Tune for real-time interaction
Default VAD settings are conservative. In a noisy office, you’ll get premature cutoffs. Adjust Silero parameters:
vad=silero.VAD.load(
min_silence_duration=0.4,
speech_pad_ms=300,
)Lower min_silence_duration makes the agent respond faster but risks barge-in errors. Enable interruption handling by setting session.interrupt_on_vad=True (default in recent SDKs) so the user can talk over the agent. The session will cancel the current TTS playback and start a new STT cycle.
For a robust livekit agents whisper real-time voice app, also log transcriptions to confirm Whisper isn’t dropping long utterances:
@session.on("transcription")
def on_transcript(ev):
print("USER SAID:", ev.text)If you notice Whisper returning empty strings on short clips, increase speech_pad_ms to capture more context around the voice activity.
Step 7: Production concerns
Scale workers horizontally; LiveKit distributes rooms across available agents. If you used the n4n.ai gateway for the LLM, you get per-token usage metering out of the box and automatic fallback if a backing provider is degraded—no code change. Otherwise, implement your own retry with exponential backoff on 429s.
Whisper calls are metered per audio minute by OpenAI. Cache common phrases? Not worth it; TTS is cheaper. Focus on trimming silence padding to reduce STT cost and avoid sending silent segments.
Finally, secure the WebSocket with TLS and rotate API keys. The agent code above reads from env, so rotation is a deploy-time change, not a code change. Add a health check that calls ctx.api.room.list_rooms periodically to confirm the worker can reach the server.
That is a shippable voice agent. The pattern—capture, segment, batch-transcribe, reason, speak—extends to any STT/LLM/TTS combo without rewriting the session logic.