Building a multilingual voice AI agent with Whisper

A multilingual voice AI agent Whisper combination gets you further than most bespoke ASR stacks because Whisper handles dozens of languages out of the box and detects the spoken language without a separate classifier. This tutorial builds a runnable agent that listens to an audio file, transcribes with Whisper, replies via an LLM in the same language, and speaks the answer back using a lightweight TTS engine.

Prerequisites

You need Python 3.11+ and an OpenAI API key for Whisper. For the language model, any OpenAI-compatible endpoint works; we’ll route through n4n.ai’s OpenAI-compatible endpoint to get automatic fallback across providers without changing client code.

pip install openai gtts soundfile numpy fastapi uvicorn python-dotenv

Create a .env file:

OPENAI_API_KEY=sk-...
N4N_API_KEY=sk-...   # if using n4n.ai for LLM routing

We assume a WAV file recorded at 16 kHz mono. Whisper expects PCM 16-bit; most clips from phones or Zoom exports need conversion, but for this tutorial we’ll pass a clean WAV.

Step 1: Transcribing audio with Whisper

Whisper’s transcription endpoint auto-detects language when you omit the language parameter. The response includes the detected language code and the full text.

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def transcribe(path: str) -> dict:
    with open(path, "rb") as f:
        resp = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",
        )
    return resp.model_dump()

result = transcribe("sample.es.wav")
print(result["language"], "|", result["text"])

Expected output for a Spanish clip:

es | Hola, ¿puedes decirme qué hora es en Tokio ahora?

The verbose_json format returns language, duration, and segment-level timing. For an agent, the top-level language and text are enough.

Step 2: Mapping language codes to TTS voices

gTTS supports ISO 639-1 codes directly, but some Whisper codes (zh, pt, fr) map cleanly while others like yue (Cantonese) need a fallback. Build a small resolver:

def tts_lang(whisper_code: str) -> str:
    # gTTS uses same codes for major languages; default to English on miss
    supported = {"es", "fr", "de", "it", "pt", "ja", "ko", "zh", "en", "hi"}
    return whisper_code if whisper_code in supported else "en"

This keeps the agent from crashing on minority languages Whisper detects but gTTS can’t synthesize.

Step 3: Routing the LLM reply in the detected language

The agent must answer in the user’s language. We set a system prompt that forces the model to mirror the input language, then call a chat completion. Using an OpenAI-compatible client means we can point base_url at a gateway.

from openai import OpenAI as GatewayClient

# Route through n4n.ai for provider fallback and per-token metering
llm = GatewayClient(
    api_key=os.environ["N4N_API_KEY"],
    base_url="https://api.n4n.ai/v1",
)

def reply(user_text: str, lang: str) -> str:
    sys = (
        f"You are a concise voice assistant. Reply in the same language as the user "
        f"(detected: {lang}). No markdown, no code blocks, speakable sentences only."
    )
    resp = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": sys},
            {"role": "user", "content": user_text},
        ],
        temperature=0.3,
        max_tokens=120,
    )
    return resp.choices[0].message.content.strip()

answer = reply(result["text"], result["language"])
print(answer)

Expected output (Spanish):

Son las dos de la madrugada en Tokio, hora estándar de Japón.

The gateway forwards provider cache-control hints and honors routing directives, so if the primary model is degraded it fails over without client changes.

Step 4: Synthesizing speech

gTTS is synchronous and writes an MP3. For a voice agent you’d stream, but a file is fine for a tutorial checkpoint.

from gtts import gTTS

def speak(text: str, lang: str, out="reply.mp3"):
    gTTS(text=text, lang=tts_lang(lang)).save(out)
    return out

speak(answer, result["language"])

Play reply.mp3 with any player. The audio matches the detected Spanish input.

Step 5: Wrapping in a FastAPI agent endpoint

A real agent needs an HTTP surface. Below is a minimal POST endpoint that accepts a WAV, runs the pipeline, and returns the transcript, reply text, and audio URL.

from fastapi import FastAPI, UploadFile
import tempfile, os

app = FastAPI()

@app.post("/agent")
async def agent(audio: UploadFile):
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tf:
        tf.write(await audio.read())
        path = tf.name

    tr = transcribe(path)
    ans = reply(tr["text"], tr["language"])
    mp3 = speak(ans, tr["language"], out=path + ".mp3")

    return {
        "language": tr["language"],
        "transcript": tr["text"],
        "reply": ans,
        "audio_url": "/static/" + os.path.basename(mp3),
    }

Run with uvicorn main:app --port 8000. Curl test:

curl -F "audio=@sample.es.wav" http://localhost:8000/agent

Expected JSON:

{
  "language": "es",
  "transcript": "Hola, ¿puedes decirme qué hora es en Tokio ahora?",
  "reply": "Son las dos de la madrugada en Tokio, hora estándar de Japón.",
  "audio_url": "/static/sample.es.wav.mp3"
}

Handling multilingual edge cases

Whisper sometimes mis-detects short clips. If the transcript is under 3 words, force a second pass with language="en" or use the prompt parameter to bias toward expected vocabulary:

resp = client.audio.transcriptions.create(
    model="whisper-1",
    file=f,
    response_format="verbose_json",
    prompt="Common travel phrases in French or English",
)

For code-switching (e.g., Spanglish), Whisper returns the dominant language code. The LLM system prompt should tolerate mixed input; instruct it to reply in the dominant language only.

Latency and cost notes

Whisper-1 pricing is per-minute of audio, not per-token, so long silences cost you. Trim silence client-side with webrtcvad before upload. The LLM call is the other cost axis; max_tokens=120 keeps replies speakable and cheap. If you run this at scale, batch transcription asynchronously and cache repeated queries (e.g., “what time is it”) keyed by normalized transcript.

What we built

A multilingual voice AI agent Whisper pipeline that needs no language selector, no separate detector, and no custom ASR training. The agent transcribes, detects, replies in-language via an OpenAI-compatible LLM, and speaks back. Swap the WAV input for a microphone stream and the MP3 output for a WebSocket audio sink, and you have a production voice bot.

For the LLM leg, routing through a gateway that aggregates 240+ models means you can A/B test response quality without rewriting the client. The same code works for a tiny open model or a frontier model by changing one string.