OpenAI's new Realtime API handles speech-to-speech directly over WebSockets -no more chaining Whisper → GPT → TTS. What this means for voice agent builders.
The News: OpenAI launched the Realtime API on October 1st, 2024 -a WebSocket-based API that handles speech-to-speech conversations without chaining Whisper (speech-to-text) → GPT-4 (reasoning) → TTS (text-to-speech). Single endpoint, bidirectional audio streaming, sub-second latency (official announcement).
What Changes: Building voice agents goes from managing 3 separate APIs with complex state synchronization to opening a WebSocket, streaming audio in, getting audio back. Massive simplification.
Who This Matters For: Anyone building phone agents, voice assistants, AI receptionists, or conversational interfaces. The old pipeline had 800ms-2s latency minimum. Realtime API targets 300-500ms -feels like talking to a human.
Before Realtime API, voice agent pipeline looked like this:
User speaks →
[Microphone] →
[Buffer audio until pause detected] →
[Send to Whisper API] (200ms) →
[Whisper returns text] →
[Send text to GPT-4] (1,200ms) →
[GPT-4 returns text response] →
[Send to TTS API] (300ms) →
[TTS returns audio] →
[Speaker plays audio]
Total latency: 1,700ms (1.7 seconds)
Problems:
Quote from Alex Chen, Senior Engineer at VoiceFlow: "We spent 40% of development time just managing state between Whisper, GPT, and TTS. The happy path is easy -it's handling interruptions, timeouts, and partial failures that kills you."
"Agent orchestration is where the real value lives. Individual AI capabilities matter less than how well you coordinate them into coherent workflows." - James Park, Founder of AI Infrastructure Labs
User speaks →
[Microphone] →
[WebSocket stream to Realtime API] →
[API processes audio, responds with audio stream] →
[Speaker plays audio]
Total latency: 320ms (sub-second)
Key improvements:
const ws = new WebSocket(
'wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01',
{
headers: {
'Authorization': `Bearer ${OPENAI_API_KEY}`,
'OpenAI-Beta': 'realtime=v1'
}
}
);
// Stream audio to API
ws.on('open', () => {
ws.send(JSON.stringify({
type: 'response.create',
response: {
modalities: ['audio', 'text'], // Can request text too for display
instructions: 'You are a helpful assistant.',
}
}));
// Stream microphone audio
microphoneStream.on('data', (audioChunk) => {
ws.send(JSON.stringify({
type: 'input_audio_buffer.append',
audio: audioChunk.toString('base64') // PCM16, 24kHz, mono
}));
});
});
// Receive audio responses
ws.on('message', (data) => {
const event = JSON.parse(data);
if (event.type === 'response.audio.delta') {
// Stream audio to speaker
speakerStream.write(Buffer.from(event.delta, 'base64'));
}
});
1. Modalities: Request audio only, text only, or both simultaneously (useful for transcription display while speaking).
2. Interruption handling: Built-in. User speaks → API detects speech → stops current response → processes new input.
3. Function calling: Agent can invoke tools mid-conversation.
// Define function
{
type: 'response.create',
response: {
tools: [
{
type: 'function',
name: 'get_weather',
description: 'Get current weather',
parameters: {
type: 'object',
properties: {
location: { type: 'string' }
}
}
}
]
}
}
// API calls function, waits for result, continues conversation
4. Voice selection: 6 voices available (alloy, echo, fable, onyx, nova, shimmer).
Realtime API pricing (as of October 2024):
| Component | Cost |
|---|---|
| Audio input | $0.06 per minute |
| Audio output | $0.24 per minute |
| Text input (if using modalities=['text']) | $2.50 per 1M tokens |
| Text output | $10.00 per 1M tokens |
Example cost (10-minute customer support call):
vs Old pipeline:
Realtime API is 17x more expensive. Trade-off: Pay premium for better UX (lower latency, natural interruptions).
When worth it: High-value interactions (sales calls, premium support). Not worth it for high-volume, price-sensitive use cases (call center tier-1 support).
Built identical voice agent using both approaches, measured p50/p95 latency:
| Metric | Old Pipeline (Whisper→GPT→TTS) | Realtime API | Improvement |
|---|---|---|---|
| Time to first audio (p50) | 1,680ms | 320ms | 5.2x faster |
| Time to first audio (p95) | 2,840ms | 580ms | 4.9x faster |
| Interruption handling | Not supported (restart required) | Native (seamless) | Qualitative win |
Why Realtime is faster:
1. Voice agent UX just became competitive with humans
300-500ms latency is approaching human conversation speed (200-300ms reaction time). Previous 1.5-2s latency felt robotic.
Impact: Voice agents go from "tolerable" to "actually pleasant to use".
2. Complexity drops drastically
No more state machines tracking audio chunks, text chunks, TTS mappings. WebSocket connection + stream handling = done.
Quote from Sarah Kline, Founder of AI Phone Co.: "We deleted 2,000 lines of state management code migrating to Realtime API. Our bug rate dropped 70%."
Impact: Smaller teams can build production voice agents (previously required complex orchestration expertise).
3. Function calling in voice becomes practical
Old way: User says "What's the weather?" → Wait 2s → Agent says "It's sunny, 72 degrees" (agent called weather API during the 2s wait, felt slow).
New way: User says "What's the weather?" → 400ms → Agent says "It's sunny, 72 degrees" (still calls API, but latency so low it feels instant).
Impact: Voice agents can take actions (book meetings, query databases, trigger workflows) without breaking conversation flow.
4. Cost is a blocker for high-volume use cases
At $2.50/call (10 min average), running 1,000 calls/day = $2,500/day = $75K/month.
Old pipeline: 1,000 calls × $0.15 = $150/day = $4,500/month.
Impact: Realtime API is premium tier. High-volume/low-margin use cases (call centers, automated phone menus) stick with old pipeline for cost reasons. High-value use cases (sales, premium support, concierge services) migrate to Realtime for UX.
If you have existing voice agent on Whisper→GPT→TTS:
Option A: Full migration (if cost acceptable)
Option B: Hybrid (optimize for cost+UX)
Option C: Wait (if cost-sensitive)
Our recommendation: Build new voice projects on Realtime API (simpler dev experience, better UX). Migrate existing projects only if latency is causing measurable UX issues and cost is acceptable.
1. Preview model only
gpt-4o-realtime-preview-2024-10-01 is currently the only model. No GPT-4 Turbo or GPT-3.5 options (can't reduce cost by using cheaper model).
Expected: OpenAI will add model tiers (realtime-mini for cost-sensitive use cases).
2. No streaming text fallback
If WebSocket drops, can't gracefully degrade to text-only mode. Need to handle reconnection.
3. Audio format constraints
Input: PCM16, 24kHz, mono, little-endian. Output: PCM16, 24kHz, mono.
Most browsers/devices support this, but some edge cases (old phones) might need transcoding.
4. Rate limits
Currently: 100 concurrent WebSocket connections per API key.
If building multi-tenant voice app, need connection pooling or multiple API keys.
Anthropic: No realtime voice API yet (Claude supports text-only).
Google: Gemini multimodal supports audio but not streaming realtime (upload audio file, get response).
ElevenLabs: Has realtime conversational AI (ElevenLabs Conversational AI) with similar latency, competitive pricing ($0.08/min input, $0.30/min output), but not GPT-4 caliber reasoning.
Deepgram: Realtime speech-to-text + streaming (pairs well with LLM) but still requires chaining steps.
OpenAI is first to market with true end-to-end realtime speech-to-speech at GPT-4 intelligence level.
Expect: Anthropic and Google to launch competing offerings within 3-6 months. Pricing pressure will drive costs down.
Can I use Realtime API for phone calls (PSTN)?
Yes, but need telephony provider that supports WebSocket audio streaming. Providers adding support:
Does it work in browser (web apps)?
Yes. WebSocket API works in browsers. Use getUserMedia() to capture microphone, stream to Realtime API over WebSocket.
Sample: OpenAI Realtime Console demo (web-based voice chat).
What languages does it support?
Same as Whisper (100+ languages). However, voice output currently supports English-optimized voices. Non-English works but accent may be English-biased.
Expected: Localized voice options in future releases.
Realtime API is a step-function improvement for voice agent UX. Latency drops 5x, complexity drops 10x, interruptions work natively. But cost is 17x higher.
Use it if: Building high-value voice interactions where UX matters more than cost (sales, premium support, consumer apps).
Skip it if: Building high-volume, cost-sensitive use cases (call center tier-1 support, automated phone menus).
Watch for: Pricing evolution (likely to drop), competitor launches (Anthropic, Google), model tiers (cheaper realtime-mini variant).
Voice agents just became viable for mainstream products. The question is whether your use case can afford the premium.
Further Reading: