AI & MLLLMOpenAITelephonyPython

Bridging Phone Calls to an OpenAI Realtime Agent

A voice agent in a browser and a voice agent on the telephone have different integration problems. The telephone version needs call initiation, routing, lifecycle events, and a way to connect the audio session to the right application context.

The voice-agent caller project in my files explores that boundary with a FastAPI service, Twilio calls, SIP, and OpenAI Realtime integration.

The application around the conversation

The service's documented responsibilities include initiating calls, handling webhooks, bridging the SIP/Realtime path, and recording structured conversation events.

Those pieces give the model a place inside a larger call lifecycle. A conversation session does not begin merely because the application requested a phone call; the relevant participants and media path still need to connect.

Structured events are useful because they let an operator distinguish the stages: request received, call initiated, session established, conversation event recorded, and call ended.

Realtime behavior changes the engineering problem

Telephone interaction is synchronous. Delays and interruptions become part of the experience rather than a background processing concern.

The integration must also decide what happens when the call ends unexpectedly, when a webhook arrives twice, or when an audio session outlives the intended task. Those are application questions around the model, even when the model handles speech interaction.

A prototype with a clear boundary

The retained source establishes the integration approach. A remaining task note calls for call and scheduler verification, so this post describes a prototype rather than claiming demonstrated end-to-end reliability.

The technical focus here is a conversational Realtime connection. In a separate project for scheduled reminder calls, launch behavior is deliberately constrained to approved reminder playback.

A public demonstration would use controlled, authorized test calls and synthetic records. Real phone numbers, transcripts, recordings, and provider configuration do not belong in a companion repository.

Related background