An AI voice agent on Twilio is a phone call wired to four moving parts: Twilio Voice to carry the call, speech-to-text (STT) to hear the caller, a large language model (LLM) to decide what to say and text-to-speech (TTS) to say it. In 2026 you have two main ways to connect them on Twilio. ConversationRelay handles STT, TTS and interruptions for you and sends your app plain text over a WebSocket. Media Streams sends you raw call audio and leaves the rest to you.
For most teams, ConversationRelay is the faster route to a working agent, with ElevenLabs voices as the default TTS. Media Streams makes sense when you want full control over the audio pipeline or want to plug in a speech-to-speech model. A third option, ElevenLabs’ own Agents platform with a native Twilio number import, gets you to a demo fastest but moves more of the stack outside your code.
This guide covers the architecture, how to keep latency low, how to handle callers who talk over the agent, how to connect your CRM, the compliance rules that apply to AI calls and what each part costs. Prices were checked on Twilio and ElevenLabs pricing pages as of October 2026.
Every turn in a voice conversation runs the same loop. The caller speaks, Twilio carries the audio, an STT engine turns it into text, your LLM produces a reply, a TTS engine turns that reply into audio and Twilio plays it back. Your server sits in the middle, holding conversation state, calling tools and deciding when to hand the caller to a person.
Telephony: a Twilio phone number (or SIP trunk) receives or places the call. TwiML tells Twilio what to do when the call connects.
Audio bridge: ConversationRelay or Media Streams opens a WebSocket between Twilio and your server.
STT: converts caller speech into text, ideally streaming partial results as the caller talks.
LLM and orchestration: your code sends the transcript, system prompt and conversation history to a model, handles function calls and streams the reply.
TTS: converts the reply into speech. Voice quality here shapes how human the agent feels.
Back-office systems: CRM, scheduling, order and ticketing systems the agent reads from and writes to.
The design choice that matters most is where STT and TTS live. Either Twilio runs them for you, or you run them yourself on audio Twilio streams to you.
ConversationRelay vs Media Streams vs ElevenLabs Agents
ConversationRelay became generally available in May 2025. It is a TwiML noun you place inside <Connect>. Twilio handles speech recognition, speech synthesis and interruption handling, and your WebSocket server only exchanges text. You bring your own LLM. TTS providers are ElevenLabs (the default), Google and Amazon; STT providers are Deepgram (the default for newer accounts) and Google.
Media Streams is lower level. A bidirectional stream, started with <Connect><Stream>, sends your server base64-encoded mu-law audio at 8 kHz and accepts audio back in the same format. You choose and run STT, TTS or a realtime speech-to-speech model, and you build interruption logic yourself.
ElevenLabs Agents can import a Twilio number directly. ElevenLabs configures the number and its platform handles STT, the LLM call, TTS and turn-taking. You configure the agent, knowledge and tools in ElevenLabs.
Factor
ConversationRelay
Media Streams
ElevenLabs Agents + Twilio
What you build
Text-in, text-out WebSocket app plus LLM logic
Full audio pipeline, STT, TTS, turn-taking
Agent config, prompts, tools and webhooks
STT and TTS
Managed by Twilio (Deepgram or Google STT; ElevenLabs, Google or Amazon TTS)
Your choice, you integrate it
Managed by ElevenLabs
Interruption handling
Built in
You build it using clear and mark messages
Built in
LLM choice
Any, you call it
Any, including speech-to-speech models
Models offered in ElevenLabs, or a custom LLM
Control and portability
High
Highest
Lower, more lives in the vendor platform
Time to first working call
Hours to days
Days to weeks
Hours
Best for
Most production agents on Twilio
Teams with audio expertise or custom models
Fast pilots, small teams, voice-first use cases
Building With ConversationRelay Step By Step
Here is the shape of a production build using ConversationRelay, the path Twilio itself promotes for AI agents.
1. Point A Number At Your TwiML
Buy or port a Twilio number and set its voice webhook to an endpoint that returns TwiML with <Connect><ConversationRelay url="wss://...">. Set attributes such as welcomeGreeting for the first line, language, ttsProvider and voice, and transcriptionProvider. Pass caller context with custom <Parameter> values.
2. Stand Up The WebSocket Server
Your server receives a setup message when the session starts, then prompt messages containing the caller’s transcribed speech. It replies with text token messages, which Twilio speaks. Validate Twilio’s request signature and keep the server close to Twilio’s region to cut network time.
3. Stream The LLM Reply
Call your LLM with streaming on and forward tokens to Twilio as they arrive, rather than waiting for the full reply. Keep system prompts short and specific to the phone: short sentences, no lists or markdown, confirm details back to the caller and always offer a path to a person.
4. Add Tools And Handoff
Expose functions such as “look up order,” “book appointment” or “create ticket” to the model. When the agent needs a person, send an end session message with handoffData and use the follow-up TwiML to route the call to a queue, a <Dial> or a contact center.
5. Log, Test And Tune
ConversationRelay has a debug option that can report speaker events and the tokens actually played. Log every turn with timings so you can see where latency comes from, and test with real accents, background noise and callers who interrupt.
Latency: Where The Delay Comes From
On the phone, silence feels broken. Each turn’s delay is the sum of several steps: detecting that the caller has stopped speaking, finalizing the transcript, the LLM’s time to first token, TTS time to first audio and network hops between each service. Any one of them can add hundreds of milliseconds.
Stream everything. Streaming STT, streaming LLM output and streaming TTS overlap work instead of running it in sequence.
Pick fast models for the voice path. Smaller LLMs often beat larger ones on the phone. ElevenLabs offers low-latency Flash and Turbo TTS models alongside its higher-fidelity multilingual model.
Co-locate services. Host your WebSocket server and LLM endpoint in the same region as your Twilio traffic.
Cover slow tool calls. If a CRM lookup takes two seconds, have the agent say “let me check that” first.
Measure per stage. Without per-turn timing logs, you are guessing which piece to fix.
Handling Interruptions And Turn-Taking
People interrupt. A caller who says “no, the other address” halfway through the agent’s sentence expects it to stop and listen. Agents that keep talking feel robotic and frustrate callers fast.
ConversationRelay handles this with its interruptible attribute, which can be set to any (the default), speech, dtmf or none. When the caller interrupts, playback stops and your app is told what was cut off, so you can trim the unspoken part from conversation history. The reportInputDuringAgentSpeech setting controls whether you receive caller input while the agent is talking. Use none for legally required disclosures that must play in full.
With Media Streams, you build this yourself. Run voice activity detection on inbound audio; when the caller starts talking, send a clear message to flush buffered audio. Twilio then returns any pending mark messages, which tells you exactly which audio never played. Getting this right, without cutting off the agent every time the caller coughs, is one of the hardest parts of a custom build.
Connecting Tools, CRMs And Human Handoff
A voice agent that cannot look anything up is a talking FAQ. The value comes from actions: checking an order, rescheduling an appointment, qualifying a lead into the CRM or opening a support ticket.
Identify the caller early. Match the caller ID to a CRM record before the greeting so the agent can say “Hi Maria, are you calling about Thursday’s appointment?”
Keep tools narrow. “Get next three open slots for provider X” is safer and faster than raw database access.
Write back every call. Log a summary, outcome and any captured fields to the CRM.
Design the handoff. Transfer to a person with context attached, so the caller never repeats themselves. On Twilio, that can mean routing to Twilio Flex or another contact center platform.
AI voice agents carry real legal exposure. These are the areas to review with counsel before launch:
Outbound AI calls and the TCPA. In February 2024, the FCC ruled that AI-generated voices count as “artificial” voices under the Telephone Consumer Protection Act. Outbound calls using them generally need the called party’s prior express consent, and marketing calls need prior express written consent.
Recording consent. US states differ: some require consent from one party, others from everyone on the call. A short disclosure at the start (“this call is recorded and handled by an automated assistant”) is the safe default.
AI disclosure. Several jurisdictions require or are moving toward telling callers they are talking to AI. Disclosing up front also builds trust.
Healthcare. Twilio describes ConversationRelay as a HIPAA-eligible service when configured properly, which requires a signed Business Associate Agreement with Twilio. Check that every other vendor touching patient data will also sign one.
Payments. Twilio notes ConversationRelay supports PCI workflows only with PCI-compliant TTS and STT providers, and card data must not be placed in greetings, parameters or handoff data. Many teams route payment capture to a dedicated secure flow instead.
What A Twilio AI Voice Agent Costs
Per-minute costs stack up from several vendors. The Twilio and ElevenLabs figures below are list prices from each vendor’s site as of October 2026; LLM costs depend entirely on the model and prompt size.
Cost Item
List Price
Notes
Twilio inbound local call
$0.0085 per minute
Toll-free inbound is $0.022 per minute
Twilio outbound local call
$0.014 per minute
US and Canada
Twilio phone number
$1.15 per month local, $2.15 toll-free
Per number
ConversationRelay
$0.07 per minute
Voice minutes billed separately
Media Streams
$0.0044 per minute
You pay STT and TTS vendors separately
Call recording
$0.0025 per minute
Plus storage
ElevenLabs Agents (if used instead)
From $0.08 per minute on listed plans
LLM usage and telephony billed separately
LLM tokens
Varies by model
Grows with prompt length and conversation history
A rough example: an inbound agent on a local number using ConversationRelay runs about $0.0785 per minute in Twilio charges ($0.0085 voice plus $0.07 ConversationRelay), before LLM tokens. At 20,000 minutes a month, that is about $1,570 in Twilio usage. Add your LLM bill, hosting, monitoring and the engineering time to maintain the agent. Volume discounts are available from both vendors, so check current pricing on Twilio’s voice pricing page before you model a business case. For a broader breakdown, see our Twilio pricing guide.
The biggest cost drivers are usually not per-minute fees. Long system prompts and full conversation history sent on every turn inflate LLM spend. Slow tool calls lengthen calls. Poor containment, where the agent hands off most calls to people anyway, means you pay for both.
Build In-House Or Hire A Partner
A competent engineering team can get a ConversationRelay demo working in days. The work that takes months is production hardening: latency tuning, interruption edge cases, prompt testing against real call recordings, CRM integration, failover, analytics and compliance review.
Build in-house if voice is core to your product, you have engineers comfortable with real-time systems and you want to own the stack long term.
Hire a partner if you need a production agent inside a quarter, have to integrate with a contact center or legacy CRM, or work in a regulated industry where mistakes are expensive.
Mix both by having a partner build the first version and train your team to run it.
Does Twilio Provide The LLM For ConversationRelay?
No. ConversationRelay is bring-your-own-LLM. Twilio handles speech recognition, speech synthesis and interruptions; your server calls whichever model you choose and returns text.
Can I Use ElevenLabs Voices With Twilio?
Yes. ElevenLabs is the default TTS provider in ConversationRelay. You can also stream audio from the ElevenLabs API through Media Streams, or import a Twilio number into ElevenLabs Agents.
Should I Use ConversationRelay Or Media Streams?
Use ConversationRelay for most agents, since it removes the hardest audio work. Choose Media Streams if you need a speech-to-speech model, a specific STT or TTS vendor Twilio does not support, or full control over turn detection.
Can An AI Voice Agent Make Outbound Sales Calls?
Technically yes, but the FCC treats AI-generated voices as artificial voices under the TCPA, so you generally need prior express consent, and written consent for marketing. Review your consent records with counsel first.
Is A Twilio AI Voice Agent HIPAA Compliant?
It can be. Twilio lists ConversationRelay as HIPAA-eligible when configured properly under a signed BAA, but every other vendor in the pipeline, including your LLM provider, must meet the same bar.
Conclusion
Building an AI voice agent on Twilio in 2026 is mostly a choice about how much of the audio stack you want to own. ConversationRelay with ElevenLabs voices is the practical default; Media Streams is for teams that need full control; ElevenLabs Agents is the fastest way to a pilot. Whichever you pick, the hard parts are the same: low latency, graceful interruptions, useful tools, clean handoffs and consent you can prove.
