How to Build an AI Voice Agent with ElevenLabs (No Backend Required)

Disclosure: this post contains affiliate links. If you sign up for a paid ElevenLabs plan through them we may earn a commission, at no extra cost to you.

ElevenLabs Agents lets you build a voice assistant that listens, answers in a natural voice and takes actions, then put it on your website, a phone number or WhatsApp, without writing a backend. You choose a voice, choose the LLM that does the thinking, write a system prompt, upload what the agent should know, and connect any tools it needs. A simple support or booking agent takes an afternoon.

This guide walks through building one step by step, using a small business support agent as the example. You'll need an ElevenLabs account; the free plan↗ (opens in a new tab) includes 15 minutes of agent conversations to experiment with. For an overview of the whole platform, see our ElevenLabs review.

What a voice agent can do

Good first projects:

• Website support: answer product, pricing and policy questions in a widget on your site.
• After-hours phone line: take calls, answer FAQs and capture messages.
• Appointment booking: check availability and book slots in your calendar.
• Lead qualification: ask a few questions and pass qualified leads to your CRM.
• In-app voice assistant: let users talk to your app instead of clicking through menus.

ElevenLabs handles the hard real-time parts: speech recognition, turn-taking (knowing when the user has finished speaking), interruptions, and low-latency speech. The new Eleven v4 Turbo voice model gets first speech out in roughly 150ms, which is what makes conversations feel natural rather than walkie-talkie.

How it works

Every turn of the conversation goes through the same loop:

1. Listen: speech to text converts what the caller says.
2. Think: an LLM of your choice decides what to say or which tool to call, guided by your system prompt and knowledge base.
3. Act (optional): the agent calls a tool, like checking a calendar via a webhook or an MCP server.
4. Speak: text to speech replies in the voice you chose.

The platform is LLM-agnostic: use a hosted model, call a frontier model like Claude or GPT, or bring your own.

Step 1: Create the agent

In ElevenLabs, open Agents and create a new agent. Start from a blank agent or a template (support, booking and similar). Give it:

• A first message, what it says when the conversation starts: "Hi, thanks for calling Riverside Bikes. I can help with repairs, rentals and opening hours. What can I do for you?"
• A language, and optionally additional languages. Agents support 70+ languages and can switch mid-conversation.

Step 2: Choose a voice and model

Pick a voice that fits your brand and the job: calm and clear for support, warmer and more energetic for sales. Test it with real sentences your agent will say.

For the voice model, choose a low-latency option for live conversation (v4 Turbo or Flash). The expressive, slower models are better for pre-recorded narration than for real-time calls.

Step 3: Write the system prompt

The system prompt is where most of the quality comes from. Keep it structured and specific:

You are Sam, the voice assistant for Riverside Bikes, a bike shop in Portland.

Personality: friendly, concise, practical. Speak in short sentences.
This is a phone call: never use lists, markdown or URLs. Spell out numbers naturally.

You can help with:
- Opening hours, location, parking
- Repair pricing and turnaround (use the knowledge base)
- Booking a repair slot (use the check_availability and book_repair tools)

Rules:
- If you don't know, say so and offer to take a message. Never guess prices.
- Never take payment details over the phone.
- If the caller is upset or asks for a person, use the transfer_to_staff tool.
- Confirm the name, phone number and time back to the caller before booking.

Voice-specific tips:

• Tell it this is spoken. Otherwise LLMs tend to read out bullet points and links.
• Keep answers short. Long monologues are much worse on a call than in a chat window.
• Define what's out of scope and what to do instead.
• Use guardrails for anything sensitive: money, medical, legal, personal data.

Our prompting guide covers the same principles for coding agents; they transfer well.

Step 4: Add a knowledge base

Upload the documents your agent should answer from: FAQs, price lists, policies, product docs, or URLs from your site. Keep them current and specific. An agent can only be as accurate as its sources, and an outdated price list will get repeated confidently to customers.

Tip: write a dedicated "agent FAQ" document with short, direct answers in the phrasing customers actually use. It usually works better than pointing the agent at long marketing pages.

Step 5: Connect tools

Tools let the agent do things, not just talk. ElevenLabs Agents can call:

• Webhooks and APIs, for example POST /api/bookings on your own app.
• MCP servers, so the agent can use any Model Context Protocol integration.
• Built-in integrations with tools like Zapier, HubSpot, Salesforce, Shopify, Stripe, Zendesk and telephony providers such as Twilio.

For each tool, write a clear name and description ("check_availability: returns open repair slots for a given date") and define its parameters. The LLM decides when to call it based on that description.

If you vibe coded your own app, ask your agent to build a small, authenticated API endpoint for the voice agent to call, and run it through the security checklist: validate every input, require a secret, and only expose the actions the agent needs.

Step 6: Test before real customers call

ElevenLabs includes testing tools: you can talk to the agent in the browser, run simulated conversations, and set up automated tests that check the agent behaves correctly. Test the awkward cases, not just the happy path:

• A caller who mumbles or interrupts.
• A question the knowledge base doesn't cover.
• Someone trying to get the agent to ignore its instructions.
• An angry caller who wants a human.
• A booking with a date that's already taken.

After launch, review transcripts and analytics (success rate, languages used, where calls fail) every week and update the prompt and knowledge base.

Step 7: Deploy

• Website: embed the widget with one line of HTML. In a Next.js app, ask your coding agent to add it to the layout so it appears on every page.
• Phone: connect a phone number through the native telephony integration or providers like Twilio, so the agent answers calls.
• Messaging: deploy to WhatsApp, SMS, email or chat.
• Inside your app: use the ElevenLabs SDKs to start conversations from your own UI.

Tell people they're talking to an AI. It's the honest thing to do, often a legal requirement, and callers are more forgiving when they know.

What does it cost?

Agent conversations draw on your ElevenLabs plan. The free plan includes 15 minutes to experiment; paid plans start at $6 a month, and higher tiers include more usage and lower-latency, cheaper speech at volume (Business advertises low-latency TTS from about 5¢ a minute). The LLM you choose may add its own cost. Check the current agent pricing on ElevenLabs↗ (opens in a new tab) before you route real call volume to it, and set usage alerts.

Common mistakes

• Prompts written for chat. Lists, links and long answers sound terrible spoken aloud.
• No escape hatch. Always give callers a way to reach a human or leave a message.
• Stale knowledge. Update the knowledge base when prices or policies change.
• Unsecured tools. A webhook that books, refunds or deletes needs authentication and validation.
• Skipping testing. Ten minutes of trying to break your agent saves embarrassing calls later.

Frequently asked questions

Do I need to code to build an ElevenLabs voice agent?

No. You can build, test and deploy an agent from the ElevenLabs dashboard and embed it with one line of HTML. Code is only needed for custom tools or deep app integration.

Which LLM does ElevenLabs Agents use?

You choose. The platform is LLM-agnostic: use a hosted model, call a frontier model through its API, or bring your own.

Can an ElevenLabs agent answer phone calls?

Yes. Agents can be connected to phone numbers through native telephony integrations and providers like Twilio, and can also run on websites, WhatsApp, SMS and more.

How many languages do ElevenLabs agents support?

70+ languages, with the ability to switch language mid-conversation.

Is there a free trial?

The free plan includes 15 minutes of agent conversations, enough to build and test a prototype.

Build your first agent today

Start small: one agent, one job, a short knowledge base and a clear way to reach a human. Create a free ElevenLabs account↗ (opens in a new tab), build the agent in an afternoon, and if you'd rather add simple speech to your app instead, see our guide to the ElevenLabs text-to-speech API.