How I Built an AI Voice Agent That Makes Sales Calls Autonomously

The short version: Most AI demos stop at "look, it responds to text." I needed an AI that picks up the phone, has a real conversation, qualifies leads, and updates my CRM — without me touching anything. Here is how I built it with ElevenLabs, Twilio, and Node.js, and the specific gotchas that cost me hours so they don't cost you any.

What we're building

An AI voice agent that:

• Makes outbound sales calls to business prospects
• Handles natural conversation with objection handling
• Qualifies leads based on conversation signals
• Updates a CRM database after each call
• Sends a summary to Telegram for monitoring

This is the same architecture behind Maya, the voice agent that powers our AI receptionist service. Built it for our own outreach first, then productized.

The architecture

Twilio (phone)  →  ElevenLabs (voice AI)  →  Webhook (Node.js)  →  CRM (SQLite)  →  Telegram (notification)

Twilio handles telephony. ElevenLabs handles the actual voice conversation — speech-to-text, LLM reasoning, and text-to-speech in real time. Our Node.js server processes the post-call webhook and updates the database.

Step 1: Set up the voice agent on ElevenLabs

ElevenLabs provides a Conversational AI API that handles the entire voice pipeline. You configure an agent with a system prompt, knowledge base, and voice settings.

// update-agent.js
import 'dotenv/config';

const AGENT_ID = process.env.ELEVENLABS_AGENT_ID;
const API_KEY = process.env.ELEVENLABS_API_KEY;

const agentConfig = {
  conversation_config: {
    agent: {
      prompt: {
        prompt: `You are Maya, a professional business development representative for QuickComet,
an AI solutions agency. You're calling to introduce our services.

RULES:
- Be warm, professional, and concise
- Ask about their current website and digital presence
- Listen for pain points (outdated website, no online ordering, etc.)
- If interested, offer to schedule a follow-up call
- If not interested, thank them and end graciously
- Never be pushy or aggressive
- Disclose that you're an AI assistant when asked`,
      },
      first_message: "Hi, this is Maya from QuickComet. Do you have a quick moment?",
      language: "en",
    },
    tts: {
      voice_id: "your-voice-id-here",
      model_id: "eleven_turbo_v2_5",
      output_format: "ulaw_8000", // Required for Twilio
    },
    stt: {
      model_id: "nova-2",
      input_format: "ulaw_8000", // Must match TTS format for Twilio
    },
    turn: {
      turn_timeout: 7,
      silence_end_call_timeout: 15,
    },
  },
  platform_settings: {
    max_duration_seconds: 300,
  },
};

async function updateAgent() {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/convai/agents/${AGENT_ID}`,
    {
      method: 'PATCH',
      headers: {
        'xi-api-key': API_KEY,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(agentConfig),
    }
  );

  const result = await response.json();
  console.log('Agent updated:', result.agent_id);
}

updateAgent();

Critical detail: Both TTS and STT must use ulaw_8000 format for Twilio compatibility. If these don't match, you'll get garbled audio or silence. I lost hours debugging this before catching it — save yourself the same.

Step 2: Connect Twilio to ElevenLabs

When Twilio receives or makes a call, it needs to connect the audio stream to ElevenLabs. I use a Twilio Function (serverless) for this:

// twilio-function/call-webhook.js
exports.handler = async function(context, event, callback) {
  const AGENT_ID = context.ELEVENLABS_AGENT_ID;

  // After the call ends, ElevenLabs sends conversation data here
  if (event.conversation_id) {
    const summary = {
      conversationId: event.conversation_id,
      duration: event.duration,
      transcript: event.transcript,
      outcome: event.metadata?.outcome || 'unknown',
    };

    // Send to Telegram for monitoring
    await fetch(`https://api.telegram.org/bot${context.TELEGRAM_TOKEN}/sendMessage`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        chat_id: context.OWNER_CHAT_ID,
        text: `Call completed:\n${summary.transcript?.substring(0, 500)}`,
      }),
    });

    return callback(null, { success: true });
  }

  // Initial call setup — return TwiML to connect to ElevenLabs
  const twiml = new Twilio.twiml.VoiceResponse();
  const connect = twiml.connect();
  connect.conversationRelay({
    url: `wss://api.elevenlabs.io/v1/convai/conversation?agent_id=${AGENT_ID}`,
  });

  return callback(null, twiml);
};

Step 3: Build the call manager

This is the orchestrator — it picks a prospect from the database, initiates the call through Twilio, and handles the result:

// call-manager.js
import twilio from 'twilio';
import Database from 'better-sqlite3';

const client = twilio(process.env.TWILIO_SID, process.env.TWILIO_AUTH);
const db = new Database('quickcomet-crm.db');

export async function makeCall(prospectId) {
  const prospect = db.prepare(
    'SELECT * FROM prospects WHERE id = ?'
  ).get(prospectId);

  if (!prospect?.phone) {
    throw new Error(`Prospect ${prospectId} has no phone number`);
  }

  db.prepare(
    'UPDATE prospects SET status = ? WHERE id = ?'
  ).run('calling', prospectId);

  const call = await client.calls.create({
    to: prospect.phone,
    from: process.env.TWILIO_PHONE,
    url: process.env.TWILIO_WEBHOOK_URL,
    statusCallback: process.env.TWILIO_STATUS_URL,
    statusCallbackEvent: ['completed', 'no-answer', 'busy', 'failed'],
    machineDetection: 'Enable',
    timeout: 30,
  });

  db.prepare(`
    INSERT INTO call_logs (prospect_id, call_sid, direction, status)
    VALUES (?, ?, 'outbound', 'initiated')
  `).run(prospectId, call.sid);

  return call.sid;
}

Step 4: Automated call queue

The call queue runs daily, picks the top prospects by quality score, and calls them during business hours:

// call-queue.js (simplified)
const CONFIG = {
  maxCallsPerDay: 5,
  minQualityScore: 9,
  callDelayMs: 60_000, // 1 minute between calls
  preferredHours: { start: 9, end: 17 },
};

function getCallableProspects(limit) {
  return db.prepare(`
    SELECT id, business_name, phone, quality_score
    FROM prospects
    WHERE quality_score >= ?
      AND phone IS NOT NULL
      AND status NOT IN ('declined', 'replied')
    ORDER BY quality_score DESC
    LIMIT ?
  `).all(CONFIG.minQualityScore, limit);
}

async function runQueue() {
  const hour = new Date().getHours();
  if (hour < CONFIG.preferredHours.start || hour >= CONFIG.preferredHours.end) {
    console.log('Outside business hours. Skipping.');
    return;
  }

  const prospects = getCallableProspects(CONFIG.maxCallsPerDay);

  for (const prospect of prospects) {
    console.log(`Calling: ${prospect.business_name} (${prospect.phone})`);
    await makeCall(prospect.id);
    await new Promise(r => setTimeout(r, CONFIG.callDelayMs));
  }
}

Results and lessons learned

After 270+ simulated calls and ~30 real calls across 78 industries:

• Voice quality matters more than prompt quality. A natural-sounding voice with proper pacing converts better than a perfect script with a robotic voice. Spend the budget on a good voice clone before optimizing the prompt.
• ulaw_8000 for BOTH TTS and STT is non-negotiable with Twilio. I lost hours debugging garbled audio before catching this.
• Business type determines best call time. Restaurants: 2–4 PM (between rushes). Lawyers: 9–11 AM. Default: 10 AM – 5 PM. Hardcoding this into the call queue improved pickup rates ~30%.
• Silence detection kills calls. Set silence_end_call_timeout to at least 15 seconds — some people take time to think before responding. Anything shorter and you cut off thinkers.
• Always disclose AI. It is the right thing to do, many jurisdictions require it, and it does not hurt conversion. It actually disarms callers.

What I'd do differently

1. Start with inbound calls first. It is easier to handle people calling you (they are already interested) than cold outbound. We did outbound first because that was the use case driving us, but in retrospect inbound would have caught half the bugs at lower stakes.
2. Build the CRM integration from day one. Don't bolt it on later — every call should log to a database automatically. Otherwise you have great conversations and no idea what happened in any of them.
3. Limit prompt length to under 10,000 characters. Longer prompts add latency, and in voice conversations, latency kills the illusion of natural conversation. Move detail into a knowledge base (RAG) where the model retrieves only what is relevant per turn.

Related reading

If you are deciding whether to use long-context prompts vs. RAG for your voice agent's knowledge: Why I'm Still Building RAG in 2026. If your retrieval pipeline is failing in production: Why RAG Fails in Production.

Frequently Asked Questions

Bottom line

The hard parts of an AI voice agent aren't the LLM or the voice — those are commodity now. The hard parts are: matching audio formats end-to-end, handling silence, building the CRM glue, and disclosing AI properly. Get those right and the rest is a weekend.

Building a voice agent for your business and stuck on one of these gotchas? Email hello@quickcomet.com — happy to compare notes.