Voice Agents

In this part, we will look at voice agents in simple words - what they are, how they work, and how you can build one yourself.

What are Voice Agents?

Voice agents are AI programs that let you talk instead of typing. You speak to the agent, it understands what you said, thinks about the best answer, and then speaks the answer back to you. This makes the conversation feel natural, just like talking to another person.

There are two common ways to build a voice agent.

Speech-to-Speech (STS)

In this method, your voice goes directly to an AI model. The model understands what you said and replies with another voice, right away. You don’t need to handle any separate “speech to text” or “text to speech” steps yourself - the model does it all.

This method is usually faster and feels more natural in conversation. But it also needs more powerful models and more computing power to run.

flowchart LR A[🎤 Your Voice] --> B[🧠 Speech-to-Speech AI] B --> C[🔊 AI Voice Response]

Chained Conversation

In this method, the work is split into three simple steps. First, your voice is turned into text. Then an LLM (a language model) reads that text, understands what you want, and writes a reply. Finally, that reply is turned back into speech so you can hear it.

This is the most common way people build voice agents, because each step does just one job. That makes it easier to build, test, and fix problems.

flowchart LR A[🎤 Your Voice] --> B[📝 Speech to Text] subgraph Voice Agent B --> C[🧠 LLM] C --> D[🔊 Text to Speech] end D --> E[🔈 AI Voice Response]

Why do we use Chained Conversation?

For most voice agents, we use the Chained Conversation method because it is simple, flexible, and easy to build. Instead of asking one model to handle everything, we split the work into three separate parts. This also makes it easier to give the LLM memory of past messages, so the conversation feels more natural.

Here’s how it works step by step. First, a Speech-to-Text (STT) model listens to your voice and turns it into text. That text is then sent to the LLM, which understands your request and creates a reply. Finally, the reply goes to a Text-to-Speech (TTS) model, which turns the text back into speech so you can hear the answer.

Since each part does only one job, you can improve or swap out any single part without changing the whole system. This is why most voice agents today are built this way.

flowchart LR A["🎤 User Speaks"] --> B["📝 Speech-to-Text (STT)"] B --> C["🧠 LLM"] C --> D["🔊 Text-to-Speech (TTS)"] D --> E["🔈 AI Voice Response"]

Creating a Voice Agent

First, install the speech recognition package. This will help us turn speech into text. Pypi page: SpeechRecognition

uv add SpeechRecognition "SpeechRecognition[audio]" openai python-dotenv "openai[voice_helpers]"

You may also need to install some extra tools depending on your operating system. Check the SpeechRecognition documentation for details.

Now let’s turn speech into text using this code:

import speech_recognition as sr

def main():

    r = sr.Recognizer()

    with sr.Microphone() as source:
        r.adjust_for_ambient_noise(source)
        r.pause_threshold = 2

        print("Listening...")
        audio = r.listen(source)

        try:
            text = r.recognize_google(audio)
            print(f"You said: {text}")
        except sr.UnknownValueError:
            print("Sorry, I could not understand the audio.")
        except sr.RequestError as e:
            print(f"Could not request results; {e}")

Next, let’s add the LLM part, which reads the text and creates a reply. We’ll use OpenAI’s GPT model for this. Make sure the OpenAI SDK is installed and set up first.

import speech_recognition as sr
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()

def main():

    r = sr.Recognizer()

    with sr.Microphone() as source:
        r.adjust_for_ambient_noise(source)
        r.pause_threshold = 2

        print("Listening...")
        audio = r.listen(source)

        SYSTEM_PROMPT = """
        You are a voice agent that can understand and respond to user queries. You will receive text input from the user, and you should generate a relevant and helpful response. Keep your responses concise and clear. Act like a friendly assistant that is knowledgeable and helpful. We will provide you text input from the user and you will respond with text output. But while outputting your response make sure it look lake a voice agent response not chat response. You can also ask for clarification if the user's input is unclear.
        """

        try:
            text = r.recognize_google(audio)
            print(f"You said: {text}")
        except sr.UnknownValueError:
            print("Sorry, I could not understand the audio.")
        except sr.RequestError as e:
            print(f"Could not request results; {e}")

        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": text}
            ]
        )

        print(f"AI Response: {response.choices[0].message.content}")

Now let’s add the Text-to-Speech (TTS) part, which turns the AI’s text reply back into speech. We’ll use OpenAI’s TTS model for this. Docs: OpenAI TTS. OpenAI also has a website called OpenAI FM where you can try out the TTS model.

You can use a different TTS model if you prefer.

import speech_recognition as sr
from dotenv import load_dotenv

import asyncio
from openai import AsyncOpenAI
from openai.helpers import LocalAudioPlayer

load_dotenv()

async_client = AsyncOpenAI()

async def main():

    r = sr.Recognizer()

    with sr.Microphone() as source:
        r.adjust_for_ambient_noise(source)
        r.pause_threshold = 2

        print("Listening...")
        audio = r.listen(source)

        SYSTEM_PROMPT = """
        You are a voice agent that can understand and respond to user queries. You will receive text input from the user, and you should generate a relevant and helpful response. Keep your responses concise and clear. Act like a friendly assistant that is knowledgeable and helpful. We will provide you text input from the user and you will respond with text output. But while outputting your response make sure it look lake a voice agent response not chat response. You can also ask for clarification if the user's input is unclear.
        """

        try:
            text = r.recognize_google(audio)
            print(f"You said: {text}")
        except sr.UnknownValueError:
            print("Sorry, I could not understand the audio.")
        except sr.RequestError as e:
            print(f"Could not request results; {e}")

        chat_response = await async_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": text}
            ]
        )

        ai_text = chat_response.choices[0].message.content
        print(f"AI Response: {ai_text}")

        async with async_client.audio.speech.with_streaming_response.create(
            model="gpt-4o-mini-tts",
            voice="coral",
            input=ai_text,
            instructions="Speak in a cheerful and positive tone.",
            response_format="pcm",
        ) as tts_response:
            await LocalAudioPlayer().play(tts_response)

if __name__ == "__main__":
    asyncio.run(main())