LiveKit: The Open-Source Platform Behind OpenAI's Voice Mode

From LiveKit, the open-source platform behind OpenAI's voice mode, powering voice agents for everyone from startups to 911 operators.

I'll be talking to one of our users about some problem. And I'm like, how many users is this impacting? And they're like, oh, it's impacting like 1% of our calls. So that's like 100,000 minutes yesterday. I'm like, how are you doing that much traffic? So we just raised our Series C and we are a unicorn as of just recently.

My daughter when she talks with ChatGPT using the voice most time she's thinking and then ChatGPT just jumped in and then they started arguing with each other.

The biggest problem in voice agents since LLM based voice agents have come to the scene has been turn detection. The conversation breaks down and the user gets frustrated and the call doesn't go to its objective when the agent and humans start talking over each other.

I would take Whisper for speech to text, pipe the transcript to Claude, send response to ElevenLabs, stream the audio back over a WebSocket, put some voice activity detectors on it. What do you think of my approach?

That is a great first pass. If you build that approach, you'll get something working, but it will be a little bit brittle and you'll likely have really high latency. People are surprising creatures. You would think that like it wouldn't make a difference if a person was talking to a voice agent or like a fake human, but they talk a lot more when they're talking to an avatar. Even when the avatar doesn't look totally real, they open up more compared to a voice agent. It's just like a wave bouncing on the screen. Every decision is emotional and emotion is embodied. It's how we feel about things. It's not an intellectual thing. It's a feeling. And I don't think the way we're building intelligence machines now approximate judgment in any way. There is no feeling. I hope you go build something weird. It's exactly kind of how I think about it that I really felt like deeply that it needed to exist.

Welcome back to TwoSetAI. Today I'm sitting down with Tom Shapland from LiveKit, the open-source platform behind OpenAI's voice mode, and they're powering voice agents for everyone from startups to 911 operators. Tom, welcome.

It's great to be here. Thank you for having me on the podcast.

For people who don't know what LiveKit does, can you explain what it does maybe towards two different audience? One is let's go with an elementary schooler and the other maybe a developer who's already building AI agents and you know yeah if they need you

I love that let's start with the elementary school kid I have two kids in elementary school so sometimes they ask me what I do at work and so I have an answer for that okay my answer for them is I ask them have you ever talked to an AI like I ask them do you remember the time we've opened up my phone and we talked to ChatGPT and they're like, "Oh, yeah, yeah, I remember that." And I tell them, "We power those experiences. We make it so that people can build voice agents that you talk to. We make it so you can talk to computers." So, that's my answer for an elementary school kid. How do I do on that one? I think I could probably do a little bit better on that. I think I could tell like we help build I think what I normally say to my kids is we help build ways for people to talk to computers like talking to ChatGPT.

How old are your kids?

They are eight and 10.

Okay. So they know they will understand you. What about developers? Your real customers.

LiveKit is a platform for building voice and video agents. You can think of LiveKit consisting of two main parts. The first part is the WebRTC audio and video transport layer. So this allows us to move audio and video around the internet very quickly. And that's something that you need if you're enabling if you're building apps where people are talking to computers. And then the second part of LiveKit is our framework for building voice agents. So we have an open-source framework where you use our Python code, our TypeScript code to build agents and then those we take care of all the like abstractions around building a voice agent like handling turn detection and making sure that like the audio is reaching the agent and then then the audio is translated into text and then that text is passed through an LLM and the output of the LLM is passed back to a text-to-speech engine and then that audio is played back to the user. So it's really those are like at a high level those are the two main components of LiveKit.

Okay, let's say I'll give you the use case. So let's say the use case is sales training, right? It's AI role play as the prospect while I as a user pitch and then the agent should take notes during the call and then switches to coach mode at the end to critique my performance. I would take Whisper for speech to text and pipe the transcript to Claude with my you know system prompt and then send Claude's response to ElevenLabs for text to speech and then stream the audio back over a WebSocket and you know put some voice activity detector.

That is a great first pass at it and if you build that approach you'll get something working but it will be a little bit brittle and it you'll likely have really high latency. You had some of the core components there that I wanted to address which are like the voice models that you need in your stack. So the you brought up each of these and I want to call them out specifically. If a user is talking to a computer you need to transcribe their audio with a speech-to-text model. You need the next model on the stack is one you brought up here which is something to know when the user is done speaking so that the LLM or the so that the voice agent knows to process what the user has said and turn it back into audio and send it back to the user.

So that's where you brought up the voice activity detection model and there's a whole like suite of paralinguistic models that we build at LiveKit that really help with this turn detection problem. So that's the second model in the stack. The next model in the stack is the LLM and no explanation necessary there. And then there's the text-to-speech engine that turns text to speech model that turns the LLM output into into audio. So those core components are there and you can put those together. Now where you're going to start incorporating problems is the first thing I want to call out is around WebSockets. So we you'll you'll want to use this goes back to like the first component of LiveKit which is we provide WebRTC infrastructure for moving audio around the internet and WebRTC is really built for real-time experiences. So and the way that makes that most concrete for me is that WebRTC is will sacrifice completeness for the sake of keeping the content current.

So it'll drop packets so that rather than making sure to keep the packets all in order so that the experience stays in real time. And so if you're using something like other approaches to it, you're going to you're going to start running into problems where your audio is going to start to lag and that's going to cause latency issues and your experience is going to fall apart. And then the other thing around like orchestrating these models together is like it's sort of like you could build your own authentication for your app for users to authenticate, but that's really not the best use of your time.

You should instead be using a framework that has already thought through the million edge cases and just makes it easy. And that's what LiveKit's agents framework does for you is like, yeah, you can put together this stack, but then like how do you handle the case when like how do you handle an example is like preemptive generation? How do you handle a case where you want the LLM to start generating output before the user has finished speaking because the user because the LLM generation is the slowest step. We've already like built that.

What is that? Why do I need that?

So like so much of what people when you're first building a voice agent the first thing that you're trying to the first thing you notice is latency and the first thing you're trying to make you're you think like oh how can I make this better is around latency. If you have a if the time between when the human stops speaking and the agent starts speaking is more than about a second and a half people start feeling excuse me people start feeling like users start feeling like it's taking too long to respond.

And so one of the ways, one of the tricks to like try to reduce the latency is you every time the user speech comes in from the speech to text engine, you start generating having the LLM generate a response, even before the user is finished speaking. So every time you get like a chunk of transcript from the user, you start generating a response with the LLM. And that way when the user and then you'll have to like the user will keep speaking, you'll have to cancel the previous LLM generation and start a new generation as you get more speech coming in. But what this means is like when the user does finally finish speaking, the LLM has already started creating the response and you've shaved off some of the time it takes for the voice agent to respond because you don't have to do everything serially. You're kind of doing things in parallel.

Wow. If you think about this is maybe how our human brains work as well because as you were thinking as you're speaking I'm already thinking and I'm thinking maybe I should ask this question as the next one. So it's already happening. So you're doing predictions basically or you're answering based on the context that's coming in that

That actually falls more into the category of that second group of models we're talking around turn detection. So not about like they were all related but the way human the it's an interesting topic in linguistics where people are it's an open topic but how do how is it that humans are able to respond so quickly in conversations and the like prevailing hypothesis is that we're modeling out and predicting what the other person is going to say before they finish speaking and already starting to generate a response while the other person is starting to speak. And you can like see that in your head. It doesn't seem it's not all that like so many good scientific theories when someone says it you're like oh yeah it seems pretty obvious that is what I'm doing when you speak.

Can I use speech to speech models like the GPT-4o Realtime that can accelerate that as well right because I'm eliminating the step of calling the LLM and then you know doing a transcription and then all that.

For most production use cases where people have built voice agents that are actually generating revenue for the business, their business, they don't use speech-to-speech models. Speech to and the reason is that speech-to-speech models, there's two main reasons. One is that speech-to-speech models aren't very good at tool calling. Text-based LLMs are much better at tool calling. And so if you have some agent that's trying to do something for a user, they'll likely have some sort of tools, some set of tools that it's calling. And the other one is I'm sorry, what was I going to say?

The other one is oh that you have more control. So if you if you're using a speech-to-speech model, you can't you don't have as much control over like how certain words are transcribed. But some of the new things coming out with speech-to-text models is that you can like prompt those models where you can tell them ahead of time this is what the user is talking about and here are some of the words that they will be and here's just general context and this speech to text model takes in both the audio and that prompt every time it makes an inference.

And so you just have so much more control and you can create more accurate voice agents that actually fulfill their goals when in the current state of the world. Maybe one day speech to speech models will be more effective, but it's not they're not currently the main thing that people are using to create a good voice agent experience. There is a lot to it and that's why starting with that's why it just makes sense to start with a framework, ideally an open source framework where you can actually see what the code is doing. It's not a black box and then use that as your starting point for building your voice agent.

Is there any use cases that you know the speech-to-speech models will be a direct solution to it that actually would work or you're thinking the model is just not good enough just yet?

There's a lot of usage of speech to speech models but they're more on the consumer side. So on the B2B side where people are building voice agents for like business use cases then people use the cascaded pipeline approach that we've been talking about this whole time. But a lot of consumer applications such as when you talk to Gemini on your phone or OpenAI on your phone, you're actually using a speech-to-speech model.

Now, what's the difference between the consumer apps and the B2B use cases that you know you're you're saying that the model choices should be different?

Yeah. I mean, we can think about it in terms of like talking to a friend. When you're talking to a friend, you're just kind of chatting with them and want to There's not like some objective that you're trying to get to. Sometimes you'd have an objective you're trying to get to with a friend where you need them to like look up the time you're supposed to meet them or something, but most of the time, you know, you're just chatting with them and it you don't need to access external databases or you don't need as much control over the conversation flow. But in a B2B use case, to give you a more concrete example, you can think of like lead qualification, which is a use case with a lot of growth in the voice agent field. So like a business gets a lead from a website and then they want to call that lead to qualify and make sure that they like have I don't know the they live in the right area code and so you can imagine like there's a whole workflow that you need the voice agent to adhere to. So you need that sort of control over the LLM.

And then also that voice agent at different steps in the workflow might need to be making external tool calls to say, oh, like, okay, this person doesn't have the this qualification. So instead, but they may still actually have this other qualification, but I need to like look up something about that qualification. I don't know, that wasn't the best example off the top of my head, but hopefully you get the idea. I should also just one last follow up on that is that there are some consumer use cases where the voice agents are using the cascaded pipeline and are very sophisticated. So like the example is like our customer Portola builds Tolans which are consumer voice agent where people talk to this character but that character like comes to know you over time and actually has objectives with his conversation with you and so like those are very sophisticated agents they build and to build agents that sophisticated they're not using speech-to-speech models they're using the pipeline approach.

I have a second point about this conversation turn thing. One thing I noticed was that my daughter when she talks with ChatGPT using the voice mode, right, it was actually a terrible experience. No offense, I think because the way kids talk and the way they pause and the way they interrupt are more unpredictable than adults. And he she sometimes she's thinking and then ChatGPT just jumped in, you know, and then they started arguing with each other and my daughter is just saying that you should stop talking.

I would like to point out that the OpenAI real time model and advanced voice call, you're you're not actually using our turn detection models. I think ours are much are very effective and in fact a lot of users end up migrating to LiveKit because they try out ours and they're like, "Oh, the conversation actually ran much more smoothly." The other point I want to make is that the problem you're talking about is definitely not limited to children talking to AI. The biggest problem in voice agents currently and it has been this since this has always been the biggest problem since LLM based voice agents have come to the scene has been turn detection because the conversation breaks down and the user gets frustrated and the call doesn't go to its its objective when the agent and humans start talking over each other.

And like you said, like people have different styles of talking to each other. And part of what you and I do when we talk is like when we first start talking, we're doing some sort of like calibration with each other and know like how long to wait. And then even midway through a conversation, I will like dynamically calibrate and be like, "Oh, this is actually kind of like a more thoughtful topic." And so I should maybe give her more space for thinking of her response. And this is all stuff that you can because LiveKit the our open source agents framework is so flexible, you can actually build that into your agent. So like we aim to have it default out of the box the turn taking to be as good as possible. We recognize that there's different use cases and that you have the flexibility to kind of parameterize the models and build your agent in a way that is going to work for your use case and the different turn-taking characteristics of your use case.

How do I do that as a developer?

The short answer is that you can adjust some of the parameters in your voice agent and the parameters around how the model works. So let me actually spit out like first the short answer which is you change some parameters. You're not doing it with prompting. So what the model is doing is it's taking the transcripts as they come in from when the user is speaking and then it's passing that to a small text-based LLM and asking the LLM how likely is it that the next thing that the person says that what the likelihood that the next token is a complete is a period is a finished statement. Actually that's not right. I said that wrong. How likely is it? We train the model on utterances from people and then one of the inputs in our not one of the inputs but one of the fields in our model training set is when the person has finished speaking. So there's a token that's like the person finished speaking. And so then when we run inference where the machine learning model the LLM is outputting a probability that the next token is that the person is the token for finished speaking.

That means that as the person is speaking the transcript is coming in and if the output of and we're running inference on this turn-taking model and this end of utterance model and if the end of utterance model says the person is likely to have finished speaking then we then tell the agent that it's their turn to speak. But if the output of the model says that there's a low likelihood that the person has finished speaking, then we tell the agent wait a few seconds before you start speaking cuz the person might start speaking again. And that is the parameter, one of the parameters you can tune is like how long you if the if the end of utterance model is said the person may not be done speaking, how long do you wait until they the end of the voice agent starts speaking? You can set it to like a very short time and if the model doesn't think the person has finished speaking, then it's very likely that the agent will start speaking and maybe interrupt the person.

But if you make it a really long time, you can give the person lots of time to finish their thought. That's maybe too much in the hopefully I explained that sufficiently well. It's kind of a complicated system if you're first hearing about this for the first time and I feel like I it's I almost need to diagram it out. Hopefully I explained it over voice well.

My understanding is it seems comes you know a natural solution that it sounds to me it's a machine learning model that you're classifying of is this end of end of speak end of the sentence end of the turn the turn you're doing predictions a classifier so cats and dogs are finished or not finished and in my naive setup I'm assuming I'm going to manage separate API keys for like Whisper, Claude, ElevenLabs and if I were to use LiveKit what does that look like?

Yep. So we're developer centric and we want to it's really important to us that the developer feels that they have control and flexibility that they're not you know the alternative for them is to use like a closed source platform where they really don't have a lot of control. So you can if you want use your own keys that's always an option but we also manage that for you with the product that I work on at LiveKit called LiveKit Inference. So with LiveKit Inference you use your LiveKit key and with that key you can access the leading voice models model providers and the models from those providers. The ultimate goal of LiveKit Inference is to create more consistent and faster latency for your voice model calls. We're building a layer in between the agent and the voice models that can do things like routing to the lowest latency server for a particular model provider. But that's kind of like future looking for now.

The main like the one of the value propositions of LiveKit Inference is it's just easier to get started with building a voice agent because you need just one key and you can easily swap them out without having to like go to another provider and sign up for their service. It becomes especially useful if you're like more towards the higher usage scale, the enterprise scale, and you don't have to like go get a contract with five different model providers and negotiate five different contracts. You can instead just come to us and get one contract.

What's the value of negotiating a contract with let's say OpenAI? Do I get a lower rate?

If you so I didn't make that point clear. Once you reach a certain amount of usage with especially on the on the speech to text and text to speech, your listeners may be more familiar with LLM. So I'll start there. On the LLM side, if once you get to a certain amount of usage, you can go to the pro the providers and say, "I want a discount because they have so much usage." So the sales motion or the experience as a as a customer is slightly different. You basically reach a cap. You get to a certain amount of usage and you're forced to go on an enterprise contract. It's not just the three models in your voice agent stack. You also have backup providers which is always a good idea in any sort of engineered system.

And you're saying if I just I just use LiveKit Inference and then you handle all those contracts like you know dynamic

You just negotiate with us. We are we become your one vendor for all of your voice models. And oh, I'm sorry to interrupt. I got excited about one other point around that, which is also there's no lock in, right? If you go negotiate a like $100,000 spend on one particular TTS, you're like stuck with that TTS provider. If another TTS provider comes out with something really cool, you're like, well, I can try that next year when my contract runs out, but I can't switch to it now because I've already committed the spend to this other provider.

Okay, let's say I'm building this LiveKit agent with I'm going to use Cursor or Claude Code. Is this the right workflow? So what I can do is I can clone the repository and point the agent at the your documents and then let it figure out the rest. Is that a right?

You are you are like so many steps ahead of unfortunately what I see as the what is the most common way people are building with coding agents which is they like they don't I shouldn't say the most common. I guess I bring it up that way, perhaps in a more negative spin because it's something I'm trying to solve and it bothers me that I have I that I haven't fixed it. So the issue is that a lot of people when they're building with any sort of developer tool and not just LiveKit, they'll go to their coding agent and they'll they'll never think to leave that coding agent or to give that coding agent more context. They'll say, "Build me a LiveKit voice agent." And the coding agent will use its training data to figure out how to build that agent.

And that training data may be stale or if there's something that's not it just may overlook something in its training data for whatever reason and so it'll make up a method for how to do something and that it's just very it's hard to build. What that ends up happening is like the coding agent is sort of guiding the user on how to build it, but it's the blind leading the blind. And the right way to do it instead is what you said, which is like if the project you're building with has a open source repo, go clone that repo and put it in your project. And when you open that project in Claude Code or Cursor or whatever, you can tell it like go look in the repo for how to do this and build this part of my agent, you can use the repo to help guide you or we have like an a doc server, an MCP server for our docs and you can also install that server in Claude Code or Cursor or whatever and you can tell the agent go look in the docs for how to do this. The most effective way I find for building agents has been to actually clone the repo.

The open source repo, but like you're just like so far ahead of like what I oftentimes see people of how they're building. I mean, you must see that too with other things you've built, right? Or and talk and how other people talk about what they're building.

Okay. My sales training app is pretty narrow app and just me and one AI doing roleplay. But I'm curious, what could I add with LiveKit?

Oh, for your sales training app. Well, the first thing that comes to mind is that you could put a face to your voice agent with the real-time avatars that are out there. I'm really excited about those. I think they're really neat. So, basically the way it works is you have another model now in your pipeline where you have the speech to text, then you have your like conversation orchestration, you have your LLM, your text to speech, and then after the text to speech, you now have a real-time video generation model of an avatar of a face and shoulders.

And that audio gets sent to that avatar, and then that avatar embodies embodies is the right word, but then says what you know that audio that was passed to it. And I'm really excited about that. I think we're going to see a lot of growth in that over the next year because those models are getting so much better. And also people people are surprising creatures. We just do like it. You would think that like it wouldn't make a difference if a person was talking to a voice agent or like a fake human, but actually they talk a lot more when they're talking to an avatar. Even when the avatar doesn't look totally real, they open up more and they say more compared to talking to a voice agent. That's just like a waveform that's like bouncing on the screen. So I think we're going to end up seeing a lot of growth in that field. And we have some of those in our in LiveKit's plug-in ecosystem. The companies that come to mind are like Hedra and Tavus, Lemon Slice, the HeyGen. Those are all like they basically render a video of a person in real time speaking the words that you give them.

I have a question for you on this one because I just interviewed the CEO of a company called Delphi. So Delphi is a digital twin. He told me that the they tried video chat and video chat doesn't work because people think it's it's kind of creepy. I wonder what's your take why in your case you're seeing working while in his case he's seeing not working.

We live in a nonlinear world now, an exponentially changing world and the models have gotten a lot better. Also, I think for it might be for their use case like there these are particular people where the person talking to them is familiar with their mannerisms like celebrities is my understanding or coaches that have followings. Whereas if you saw my avatar speaking and you weren't familiar with my mannerisms already, you'd be like, "Oh, that must be just how that person talks."

And your avatars are real humans or are they more cartoonish characters?

So, the companies we support that have plugins in the LiveKit agents ecosystem are almost all like photorealistic ones. I could think of a couple where you can make them like cartoonish or characters. I think like the magic of this stuff is the photo realistic person that you're talking to. When you as soon as you make it a character, people look at they're like, "Oh yeah, I've seen that before. I've seen cartoons before." But when you make it photo realistic, I think it's a little more interesting. I don't know. Oh, I don't totally get people, but they seem to talk more even when the agent is when the when it's not when it's uncanny. Yeah.

Do you tell them that, okay, you're talking with an AI agent that look like a real human?

They must know. I mean, I'm not a developer of one of these products, so I don't know. I just I only hear about like from the developers who are building with these about like them talking about what I told you about where like there's this kind of ramp up in how much people interact with a computer. If you give them just like a little chat bubble, they don't talk that much. You give them a voice, a way for them to talk and they interact more with it. And you give them like something for them to put their eyes on where they feel like they're talking to a person, even if it's not real, they talk even more. There's a new company that hasn't launched yet that we're working with and their their agents are really good. I mean, even the top tier ones like Tavus and HeyGen, which are more like the higher price, better models out there, or not higher price, but the more like yeah, they're they're ones that aim to be on that side of the market of the higher price models. They are pretty they're pretty remarkably good these days.

What do you think is going to happen?

So, I don't know much about entertainment the entertainment industry and what people like. So like I it's definitely possible that like in podcast you could just have like the host the host feels like their avatar is well trained on the sorts of questions they ask and their interests and we'll do a good job of representing them and then the person the guest may feel that their avatar is well-versed on their their knowledge and somebody might find that to be an entertaining show. It reminds me of like NotebookLM where people like put in a bunch of content and it gets turned into you know a podcast that and people find that as a useful way to consume that information entertaining way to consume the information they uploaded. So, I don't know about what's going to happen on the in entertainment, but I do think that like humans are we definitely know real-time experiences feel very or in real life experiences feel just so different.

Like I talk to my co-workers every day and I experience like we're talking in right now or that's in a virtual room and I don't feel I mean I feel happy to see them but it's it's not like when I'm get to be around them I'm like oh this is so cool I get to spend some time with them in real person like I feel something very different and I just feel like they're will never be able to replace that experience that people will never replace talking to each other because it just feels different. It's it's what we've evolved for.

Okay. What else can I do? I'm going to put a video avatar now to for my sales agent. Anything else I can do?

I think there are so many ways you can make that agent more effective at coaching the user. So I think a lot around customization and personalization for that particular user where you can really come to learn that user and things like how they speak. So, like you could even change your turn detection settings around how they speak, but also you could develop such a good understanding of where that person is in their journey as a and whatever you're training them on to like develop really good material that gets generated in time in real time for that person where you have like you're I mean very similar to like if you were teaching me how to do something you would see me go off the rails in similar ways and like you could have your you could teach the model.

Oh, that person's going off the rails in this way again and I am very specifically I have a specific way of wanting them to go back on target. So, I think the next horizon is like user customization and there's just so much neat stuff you could do between our agents framework and the capabilities of these models to build that sort of user personalization.

What about having a group training session or what else I can imagine? Let the coach the AI coach critique my body language. Are those possible?

Yeah, I've seen those. So people are I mean you just have another machine learning model in the in the mix, right? A friend of mine built was building a company that was doing that where he was I'm I would have I'm sure failed his his app's coaching test now but it was about like how you show up to the meeting and how confident you seem and like how many how well you speak without disfluencies and that sort of thing that like those are all sorts of neat things that you can build. I mean, you have such a interest in depth of experience in machine learning models, so I could see the direction you're going with this without having to worry about the boilerplate stuff that everyone building a voice agent has to figure out. Like without having to worry about how to make turn detection fast and how to make it a low latency experience. Instead, it's just something where you can like that's our that's our main value proposition, our north star is like take care of all the boilerplate stuff for developers so they can focus on their own custom app.

I get to do the fun stuff.

Yeah. Yeah, I get to do the fun stuff.

Cool. What are the most interesting use cases you've seen people building with LiveKit?

I brought up Portola a moment ago and they just come to mind again because they're they're an interesting team that has built this great product. And part of what's so interesting about that use case is how deep they've gone on personalization. Like how deep they've gone on really understanding their users and trying to understand what their the user is feeling about something and making it so that their their voice agent that they're talking to the Tolan can have a really fulfilling conversation with the user and that's like one of the interesting ones that comes to mind. I think the other interesting one that comes to mind is more from a business perspective of just like how much growth we're seeing in some of these. So like and there's a lot of use of lot of growth in voice agents for healthcare for doing things like calling insurance companies to get prior authorization.

And I mean part of what makes that interesting is I'll be talking to one of our users about some problem they're having and I'm like well how many users is this impacting? And they're like oh it's impacting like 1% of our calls. So that's like 100,000 minutes yesterday. And I'm like what? How are you doing that much traffic? Like how is this stuff, you know, doing so well. And that's part of what makes it so interesting is like I feel like I'm living in the future. It's just not evenly distributed. And that like we're seeing incredible growth that in some of these from some of our companies in some fields, but they haven't taken over those fields yet. Those fields are still in the process of transforming, but they're changing quickly and the growth is fast. Those are two that come to mind.

If you were to start a voice agent company or video agent, what would you build?

My last startup, we were trying to build something in the voice agent space and we just wanted it to be generally useful and it wasn't something that I felt like I had a strong like what we were building wasn't something where I deeply felt like this needs to exist. I just thought like, oh, this generally seems like it'd be useful and there's a value proposition to it. Let's go out and try to sell it to other businesses. And that was and that company failed and I compare that to my first company. My first company was a sensor that I developed in grad school and to me very deeply I felt this thing needs to exist and it was weird and different and like a lot of people told me it wasn't a good idea but I was like so set on this being like my thing and I think that's why that company succeeded and I've been I think that was what would guide me for the next voice agent is I wouldn't want to build something where I'm like oh there's there's clearly a market for that and an ROI for it.

It'd have to be something that is like weird and different and like something that I would feel like this needs to exist because it's just like it speaks to me in some way. And I don't know what that is, but that's my guiding principle around it is that like it would have to be something that just seemed weird. And my coworker Jesse put together like a what am I looking for? Like a starter kit for a hackathon we did with Google and YC. And he ended it saying like, "Okay, there you go. That's the starter kit. I hope you go build something weird." And I just love that he said that cuz that's exactly kind of how I think about it is like I'd want to do something just different that I really felt like deeply about that needed to exist.

Do you feel it's mostly replacing old systems versus like, you know, creating entirely new experiences?

I suspect that I'm probably I don't have a this is like anecdotal as I search through my mind on how to answer that question, but I suspect a lot of what the work so most of voice agents in production now are telephone-based voice agents and they're doing things like the healthcare example I gave or customer support or lead qualification and in those cases they are replacing workflows that humans are doing. But I want to be careful of saying replacing because like one of my customers tells the story about like there the call center that they're working with, they have eight people working there and there's one job they don't like doing and there's another job they like doing a lot more. They like hunting for new business. They don't like supporting doing the customer support side. And so more of the people get to do the hunting for more business now than less of the customer support side because they're augmenting their staff with voice agents.

So I want to be careful to say that we're just out there replacing humans because I don't think that's actually what's happening out there. Anyway, where was where was I? So I think there's some amount that's replacing existing workflows, but I also think there's a lot of creative new and different use cases such as the example I gave with Tolans, Portola. Like before, it never existed that you could have like this companion that you could talk to about your stress around your studying for the nursing exams or whatever. And there's there's other new and creative ones that I mean those are my favorite just dovetailing off of what we were just talking about. The things that people are building where I'm like, "Oh, that's weird. I like it." You know, that's because it's something new that I've never seen before. Those are the ones I like.

The existing ones like the first thing that comes to my mind is the IVR systems, the ones that you a long menu that you have to calling the insurance company. It's still everywhere, right? Do you think in the future we can skip that whole thing and just talk with an AI agent with an natural conversation?

Yeah, I think that future is here. It's just unevenly distributed. So, I think I love when I encounter voice agents in the wild as a consumer and get to actually see them. And I think it's happened to me like just once or twice, not all that often, but I think we're going to see it more. What's that?

I have never run into one. Which one did you talk to?

I think it was Viori that had one. I had to call customer support, their clothing clothing brand, but I'm not sure. This was before the holidays and so I'm hopefully it's still up. I don't think I wish I came up with an example off the top of my head that I've encountered as a consumer that's one of our customers, but it hasn't.

I would say hallucination is still a problem, right, for more highstake companies to completely adopt it like the insurance company, healthcare.

Yeah, I think hallucinations are still a risk, but we're seeing so much growth in these regulated industries. I think because the way you we're getting better at contain or preventing hallucinations and part of the way you do that is you have shorter prompts with less that the agent can lose and more you can you more tightly control that agent and there's lots of like the models are getting better at this and the frameworks like ours are getting better at this where in our framework you can have lots of different sub-agents that are more focused. I suppose the point I'm trying to make is that the AI industry is getting better at preventing hallucinations by both improving the foundation foundational models as well as the frameworks that people are using to build their apps are building better controls into giving developers more tooling to control the agent.

And to use the LiveKit agent example in particular, if I'm going to have a 30 minute call with a onboarding call with a new user for something that's higher stakes, I'm not going to use one giant prompt because that's going to likely lead to the agent hallucinating something because it loses some of that when you give it really big prompts. Instead, LiveKit gives you the ability to break your agent into lots of small agents that get passed from one where the user gets passed seamlessly unwittingly from one agent to the next so that they each agent is more focused and less likely to hallucinate because it only has like these three things it's allowed to do and there's lots of guardrails written around that one little agent around what it's allowed to do.

How do developers actually find you? Like what does a typical customer journey look like?

A lot of word of mouth. That's like our main growth driver is word of mouth, which is really great to see. And then they also find us through like our channels where we post cool demos. So we're on YouTube and X and LinkedIn where we're posting cool demos and they can also come to our website when they're searching voice agents. They'll find our website, livekit.io.

Cool. What about other comparing with other providers like Retell? I think they're more a GUI kind of user interface style, right? You kind of you put in your prompts, click around, and then set up two calls. LiveKit is completely coding. What do you think is the difference and why?

Yeah, I think that's a good way of that's how I generally describe the voice agent landscape to people who are new to the industry like there you can create two different groups. There's the closed source voice agent platforms and there's the open source voice agent platforms. The open source ones are LiveKit and Pipecat. And then the closed source ones are Retell, Vapi, others. And generally that's like the first important distinction when you're making a decision about which one to build with is which one like shows you what's actually which one gives you all the code so that you can see what's happening and gives you I mean I framed that in such a way that wasn't very objective, but like which I would want more control. I would want to know which what exactly is my agent doing and how can I change that agent.

And you've raised close to 100 million, right? 80 80 some million dollars and you are still open source and anyone can fork your repository. What you what would you say is your moat?

So we just raised our Series C and we are a unicorn as of just recently. Yeah. So that's thank thank you. Very exciting. I think our moat is that we do the full stack. So we have the WebRTC layer and we're really good at WebRTC infrastructure. And it turns out that's really important like how fast the you can move audio around the internet in a very reliable way is really important. And then we also have all this tooling for people to build their agents. And that's mostly what we've been talking about today, the open source framework for how to build agents. But there's another layer above that which is the running your agent. So people can we host people's agents on the cloud for them. And it's really fast to host your agent on us. It's convenient for the developer but also really fast. And that's one of the ways we make money is your agent is fast if you're hosting on us. And we charge money for doing that just like any sort of cloud would.

And so those are like and then we have the observe part which is the other part of our platform where we show people what's what their agent did and how to test their agent. So I think it's owning that. The reason I'm bullish about LiveKit and our moat is like our first moat is just quality. I should have started there. Like all these companies are migrating to their voice agent workload to us and enterprises are picking us because the quality is there. That's the best platform for building a voice agent. And then the second one is that we all own the we own the entire stack.

What if OpenAI or Google if they decide to just fork your repository and build native voice agent infrastructure into their models, right? And the speech to speech gets good enough and latency, you know, gets better. Do you think you still have that moat?

Yeah, we understand the problem more deeply and we already have the community's trust for it and I don't know how do I we work with those companies so I want to like acknowledge what great products I think they have but I think that a lot there's a lot of value that comes from focusing on one company focusing on one particular thing and really doing a great job of it.

AI is changing the gap between senior and junior. Some some some people said that it's widening and some people might be saying that it's actually shrinking. What are you seeing at LiveKit?

Is that gap shrinking here?

Senior and junior engineers. They used to have years of gap.

Yeah, I see. So, the first thing I see is that the toothpaste isn't going back in the tube. Like all the engineers I know are using coding agents and that's not going to change. We're not we're not going to revert like to people doing everything by hand. And I think that I don't know my first reaction to that is to think of my own coding projects where like if I would I don't under let me say it this way I don't understand how people are running like six Claude Code sessions simultaneously from their phone in remote control mode and building something useful at the end of the day that actually is actually works as you expect because I have to babysit my coding agent still like I have an idea I come in with an idea of the architecture that I want from my experiences of building things.

I know what is good architecture and I can keep the agent on the rails and I feel like that's something that requires human judgment and that's not going to go away. And so and that judgment often comes from training and experience and I think you can have those things as a junior engineer. Maybe you don't have as much experience but you can have the training and kind of the intuition and I think you have that as senior engineers too. I think we're just kind of going through an interesting stage right now in software development where like you go on Hacker News and like 10% of the posts are fear, uncertainty, and doubt around coding agents. People panicking about it. And I think we're going to get through this and realize that like engineering is always going to exist. We just have some better tooling now.

Different roles to play.

Yeah. Different Yeah. Different roles that but these these things are still tools and you still need engineers to know how to build things.

Let's say if one engineer with AI can do what five used to be able to do, how does that change how you think about team size and career growth at your company?

I should be I'm not on the hiring side of the company. So I actually don't know what our take is on these things, but I do know that just from looking around the company that we hire people from at various experience levels. I just I think that I think that the these tools can help us move faster, but they still need people to work the tool. Like the tool doesn't make the human go away. Especially in this case like there was the I think before going on earlier this morning I was on Hacker News and I saw a post about like Amazon having an all hands meeting about people needing to about new policies for using coding agents because they keep having outages from people shipping code that coding agents wrote and that the thing that needs to be I think what the policy they were talking about was that they're that like every every code has to be every PR has to be reviewed viewed by a human sounds sane enough, but like you know, it's like I still think that like there's going to be a place for humans designing these systems and making sure that they're they're built right.

I just think it's it this is just the latest in tooling around it. But you also need someone to design it, right? Like you can't someone still has to write the spec and that person who's writing the spec has to understand how the code works.

If there's one thing you can teach me or teach anyone who's trying to build a company, what would it be?

Yeah, I think the weird thing I was saying earlier is definitely true that cuz I that seems to be a throughline with a lot of the conversations I have with other founders that are early in their careers. Like I'm building this cuz it seems like there's a big TAM, but it's like but do you care about that? The other one and this is maybe on the surface the opposite advice is like growth just solves everything. It's all just gets if you can go out and get users that get paying users then all the other pieces can fall into place around raising money and getting finding co-founders, getting employees, all the other pieces work. If you can figure out how to grow your product

Make your product not so weird. It will happen right once the majority of the population think you know this is a needed product then you're not weird anymore.

Yeah. Yeah. And I guess for that second advice, I mean, I think that maybe reflects how I'm thinking about things too in that like I don't want to do B2B sales ever again in my career. I think in my first startup, I was I never got out of the sales role and I never wanted to ever be in a sales role. And so if you if you really are set on wanting to do a startup, the fast the best the tried and true path to like actually getting to do it is to go do sales and get customers and sell your product and everything else will fall into place. I'm now at a stage where I will have lived my the rest of my life well if I never sell something ever again. Not because I of a disrespect for the sales craft like I think it's just it's that I don't like doing it.

And but I have a tremendous respect for how difficult sales is because of all my years doing it and for the people who are good at have a lot of respect for them, but I just don't want to do it anymore. So like I'm looking, you know, that's why I'm espousing this idea of go do something weird that you don't have to sell. Like it's just weird enough that like for whatever reason it catches

It will sell itself because you're enough. Okay. I take that as the key advice from this episode.

Yeah. And I think there's probably a reason I'm working at a company now and not building my own startup, right? This is my north star, building something weird and not building value. It's probably best that I stay employed. My happy place is being deeply absorbed in a problem and not context switching or talking to people all day that about the same thing I've already said to them said to people 100 times before. That was the part that crushed me at my last company doing sales is like I've had this. It was like watching the same movie. Every sales presentation was watching the same movie for the 10,000th time. Like here's the part where I say this. I felt like I was my own little puppet. Like here's the part where I say this about our value proposition. Here's the part where I say this about this objection that the customer the prospect is having.

I was reading a book about like written by someone who's doing hostage negotiation and he was talking about it's called Never Split the Difference. I don't know if you've heard about that.

Yeah. The Chris Voss I think right? Yeah. Very famous negotiation coach

Right? Sometimes I'm thinking, should I treat sales calls as hostage negotiation? So, it's more fun and then, you know, trying to read the read the reaction of the other party and try to make a deal. But it's very hard. It's so hard.

Yeah. I think that's probably what's I mean, the sales people I've worked with have been so good. I think that they are fascinated by people in the same way that I'm fascinated by technical challenges. Like a technical challenge. I want to understand all about it and how you can make it work. And I think they're instead fascinated by like who is this person I'm sitting across from and how do I bring them along in the journey I'm trying to bring them which is just I like people but that's not my fascination.

I see. Yeah, that makes sense. It's it's like different machine you have to understand and read and analyze. Is there a darkest day that you had building this or being part of LiveKit?

Not LiveKit. I just told I just confessed my darkest days from my first startup doing sales and just being done with done with sales. But at LiveKit it's all been good.

Okay. Okay. LiveKit is good. Good life then. Yeah. When you feel vulnerable who do you talk to someone like how do you solve it? Who do you talk to?

Adrian. I talked to the CTO of my last company who was the of my first company who was the co-founder of my second company and is now the lead engineer on our inference product at LiveKit.

Okay. How does he help you? He sounds like another technical mind.

I think he just I mean we've been working together for 15 years. He knows the sorts of things that get under my skin and can kind of talk me down from it. When it comes to work-related stuff. I mean, I talked to my wife too about work-related stuff as well in sort of a different way where it's less in the weeds generally, but it's more and more about kind of the higher level work stuff that's getting under my skin, but I feel like Adrian has been always there looking out for me. Running your own business consumes you. It has to because like it only survives if you make it survive. Yeah. It's not like it's not like one of those things where one of those choices you make other people make choices in life where they work can work a job and then they can leave it at the job. That's not the path you've chosen. But it's also more interesting like for those hours that you have to be at work that you're working. It's interesting.

It's your journey, right? That I think that's what's so fun about it is it's your own journey and you're making progress on your own journey. It's your own story. Whereas if you're building widgets for whatever company, it's not your journey. There's no there's no sense of the human journey.

Do you feel better now? Like now that you're working for a company, you have better lines with your with your work.

No, because LiveKit is a rocket ship. I dream about LiveKit at night just like you were talking about. And yeah, it's just like I'm I was telling our co-founder David this about the other day. Is like I'm just trying to not fall off the rocket. There's just so much happening cuz we're growing so fast and doing so many things.

How big is it?

It's all consuming. It is 100 people around 100.

Okay. And you have conviction about it?

Oh, I've never been a part of something like LiveKit. I don't think I added this to my list of reasons why I'm bullish about LiveKit, but another reason I'm bullish about it is like people are going to be talking to computers. They're going to be talking to robots and they're going to be talking to websites and we power that. This is the future.

But you're at the start of it. So Mhm. You could retire and never do sales again.

Yeah. Hopefully. Hopefully. Then I will have lived a good life. Not because like I want to follow that up with for my friends in sales are listening that not because I don't respect their craft. Their craft is just too hard for me. I just can't do it.

I mean, you are a parent with two kids same age as mine. And what are you teaching your kids, you know, today with this AI hype, everything going on and why?

I'm hoping that they find the things that they find interesting and weird to them and that they pursue those. I hope they have the opportunities to pursue those things. I don't like that path that you described earlier. I came from where I went to high school is very similar. Everyone was talking about trying to get into the best school and I think that's the wrong framing. It should be like what is get into the right school for you. Take the right path for you. I hope that's more of what the dialogue is for our kids as they start having to figure out what their professional paths are.

I'm just copying people around me and I thought that's the right path.

Oh, isn't that so true? Like that's what we do is like I used to think about that in my first startup when we were working out of San Francisco for a while and all anyone seemed to value was growth and money and I was like maybe we can value other things. We choose to value other things but everyone around you is valuing that everyone else is obsessing over their growth and stuff. It yeah it takes a lot of guts to value other things.

That's the thing. How do you teach your child to teach your children to have the guts to do what they truly want to do?

You model it like you're doing now. You're modeling it with pursuing your own path. Not doing what other people what seems like the right what other people think you should be doing.

That's why I'm really poor doing a podcast. Yeah. Do you feel there's some some type of human judgment that machines will not be able to replace?

Yes, absolutely. Because judgment is actually an emotional thing. You hear about this with the experiments with people that have there's how does that work? Again, there's some people that I think they're stroke victims that some of the emotional centers of their brain get knocked out and they can't make decisions because every decision they don't they can't make the decision whether or not to pour cold water or hot water into their cup. They can't make the decision whether to like eat an apple or an orange that because every decision is emotional and emotion is embodied. It's how we feel about things. It's not an intellectual thing. It's it's a feeling and I don't think at least the way we're building intelligence and machines now it's not any it doesn't in any way approximate judgment. It there is no feeling.

I think you'd have to I'm not sure I've thought about that before. How would you build feeling into an LLM? But it's it's just not it's not how we're building them. You need judgment is emotion comes from emotion. And another concrete way of saying that is that when people you present people with Sam Harris did a famous study of this where not famous study the famous Sam Harris did a study of this where they took fMRI images of people's brains and they present them with images of things that are disgusting like a food they don't like or they present them with images or statements of beliefs that they don't actually believe. And the same parts of their brain would light up. Like disbelief is a disgust. If you tell me something I don't believe, like that you know I live on top of a mountain. I'm like I don't I live in a valley, live in a Bay Area in a flat part. I would like I don't believe that. I feel disgust around that. And if you close your eyes and present yourself with some ideas you don't believe, you'll see a similar disgust however small emerge.

I don't know. What do you What do you think around that? Do you think that machines will ever have judgment?

I feel like it's like imagine putting a soul into the machine. That's what's missing. The representation is emotion. I think you we can feel we can we describe the these emotions but it's set of data that's never collected and cannot be collected because sometimes we cannot describe it right we have our language but language has a tremendous limitation. That's why you know when you when you this is really a big tangent when I when I look at like you know Zen Buddhism you know scripts I have no idea what they're talking about and then they're all saying like you know I don't know what they're talking about they tell you stories and then you ask them what did you learn from learning Buddhism for so many years for 50 years they will tell you nothing that nothing has a lot of things that they cannot be described and that's not captured data and that will not go to LLM. That's my read. That's my interpretation.

Oh, that's so interesting. Yeah, that is something I need more time to think about and digest. I love that idea of what you're getting out there idea.

Me too. Me too. I'm done, but I still have the rapid fire question. Let's go through it because it will still add some value and then it will just three seconds each.

Thank you so much.

But yeah, thank you so much as well. Let's go. Let's go through the rapid fire questions. Okay. One is LiveKit voice agent framework open source? Is LiveKit's focus video conferencing or voice agents?

Voice agents.

Okay. What is the easiest way to get started with building a LiveKit voice agent?

Go to our log into LiveKit and you can use our I did not say this in a rapid-fire way. Log into create a account on LiveKit and start building a voice agent.

Awesome. Okay. I have a traditional closing question. If you could recommend anything, well, if you could recommend a book to let's say yourself when you were 20 years old, when you just graduated from college, yourself today after building two startups and maybe yourself 20 years from now, what would they be? No pressure on us three. Just one or two is great.

A book for my younger self. I've enjoyed coding so much. I didn't start coding until grad school. So, 20 years ago, I would have recommend a programming book for myself. And then my current book that I recommend for myself is the what my 20-year-old self would recommend for me, which is to go read Homer's Odyssey again, read the classics again. Because that's what I was really into back then. And I feel like I've swayed so much towards the intellectual technical interests that have lost some touch with some of the more like emotional artistic interests of mine. And then 20 years from now, I do not know how to recommend a book for myself in 20 years.

Yeah, we'll see. We'll exchange ideas along the way.

Yeah, that sounds good.

Okay. Thank you, Tom. Like, you know, I know this is such a wonderful conversation. I feel like I learned from you everything I need to know to start building a voice agent. It sounds like LiveKit is the solution. Also, thanks for your generosity of your time and sharing your deep thoughts and wisdom with me as well.

Yes. Well, thank you for sharing your deep thoughts and wisdom with me, too. That this was this was wonderful. Thank you.