Everything You Need to Know About Browser Agents ｜Devi Parikh @Yutori

Welcome back to TwoSet AI. So today I'm talking with Devi Perak, co-founder and CEO of UTory. This company is building AI agents that navigate the web for you. Devi, thanks for joining. Yeah, I'm so excited to be here with you today. Honestly, your background is just so expensive, you know, from research to art and to teaching. I don't think I can do you justice. I'll let you introduce yourself.

Thank you for having me. you ⁓ Yeah. Hey, everyone. ⁓ My name is Devi Parikh. I've been ⁓ working in AI for about 20 years now. My background was originally in computer vision. Over time, I started getting interested in seeing if we can find ways in which people can interact with these systems more naturally, which is what sort of drew me to multimodal problems ⁓ in AI at the intersection of vision and language. So things like, we describe images in sentences? Can we answer questions about images? Can we have a conversation about the content of an image? This was back in 2014, 2015. So this was at a time where the first deep learning excitement had already happened. And so we were starting to feel like, these models are onto something. And so it was sort of fun to explore at the boundaries of what's possible. But it was before all of the current excitement around GNI and LLM. So these models didn't really work. ⁓ all that well. So was very much in the research ⁓ domain. ⁓ And then over time, I started getting interested in seeing if we can use AI as a tool for enhancing human creative expression, which is how I started getting involved with generative models for images and videos and other modalities. Some of that intersected with generative art that you were alluding to. I was in academia for a while, faculty at Virginia Tech and then Georgia Tech. And then I was at Metta for about eight years. for the first five years at FAIR, the Fundamental AI Research Organization, and then in the GenAI Organization for a couple of years. This was up until early 2024, which is when my co-founders and I left Metta to start UTORI.

You've seen it all, right? From the very beginning.

It's been a little while, yeah.

Yeah. I mean, I love starting with this question. So I'm going to tweak it a little bit. ⁓ So hear me out. Can you explain what Utory does as three levels? How would you explain it to, let's say, elementary schooler? ⁓ And then maybe your neighbor who's not in tech. Or the last one would be an AI researcher at OpenAI.

So I think, yeah, I was going to say that I actually might give the same description to all. Maybe the last one is a little bit different, but let me give it a shot, and then we can tweak it based on what you think. So what Yutori does is, right now, if you think about how you interact with the web, you go to a browser, you open up a website, you sort of click around on buttons, you might have to fill out some forms.

dancing? That's good.

to get the thing done, whatever it is that you're trying to do, that's how you do it. And what we are working towards is a way in which we interact with the web that's entirely different, that's at a different level of abstraction in the sense that you don't have to click around buttons anymore. You're not filling these forms out. You're just kind of describing what it is that you're trying to get done. I'm trying to buy this product. I'm trying to book this ticket. I'm trying to ⁓ look up certain information. And you just describe that, and you have this AI that is in the background just doing this work on your behalf so that you no longer have to do this mundane clicking around, and AI will do it for you. ⁓ And either whatever it is that you're trying to get done will just get done, or whatever progress it makes towards that, it will let you know of updates along the way as it's doing it. ⁓

Thank

Yeah, so hopefully this is sufficiently accessible for an elementary student. I might sort of tweak it a little bit, but hopefully this is sufficiently. ⁓ Yeah, yeah, that is one way of looking at the Reconcierge service that's just available to many more people. Yeah. Yeah. And then for an open AI researcher, I would say we're building web agents, right, which is, ⁓ yeah.

It's like on the edge service. Not for the elementary schoolers, maybe they don't even know what that is. Yeah. that's easier. seems like easier. Awesome. Cool. I saw somewhere mentioning maybe on your site talking about AI Chief of Staff. Is there another way of how you position it?

So. Yeah, so when you think about this entity that can help you, ⁓ but you just describe what it is that you're trying to get done. And if there is this team of ⁓ agents, web agents in the background that's doing it on your behalf, that is like having a chief of staff in your life ⁓ that's taking care of a lot of things, especially when you start intersecting it with a little bit of proactivity where you don't even have to ask. And there's this entity that just notices based on your context. that here's a thing I can probably start working on that would be helpful. That's how it ⁓ would show up in your life.

Yeah, that's crazy. I really cannot imagine the world would live like that. A little bit, but yeah, it's still very, it's hard to grasp at this moment.

Yeah, I feel like it's hard to fully grasp and concretize exactly what this interaction would look like. But the feeling of what it would feel like to have this entity that just takes care of a whole bunch of things in your life, that I feel like is a fairly relatable feeling. exactly.

Yeah, it's exciting. It's very exciting. So can I ask why did you build U-Tory? You know, like I'm holding a Silicon Valley perspective. You had arguably the best research job in AI space. So what made you personally decide to do this?

Yeah, so a few different things. ⁓ So yeah, you're right. I was a senior director in JNI. JNI at the time was sort of the top priority organization at Metta. I was leading all of the multimodal research teams. So yeah, awesome people, awesome work happening. The company cared about it. So that was all great. I think for me personally, ⁓ as I grew in my role, ⁓ I was sort of starting to miss driving the work itself, being more in the weeds. My position had become more and more of making sure the right people are talking to each other and stuff isn't falling through the cracks and someone might be complaining about something on any given day. So I just missed being in the weeds, being in the details, and driving the work itself forward. So that was my ⁓ personal motivation to consider a change. The other ⁓ very important thread here is that the three, the three founders, we've known each other for a long time, very good friends, worked together very closely. ⁓ We had always talked about like going back six, seven years had talked about doing something together at some point. And so my co-CEO ⁓ Abhishek Das, he was independently also thinking of transitioning away from his role leaving Metta and he was thinking of starting something. And so that timeline aligned well. And so he was like, we should do this together. And then our third co-founder ⁓ Dhruv also joined us a couple of months after that. yeah.

Mm-hmm. Perfect timing seems like. How old is you, Tori?

Thank you. Yutori, so we left Meta in March of 2024. We weren't sure exactly what we would be building. So I'd say maybe it took us a few months to sort of have that vision and sort of make it crisp in terms of what we're going after. So depending on how you count out, maybe about a year and a half or so.

Mm-hmm. So this is a philosophical question. After 20 years in AI, right? What motivates you now versus, you know, if you remember what motivated you at the beginning?

Okay. Yeah, I think the motivation at the beginning and even now is somewhat similar. I think I have this idea that a machine can be doing all of these things and can sort of, that us interacting with a machine resulting in ⁓ higher efficiency or more productivity or just more space. to allow us to focus on the things that we care about, whereas the things that we would rather not be doing start getting taken care of for you. ⁓ I think that's fascinating. So early on, I think it was just the tech fascination, the fact that a machine can be doing this. You can give it an image, and it can recognize objects for you. And then over time, it's even give it an image, and it's going to talk back and describe it. And you can ask it questions, and you can have this conversation. And I think right now,

Yeah.

We are all just so used to that, that like, of course you have a conversation with the machine. But 20 years ago, that was like far from being true, right? Like that we were never close to that being the case. And so this idea that like maybe even one tiny step in that direction was fascinating from a tech perspective. So I think that was the early motivation. And I think now I'm quite excited about how we interact with this tech and how, like I was saying, if it can play a role.

Yeah.

in giving us more time and space so that we can optimize our lives in ways that we care about, right? It's not about a prescriptive notion of like, you should be spending more time doing this or more time doing that, or you should be more productive or more efficient. It's not so much about that. It's more than whatever you care about. Do you have more time and space and energy to be able to go after that? If we can make a dent towards that, I think it would be pretty amazing.

I'm yeah, I am 100 % with you. mean, I need more time. have like two young kids. I'm assuming you have kids, maybe you don't have kids, but still right. The information is just everywhere. There are a lot of things, everything scattered. We have so many things to do. So that's really challenging. I want space too. So I'm counting on your tool. I mean, I tried it out already, but I'll tell you how I feel. So.

Exactly.

So when I first see Scouts, Scouts is one of the products that you're talking about that navigates the web and finds things for me. So I'm thinking if I were to build Scouts from scratch, like as an outsider, here's how I would approach it. Can you let me know if I miss anything or tell me why in my audience, like why my approach may not work? I have my very naive thinking of this. So.

Mm-hmm.

So it goes out, I have a query, I ask my question, and it will give me some answer in the format of an email. So I'll start by, so search is involved. So I'm going to start with some sort of search API, so Bing or Google, to find the URLs based on the user queries. And maybe do some fan out query rewriting to broaden coverage. And then, know, semantically match the results to the user intent, like, you know, the simple things, cosine similarity and things like that. And I'll summarize each link and then I'll generate an email digest, an email to the user. So this, is there anything wrong with this approach? What would break?

Yeah, yeah. This will be fun to go through. So let me first describe for the audience just what Scouts is, and then I can comment on what you were saying. So Scouts ⁓ lets you monitor the web for anything that you care about. So if there's ⁓ a product price that you are tracking and you're waiting for it to fall below a certain threshold or product availability or

Sounds good.

tickets to certain local events or any research updates on specific AI topics or news about startups or fundraisers or anything of that sort. can set up a scout to monitor it. so anyone who's interested in trying it can go to utorii.com. It's now GA as of last week. Actually, I don't know when the podcast will be out, but as of this... Yeah, it is generally accessible now. It's no longer behind the wait list. So anyone interested can go try it out. So that's what the product does.

It just released. Yeah, yeah.

Now, to your point of, it be a Google search and some assessment of similarity to the query, and that gets sent you over email. So I think there are a few things where this wouldn't work. Let me first start with the more, ⁓ maybe the one that's easier to see why it wouldn't work. So for example, if you said, let me know whenever ⁓ this tennis court ⁓ becomes available for Monday 7 a.m. slots, right? So like some local tennis court in my neighborhood. Now if you think about that, like Google search is not going to find that, right? It's this local tennis court website. You have to go in, then you have to pick the date, right? Like when you're saying that for Monday 7 a.m., is this available or not? You have to go in, you have to pick the date, you have to pick the time, and then you have to look, is this available or not, right? There isn't...

Hmm.

Right, like that isn't just a publicly available list of all slots that are available. You actually need to fill out the lightweight form to be able to get that information. this is one. Exactly, So this is one. ⁓ And this might feel like, it's a niche, like just about local tennis courts. But if you think about it, the heavy tail of information on the web, which is sort of a significant mass of that, is about.

So it requires clicking and then, yeah, okay.

information that exists in databases but is not just listed on the web page. And you have to click around and do some interactions to actually pull. So restaurant reservations is the same thing. Tickets to concerts is the same thing. The price of both availability and the price. Product availability, product prices, same thing. So a lot of times, even for apartments, if you're on whatever apartment website, there are filters that you have to pick on, like which neighborhood or what. range of rents. Are you interested in two bedroom, two bath? There's a whole bunch of check boxes and sliders that you have to interact with to find the relevant results. It's not like there's some long list that Google can just sort of crawl and find that for you. So there's vast.

Not crawlable, right? This is not crawlable information. Right now, it really requires the human being to go in. I really have to go to the website and do some filtering and maybe get that data I want. Yeah, never thought about this use case. Go ahead.

Exactly, exactly, exactly. Exactly, exactly. so that's exactly where ⁓ we train our own in-house browser use model. So the model that does this clicking to find the information that is relevant to you is a model that we post-trained in-house. And that's a big part of the stack that we've built. So this, feel like, is one clear ⁓ set of examples of what would work with just Google Search. Even for, yeah, sorry, go ahead.

⁓ I'm curious, like, what does the model do? What does really the model do? Because clicking is part of the agent flow. It's not a model. The model is, I'm assuming, the visual language model you're talking about.

Right, so the model takes in as input the screenshot of the web page that it's looking at, with whatever the history of the trajectory is so far and whatever the query was, whatever the task was, and it decides.

What why? Yeah, why screenshot?

So we actually put out a blog post on exactly this, that like why screenshot and not like the DOM and HTML. Exactly. So when we first got started, our thinking was that the rendering of the website is very much intended for a human to consume it. But behind it is this machine code, like there is the HTML, there is the DOM, there is all of that information. So it makes sense for the machine to just use that directly, like why use this rendering?

Like you read the HTML, right? So, yeah. Yep.

And initial experiments is that is what we were doing. And what we found over time is that different websites are built in such different ways that there is so much noise if you just sort of take that raw HTML DOM information that it's very hard to train models to be reliable enough. And so then what you end up finding yourself doing is to kind of process that DOM and process that HTML in some way to extract the information that's relevant. to then feed it to the model in trying to make the model more reliable. But the problem is that the way in which you parse and sort of clean this up becomes so website-specific that you just can't scale this. So if you wanted to build this for any one website, it would make a lot of sense to do it this way. But if you're trying to build web agents that are general, that can interact with websites just across the entire web, this is just not going to be scalable over time. And so what we found is that just using the screenshot is what gives you that generality.

Question for my professor. So when you're talking about this, just reminds me that you're talking about dynamic content like generation, right? So the JavaScript rendering is part of it that we won't be able to see it as well. Is there a way, like, ⁓ so let's say, is it possible to somehow use a tool like, I don't know, Selenium or

Go ahead.

and just render it first so you have the information. And then you can still go with that approach without the screenshot. That's you're talking about the one time using. Right.

But it. But even if you render the page, is still the underlying ⁓ code is still very messy. There's all sorts of things. There's like a whole bunch of hidden elements that are being described in the HTML that are not being rendered. And your model is now going to have to figure out that this stuff that's being described is actually not even available. It's not even something that you can interact with.

And NLMs cannot distinguish that because it's so messy that it doesn't do the job.

Yeah, yeah. And your context also blows up, right? Like you're putting in these massive numbers of tokens to ⁓ get to that. And so anyone who's curious about this, the blog post that we put out there walks through several specific examples of the different ways in which this sort of breaks down. And so ⁓ it's worth ⁓ walking through that.

So, yeah. Yeah. That's very interesting. So this is, is this an uncommon approach to taking screenshots or

So it is ⁓ depending on who you talk to. I think there are ⁓ teams who over time have also figured this out. But I still see a lot of startups in the ecosystem ⁓ going the route of not using screenshots. And the reason for that, in some cases, that is very well justified. Like I said, if you are trying to ⁓ optimize for workflows on specific websites, it makes a lot of sense for you to build this cleaning up and parsing process. for that website, because then that also gives you a cheaper model. There's very limited tokens. You don't have to process the image. The model can be faster. And so there are situations in which that makes a lot of sense. We had been seeing ⁓ companies' teams try to go after generality while still trying to stick to the text-based ⁓ representation. ⁓ Yeah, I think some of the larger labs probably have already arrived at similar conclusions as we have. But yeah, we were seeing a bit of a mix on this front, which is why we sort of decided to write this up and put it out there.

I just recently started using Chrome extension from Claude. And I'm noticing that they are using screenshot. They're basically taking screenshots and then doing actions. So they have action plan. And then it will ask me to approve it. And it will take action on the browser. But the way they do it, I see the steps. the first step is taking a screenshot.

Yeah. Mm-hmm.

So.

Yeah, yeah, yeah, yeah, makes sense. A lot of the computer use models from the larger labs, I think, work that way. Yeah.

So the green minds think alike.

⁓ Yeah, good. Yeah, the community has a way of sort of arriving at similar ideas over time. it's sort of, yeah, just kind of moving that along kind of thing. Yeah.

You mentioned about this clinking, clicking through that this is multi-step, right? know, human has to come in and you have this approach to solve that problem. Did I miss anything else with my naive approach? You had something else, I think.

Yeah, so this was sort of the ⁓ more clear set of all of this information ⁓ that lives behind lightweight forms that you have to ⁓ fill out with interaction. The other relevant bit is also that if you are, for example, if there is something, if you, let's say, an example, scout that some of our users set up is monitoring for, like a startup founder might set up a scout to monitor for ⁓ any negative reviews for their competitor. because then it is sort of like a lead generation for them to reach out and potentially sell their product to them, right? So if ⁓ you think of this use case, sometimes these negative reviews might just be like on some subreddit on some thread, someone has left some comment about the product, right? It not be something that is like a post that has gone viral or anything of that sort. It's some deep conversation that's happening in the corner.

You're so smart. Mm-hmm.

And it's unclear if just a generic Google search across the entire web will manage to surface this for you. And so part of our stack is a whole bunch of APIs and a whole bunch of MCP servers beyond just web search that go and find information from all of these different corners of the web, which means the coverage of the information that we can return to the user is much higher than if you just took a web search and put it ⁓ in like, yeah, like founders results and surface to users. So these magical moments of like, I would have never found this myself is much harder to get to if you have sort of just the web search piece of it. ⁓ And it's some of our, and I continuing this example of sort of founders, some of our founders have some of our founder users have told us that if there is like a YouTube video that has gotten a whole bunch of views that is reviewing, let's say their product. that would have found itself to them anyway. Someone or the other would either ping it to them or it might find it to them anyway. But what they find valuable is the rest of this tail that they would have just never come across. And they're not going to have the time to try and dig it up. And so that level of coverage is something that you can get with these massive number of tools. Now, you think of a few, so one response you could have is that, OK, I'll just have these many, tools, not just one Google search. And then I'll just repeat what I said anyway.

Right.

The reason that is not going to work very well is a couple of different things. One is when you start getting to like a hundred different tools, these models start falling apart a little bit in terms of being able to orchestrate across all of this. So we, the architecture that we've built has a hierarchy to it, where there are sub agents that are sort of responsible for different kinds of tasks. And then those sub agents based on whatever sort of their responsibility is have access to certain tools that are relevant for that kind of work.

Mm-hmm.

Right? So like for finding information across like reddits and various other ⁓ social media platforms, there's one agent, one sub agent that does that and similar for others. So that's one. It needs to be this sort of hierarchical multi-agent setup that gets to it. And the second is even if there was somehow one powerful system that could deal with 100 different tools, the context windows are going to blow up because each of these tools is going to learn so many.

Yeah. confusion, right?

Exactly. And so having this hierarchical setup allows you to deal with that context explosion in a reasonable way. And finally, even if somehow over time all of this stops being an issue, costs are going to be very expensive if you of cram everything into just one overall agent. And so my point being that all of this orchestration and trying to balance the quality of and coverage of information that we are delivering with actually sort of keeping costs in control in a way where this product can go GA, it's not something that is easy to pull off. And a lot of our effort has gone into that.

Yeah. Yeah. That's a lot of thinking going into it. Do you have like a proxy or at least an example infographic about your subagent, agent, or orchestration that you can share? No, I'm if you have it, I'm happy to share with my audience as well because people are intrigued by how these things are designed.

let me. Yo. Yeah, yeah. Is it, actually...

There's a share button, like a screen.

Let me see if I can find a scout that would be a good one to show.

This is way more complex than what I initially thought.

⁓ Yeah, and one, actually, we can also walk through another relevant bit about this. ⁓

Okay. That's okay, you can share later. I can just put it up. So that's funny. Yeah, I tried it out myself and I really like it. It served as something that I didn't, I don't think I could have found by myself. So I'm like, this is very interesting. and, uh,

If I find one after the call, can I also? Yeah. Mm. Mm.

The diversity of the results, I can clearly see that it's not just a similarity matching thing.

Yeah, yeah, yeah. And so what it ends up doing is all of these individual agents are first optimizing for coverage and recall to sort of very exhaustive, find as much relevant information as it can, which obviously has the downside that precision may not be great. Right. And then the orchestrator at the higher level is in charge of making sure that taking all of this sort of the high recall

Mm. Yeah. ⁓

information that's coming in and then optimizing for precision. That's the one assessing that, which of all of these things that have been found is relevant for the user. That is what gives you a good pattern.

You find a lot and then you trim it down and present it to the user, right? That makes sense. Yeah.

And the other, and after this, I'll stop. But the other relevant bit here is that there is a sense of continuity to these reports, right? Like we don't want scouts to tell you something they've already told you. We want scouts to tell you new information relative to what has already been reported in the past, right? And so that takes a certain level of, ⁓ yeah, like having the context of what you've already been told to then

Bye. Yeah.

and the information to you in a contextually relevant way. Yeah, there's that level of.

I just remember, this is my experience, I'm having an email almost every day and then I just noticed that, it seems like Niu's what emailed me yesterday. It was more like a very nuanced and subtle thoughtfulness, Very nice. I'm like, oh, this is nice. If I talk with Chajabutie, tell me the top news today. And then it would just repeat the same news every time I asked it. It's the same news.

Yes. ⁓ Yo, yo. Yeah, exactly. It's the same scout that is coming back to you every day with any updates. And so you can feel that continuity in how it's reporting the information to you.

It feels, it definitely feels more human and more feels like I have this like intern and helping me out and doing a lot of legwork and amazing job. And you, by the way, just now you mentioned something interesting. so in order to search like every corner of the web, maybe this is the wrong way of saying that, but more corners of the web than the SERP of the world, like being, being bought or whatever. ⁓ how do you prioritize what's important to surface?

Yeah. I think it's kind of baked into the model, like the top level orchestrator ⁓ that sort of gathers all of this information that all of the subagents found and sort of is the final one that decides what to report back to the user. ⁓ I think it's kind of baked into it partly through the pre-training process and just sort of the common sense that ⁓ LLMs have in this regard and partly through our prompting that whatever, like we tend to have a point of view that the user has set up the scout probably because they are trying to do something with this information. So even though the scout is not going to finish that task for you, the scout is just going to find the information for you, there is this point of view that whatever information is probably the more actionable and probably the one that's relevant to any subsequent decision making, that should be prioritized in ⁓ the report and how that's talked about. So it's more of this kind of soft nudging in this direction as opposed to, because it's a very horizontal product, right? You can be monitoring for anything. It can be products, apartments, news, startups for your work, anything of that sort. So it's not like on a per domain basis, we can go in and be like, this should be more important or that should be more important. But this point of view that we should be helping the user make decisions or take actions in the most informed way.

Yeah. Right. Right.

And that soft nudging is through ⁓ which that surfaces. The other nudging that we have in there is that ⁓ there is a general sense for some sources are more reliable than others. If there's a random comment on some Reddit thread on some subreddit, versus if there's a news article about something, then there should be an implicit prioritization of which one is ⁓ more likely to be reliable information. ⁓ So yeah, that's baked in there.

You mentioned about a

a little bit as well.

lot of the, you build a lot of MCP. was thinking of like, you connecting to a lot of the data sources that covers, you know, different areas of the web.

in in Yeah, yeah. So things like flights information, ⁓ Reddit, like ⁓ LinkedIn, ⁓ various tools of this sort ⁓ that we know ⁓ are relevant for a lot of the things that the users are trying to do, ⁓ use those. And that's good. ⁓ One, for just getting more coverage of information, like you were saying, trying to get to as many corners of the web as possible. But it's also good for ⁓ latency. If you are going to try and click around and find that information, that's going to take longer. Every click means the page needs to be loaded, that screenshot needs to be processed, and then you decide the next action. Versus if you're just making a call to an MCP server, that information just comes to you much faster. So there's that benefit to it.

⁓ Do you need user guidance on giving you a hint which the corner of the web should you look at? Like I can give you a specific link, right?

Yeah, so in situations where the user already knows where this information lives, ⁓ they should just put it in the query. And we see a lot of our users do that. They're like, this ⁓ is what I want to be monitoring. If you know there's a very specific product whose price you're monitoring, and you know exactly the URL of that, and you're just waiting for the price to drop or just waiting for availability, ⁓ yeah, it makes sense to just give the Scout that link. And the Scout will do the appropriate thing. That is what it will monitor. So the more specific you are, either in terms of the source of the information or your preferences that this is exactly what I'm looking for, ⁓ the better it is, the more catered the result will be to what you're looking for. In some situations, it's sort of a more generic that I just want to stay updated on anything about this startup. And you don't know where exactly that information is going to happen. The other relevant thing here is that you can reply to the Scout report email to give it feedback.

Really? I didn't know that! I can email back? What am I gonna get who is emailing back?

I don't know if you've. So you can give it feedback. So you can say that ⁓ this kind of thing is not what I'm interested in, or I want to see more of this, or this is not what I meant. And so it's a way for you to nudge the scout over time, which makes it more and more personalized and higher and higher ⁓ signal for you. So this is something that we had shipped recently. so far, feels like users are finding it very natural to just reply to the email, give feedback, similar to your internal energy that you are using. ⁓

Yeah. Yeah.

It's a, that is a way in which you can sort of make the scout be high and higher signal.

So that reminds me of like I use skill.md on cloud. So I'm a heavy user of cloud. So that it kind of retains the memory for me. So it remembers what I want to do, like my style of writing, for instance. So you are implementing something similar, like personalized memory or context management for each user.

It is right now, it's for each user for each scout. So for the scout, whatever feedback you've given it, that is accumulated over time and it keeps track of all of that feedback and uses that as context the next time it's sending you a report. It is on our minds to also generalize this to be more for every user rather than on a per-scout basis, but right now it's on a per-scout basis.

face. That makes sense. think my, I may ask scouts to do different things. So it kind of makes sense as well. Right. I have a, have a grand scheme about who I am and what I want, but I can deploy two scouts at doing like completely different things. that makes sense.

So. Yeah. And how much of the feedback is telling us something about your preferences as a user and how if it is feedback to this specific, like the specific thing that you're trying to get that scout to do ⁓ is something that would need careful treatment. ⁓ Yeah.

Mm-hmm. Yeah. I mean, this is another area you can, you can write about a blog post on memory, like memory and context management. A lot of people, the audience would really love to see those, right? Yeah.

Yeah, that's actually something that we're working on right now, the next blog.

I would love to see it please, Lee. Announce it on LinkedIn. I'm happy to broadcast for you. I can post on the community at YouTube as well. Yeah. And you also mentioned LinkedIn. I mean, I'm trying to be a heavier LinkedIn user. But LinkedIn is notoriously hard to scrape. How you doing it? Is there any, yeah, is there any challenge, technical challenges working with LinkedIn?

Yeah, yeah, that would be awesome. ⁓ I mean, we use ⁓ existing tools that are out there. We haven't built anything ⁓ specifically in-house for it. ⁓ Yeah, and so we're using some tools that are out there. ⁓ Scouts right now can only monitor public information. They can't log in on your behalf. So scouts, for example, won't know who your connections on LinkedIn are or anything like that. It's just that if there are public posts on LinkedIn that are surfaced,

Okay.

Like even if you Google for LinkedIn posts, you will find things, right? So ⁓ just that public information that we can read. don't, yeah, we don't, yeah, we're not going behind all the walls right now, so we won't know your connections.

Right. Yeah, yeah, pay word, pay word, content and account logging into content. I mean, log into your account is another level of, you know, digging deeper, speaking of taking actions and clicking more on behalf of the human. Is that on your roadmap?

Yeah. It is on a roadmap. We do want to get to that over time. It is a way of digging deeper, and it's also what needs higher user trust. And so we kind of have to own that trust and build our way towards it.

Yeah, yeah. Yeah, yeah, I mean, I can have wild imaginations of what these scouts and agents can help me to do right, right. For instance, I can ask the scouts to book me watch for airline ticket to Japan because we you know, my family we go to Japan every year. And then I would love to know when when I should buy and why don't you just buy it for me, right?

Yeah, exactly. That's exactly what a lot of our users have told us that like scouts right now will notify you when the price drops or when something is available. But every so often you might not see that notification in time, right? By the time you see it, the price might have gone up again, or maybe it's no longer available. And so one of the ⁓ experiences that we want to build out is the scout. And one, instead of email, maybe texting you and being like, this is available. Do want me to buy it for you? And you sort of just sort of

Yeah.

say yes, and it goes ahead and does the rest of the workflow. So those kinds of things is something that we're working towards. On flights, it's interesting that you mentioned this. One of our users had set up a scout for, they had a lot of flexibility on when they can be traveling, but they really wanted to find a good deal. And so they had said that over the next six months, anytime there is a flight from San Francisco to Delhi, or I think it may have been Bangalore, that is less than $500, let me know.

Yeah. Mm-hmm. Okay.

the flight. They found the flight that way. They bought the ticket. And this is the kind of thing where you are not going to sit there, check for six months. Right? Exactly. You're just not. ⁓ Yeah. ⁓

No. No way. Yeah, I need to share scouts with my husband. feel he's spending too much time looking for flight skills. I'm like, let's stop looking at this. It's crazy. Yeah. But I would think some things change faster, and some things might like flight price, right? But some other things might change slower. Maybe like campsite available somewhere, Yosemite off season.

Ew.

Do you have some type of adaptive reasoning or whatever you call it to decide when you check and how often you check?

Yeah, so one, the user can specify this. So when you set up a scout, ⁓ you can say that I want it to check every week, every day, every hour, every month, whatever ⁓ makes sense to you both based on the domain, like whatever it is that you're trying to check, and how much you care about getting that information ASAP. So there's that. You can also say that let me know whenever blah happens, in which case you are not necessarily specifying the frequency directly. And there we have a little bit of intelligence built in that based on how often it's checked and hasn't found anything, based on that it decides when to check again and things of that nature. I think there's much more that we can do there in terms of being intelligent about how these scouts are scheduled. We've only scratched the surface of that. Yeah, that's where we're at right now.

That's super smart. I would, I mean, my naive approach would go with maybe a fixed range of time, like, you know, 20 minutes every 20 minutes, but then you have a cost consideration on top of it, right?

Exactly, exactly, exactly, exactly. And it's interesting to think about the cost implications of this. If you think about an LLM, it's basically a for loop over token generation. You're generating ⁓ tokens ⁓ one at a time. If you think about what an agent is, it's basically a for loop around an LLM. Every time it's looking at, if you think about web agents, every time it looks at a screenshot, it decides what action to take next. That then changes the website. So looks at that screenshot, and then it decides what action to take next. And each time you're deciding what action to take next is a call to an LLM. So this is agent is now a for loop over an LLM. And a scout is a for loop over a whole team of agents, because it's constantly monitoring. It's basically a crowd job over that. so, this is one perspective of ⁓ getting a handle on what costs end up. could end up looking like, which is where the fact that we have trained, we have post-trained our own in-house ⁓ web agent, like the model that takes actions on the web, becomes relevant. Because it's in-house, ⁓ that has sort of a significant dent on how much it costs us to serve the product, which is what makes it even feasible for us to open up Scouts ⁓ to everyone to use.

Yeah, I mean, that's why, you know, I feel so lucky that I could invite you and your co-founder to the O'Reilly's conference on context engineering next year, because I feel like you're dealing with the hardest topic.

Yeah, yeah. Dhruv Batra, my co-founder and our chief scientist, is going to be giving that talk. And he'll cover a lot of, like even the blog post that I mentioned that's going to talk about context management as one of the pieces. He'll cover a lot of that content in that talk. So yeah, it should be quite fun.

Right. Yeah. Yeah, I cannot imagine if let's say you have a thousand scouts right now and how do you scale it up to like millions? What's it what if right that's a good problem to have for you. ⁓ How do you plan to solve that problem?

So we kind of did all of that homework ⁓ before we went GA, right? Like the product was behind a wait list for a while and we did a lot of this sort of cost optimization and like making sure we can handle everything at scale and all of those things is what got us to being ready for GA. ⁓ And so, yeah, we kind of are ready to, we have been scaling up in the last week with all of the new users that have started using the product. And we are well set to be able to ⁓ keep going on that. It will be more sort of the infra, like the vendors that we work with, the partners that we work with. And expanding that infra would be where the bottleneck is. But in terms of the agent loop itself, we're in a good place.

Good to know. Yeah, fantastic. I can't wait to see your blog post some more on the technical side of things. Did I miss anything else? So you covered a bunch of things that I missed with my naive approach. Anything else I missed?

Thank you. ⁓ I don't think so. I don't think so. We talked about the natural language feedback ⁓ to reports, which also plays a role in this orchestration to make it more personalized to the user. But yeah, I think we kind of covered the big pieces.

My conclusion is I don't think I'm going to build it myself. I would love to make an amazing product that actually can do it. Seems like a lot of work.

Yeah. There are tools out there that are built in similar ways to what you were describing. And I'm curious to dig deeper. It's worth setting those up, setting up some scouts, like comparing the results for themselves to get a qualitative sense for how this compares. ⁓

Yeah. Is chat dbt pulse doing something similar? know, they just launched pulse, I think maybe two, three months ago. So it's also a proactive AI kind of deliver something updates personalized on a schedule from the outside. Sounds similar to scouts.

Yeah, yeah, I think ⁓ the proactive nature of it ⁓ is relevant. ⁓ They sort of fully, like you don't have control over, like they just set up the pulse based on all of the context that they have access to as opposed to you sort of saying that like here's a product, here's a thing, right, like things that are on your mind. ⁓ There isn't a way to trigger that. There also isn't, excuse me, it's at a fixed schedule.

Thank

And so if you wanted to say that whenever this happens, let me know. That's not the kind of ⁓ interaction that ⁓ it can support. ⁓ And my guess would be that the exhaustive nature of what I was describing with finding information from all of the different corners of the web, ⁓ my guess would be that that level of coverage is not something that we've seen other products be able to provide.

One of the unique thing about, uh, you Tori scouts was this is, um, sounds to me very goal oriented. I mean, it's helping me to, cause there's condition on top of the queries. maybe let's say I'm monitoring for price drop for my flights. Then there's a condition that needs to happen. The comparison have to happen. So it's not a search problem. You have to pack it over time as well, basically. Right.

Yeah, exactly, exactly. Exactly, exactly, exactly. And exactly, yeah. So this thing of, let me know when this happens that has this conditional nature to it that you're describing. You can also set up scouts at a regular cadence. And if, like, every day at 9 AM you want the latest on something, you can also schedule it that way if you want. But a lot of the scouts do tend to be conditional. ⁓

You know what, I don't think I'm using Scouse to its fullest power. All I'm doing is, can you tell me most recent updates about AEO and geo AI visibility since I'm working on AI visibility? I would love to know that. First hand, what's the latest research? I don't think I'm using it to its fullest power.

it. Yeah. Yeah, this is actually something that was very frequent feedback that we were getting while the product was behind the wait list where users will like, they'll set up their first few scouts. takes like a day or two to get some reports. And then it kind of hits you that like, wait, this is pretty powerful technology. And then they need it feeling after that is like, wait, I don't think I'm leveraging it as best as I can. And even a little bit of FOMO that like other people are probably using it way better than I have figured out. And so as part of the GA launch.

Yeah.

We've done several different things. Like now, if you go to utor.com, there's like 70 different examples of scouts ⁓ that you can create based on many different categories so that you can get more ideas for the kinds of scouts that you could create. And the second is when you first onboard of the product, this was not the case when you started using it because you had access earlier before GA. But now when you onboard,

No, yeah.

We ask you some questions about you, like what do you work on, and are there any products that you're looking ⁓ forward to, and things of that sort. And we recommend automatically generate four scouts for you, that if you like those, you can go ahead and create them with just one click of a button. And so that, I think, lowers the barrier to being like, what kind of a scout should I be setting up? ⁓ So yeah, that should end with this.

That reminds me of the prompt engineering phase of AR era. We're learning how to prompt, right? It's talking to the machine. It's one step, one turn. And then now I have to talk with an intelligent agent. I feel like I need to learn how to communicate with the scouts.

It shouldn't be that way. The hope is that our goal is to build the product in a way that it naturally meets you where you are rather than you having to adapt and figure out how to talk to this thing. We put a pretty heavy emphasis on you being able to be as natural as possible. I'm sure there's more work to be done in that direction, but that would be the hope. The hope is that you don't have to adapt to your scout. The scout adapts to you.

Yeah. Yeah. You just confirmed, I believe, the prompt engineering, there is a learning curve for ordinary user, right? Yeah. What's your take on, know, hallucination? Like, you know, with chat bots, hallucination means wrong information, but with agents, it could mean like wrong actions, right? How do you handle that? For instance, if let's say I can use Scouts to buy me a flight ticket to Japan, and then it tells me $500.

Yeah.

And then it actually acted on my behalf and bought it at $800 or $8,000. Oh, I'm going to be really mad. And plus with agents, know, these errors compounds as well. For a 5 % error rate, 10-step can compound to 40 % failure rate. How do you handle that?

Thank Yeah, yeah. So one, right now, Scouts is read-only, right? Scouts does not make any reservations, any bookings, doesn't buy anything on your behalf. It only brings you the information. So right now, in the product, this is not a risk that we run off, like, yeah, something being done that wasn't your idea. Even in the context of information, though, like, it could still tell you that, the ticket is $500, and then you go and check, and it turns out it's something else that's going to be super annoying, right?

We're safe. Yeah. Right.

Yeah, a lot of the evaluations that we do internally, all the way from the model that takes actions on the web, so evaluating it for every decision that it made, every step that it chose to take, all the way to the quality of the report and the accuracy of information in there. Through that entire stack, we have a whole bunch of evals that we run both automatically and with humans in the loop. So yeah, that's the main thing. That is the main quality bar. that we hold before we ship things, before we ship any changes. In the reports, we give you citations for where that information was found. And so you can click on those to verify it, and that gives you a little bit more confidence. And finally, each report has this Inspect Work button, which I don't know if you've had a chance to try. But if you click on that, it shows you what the agents were doing behind the scenes that resulted in this report. So it's a way where you can go in and check that, like, wait, where did it find this? Or what sources had it looked at? Which all helps with just ⁓ getting more confidence ⁓ in how this works. For going forward, that is exactly what we need to be working on. That is what we are here for, that trying to get these agents to be ⁓ reliable, having the right evals, having the right guardrails in place, and in setting user expectations, right? Like in Scouts. We are not telling you that, this is your chief of staff. We are very clearly saying, this can monitor information for you. It doesn't promise to do anything else. And that same thing we'll have to carry over as we keep adding more capabilities, being very clear about what this can and cannot do ⁓ is going to be important to just build that trust over time and go deeper.

Once you crack that nuts, I'll give you, I'll give my Scouts my credit card. Please buy my ticket.

Yeah, and even on that front, there's a whole bunch of guardrails, right? Like you could say that anything that's over $50, make sure you double check with me. And only if it's less than $50, the autonomy to go to go make that purchase, you could be approving sort of transactions only for this specific thing and not have it be. So there's a whole bunch of guardrails even on that front, like not just the model, but even just how the product is built and designed and how the transaction is even happening.

That's really good. Yeah. Yeah. It's a very human process. If I hire somebody to do these things for me, it's similar interaction as well. Like I give you a budget, like this is more than 200 bucks. Don't buy it. Something like that. Yeah. So you lower your risk. Yeah. Do you have a blog on eval as well? I'm going to share it as well.

Yep. Yep, exactly. We don't have a blog on eval. So the blog that we had on web, ⁓ like on Navigator, ⁓ our N1 model for taking actions on the web that walks through a benchmark and compares our model to others and things of that sort. But we haven't written one specifically just on how much of the eval we do across the tech. That's a good idea. We'll have to consider that.

for data. Yeah, let me know. I'm happy to share that. It's really valuable information for anybody who's building an AI space, And speaking of the user experience, right now, the output is an email. MyScout sends me an email digest of its discovery. ⁓ Is that what you think? Is that what you envision? What do you think the ideal experience might look like in the future?

Yeah, yeah. This is something that we brainstorm about a whole lot. That like, should this ⁓ delivery mechanism be? And the thing is, Scouts, it's such a horizontal product that it's unclear or it'll take a lot of thought to figure out. Because if you are looking for, if there's some information that's changing reasonably quickly, and if you care about it, so you want to know ASAP, I feel like we should just text you, right? Like email is not the right place. ⁓

looks like. Mm-hmm.

for that to happen. And on the other hand, if you have these very news-oriented specific area that you're interested in and you want to know the daily updates, that's much closer to a newsletter as opposed to a price alert versus a newsletter are two very different things. But Scouts can support both kinds of use cases. And so for this ⁓ digest-like thing, maybe email is more appropriate. And maybe multiple scouts, like if you have several scouts that are all news-like, maybe right now you will get one email per scout. But maybe it's better if all of the reports from all of those news-like scouts were put together in one digest that's ⁓ sent. ⁓ One thing that we've experimented with is this findings tab ⁓ on the web UI. So these scout reports get sent to you over email, but they're also populated on the web UI. So each scout has a page that has all of the reports that it has sent you. ⁓ And in that Web UI, there's a findings tab where you see all the reports of that day. So the last 24 hours, all of the reports show up as like these very easy to swipe through cards that you see through that are quite sort of mobile friendly. ⁓ And we found them to be a nice experience. So that's one ⁓ different interface that we've ⁓ expanded. We do not have a mobile app. So it's a web

Mmm. Do you have a mobile app yet? Not yet, It could be, okay.

Yeah, it's a web app that you can use it on the mobile as well, but we don't have a dedicated mobile app.

What does the name Yutori mean?

So utori, it's a Japanese word for the sense of well-being that you experience as a consequence of mental spaciousness. ⁓ So going back to early in the conversation, like if you don't have a gazillion things coming your way, you're not trying to context switch every few minutes. You have the space and time to focus on what's meaningful to you. Our goal is to deliver that feeling, that feeling of utori.

You're making me cry for this vision. like, yeah. I feel like, you know, we all need more space to live our lives. And then we are just occupied with so many thread of things every day. If I count, I probably do have millions of things, you know, and you probably have even more.

the Yeah, yeah, yeah. We all need the space to live our life and what that life means is different. It's different for different people. We all need the space to do more of the things that are meaningful.

Yeah. Who should use this product?

I want to say anyone, but I'm trying to be a little bit more.

That sounds like against YC or typical VC recommendations. You can't be making someone something for everybody.

Yeah, so I think the way we think of ⁓ our target audience is ⁓ busy professionals, ⁓ tech savvy, like people who are maybe using other AI tools in their workflows ⁓ already. Yeah, so busy professionals ⁓ who value being able to have more of this space kind of ⁓ people.

I think there's a possibility that you are for everyone. You know what? I ⁓ have a friend who's a retired CMO and then she helped Twitter go public. She had two IPOs ⁓ under her belt. So she's kind of semi-retired and she just, she's like, I asked her to bounce ideas. Do you, if you want, can make an intro. Maybe she can help you to bounce some ideas, how you do the positioning and messaging, which is her expertise.

Hmm.

Yeah. Maybe, maybe you'll say like, is for everybody. One of a kind, like one of those rare product that is truly for everybody. Like chat to PT, right? Chat to PT is for everybody.

Yeah, yeah, yeah, yeah, that be great. That would be great. and Yeah, yeah, yeah. do think that ⁓ a lot of people, like we have a certain target audience in mind. And so like when we make product decisions, we sort of use that as a way of prioritizing and all of those things. But if I take a step back and just this feeling that you mentioned, like I want more space for the things that I care about ⁓ is a thing that I think a large ⁓ percentage of people ⁓ probably feel in one way or the other. For some of those, technology can help, right? It's not, yeah, there might be other reasons for which you feel that way, where technology may not be able to make much of an end. But I think there is a fairly large slice of people for which technology can help with it.

I thought about more extreme measures like moving to like really suburb like third tier cities in China or you know, Mongolia or whatever. Just to start a different life. I don't know. That's extreme. Maybe I should hire scouts first.

Yeah, yeah. But I think even in that characterization, there's an assumption of what it looks like to have more space. There's an assumption that it's a slower life that is more space, that it's a rural area that is more space. But I think the thing that I like to push on is that I don't necessarily make that assumption. It might be that for you, making space means being an SF, being at a startup, working a whole bunch of hours.

Yes.

But being able to focus on the things that you want to focus on, right? Like even in the context of being an SF ad to start up, being a founder, and let's say you decide to, I don't know, spend 17 hours of your day working, right? Like you do you, but in those 17 hours, there are probably things that you do want to spend those 17 hours on. And even in the context of work, there are probably things that you would rather not be spending those 17 hours on. Then our job is to make that possible for you, it's easier for you. So whether this space looks like.

That happened. Yeah.

being in the suburbs and having a slower life or whether that space for you looks like 17 hours to focus on this one thing, whatever the case may be, we want to give you the tools to be able to do it.

Man, I love this conversation. feel like this is like also questioning my, know, existential ⁓ thinking about how I'm living my life. Yeah, these days. Okay. I have a traditional closing question. If you were to recommend one book to like a 20 year old and one to a 40 year old and maybe one to the 60 year old age group, what would it be and why? I mean, no pressure. You don't have to give me all three. One or two is great.

Thank you Okay. you

I'm happy to get exposure to more books and wisdom.

Yeah, yeah. ⁓ So I don't know if I have this ⁓ broken down by age groups, but I'll talk about two books and then we can sort of see if there's an intersection with age ⁓ in some way. So one is this book called Beginning of Infinity ⁓ by David Deutsch. And it's actually a book that I haven't read. My partner has read it.

Mm-hmm.

and he loved it. And whenever he's reading something that he really likes, he has this habit of telling me a whole bunch about it. And so it's this very, it's a very interesting experience. It's like secondhand reading, but with this filter of extreme enthusiasm, right? Because he is so excited about this book that I often I'm like, this might actually be a better experience than if I had read ⁓ the book myself. But so what it's about is this idea that all problems

Can read it? Yeah. you

are solvable. So problems are inevitable. All problems are solvable. Whenever you solve a problem, that's going to lead to new problems that are in turn solvable. So that's the thesis of the book. so it's sort of this very interesting, one way of interpreting it is that it's like extreme optimism. When you say that all problems are solvable, Sort of it's, yeah, it's this very enthusiastic optimism to the world while still talking about problems head on, right? It's not like a delusional way of being optimistic. It's like centering problems, but has this highly optimistic view to it. And it has this sort of very error correction way of looking at the world, right? That all statements that we believe are true are really just conjectures that we haven't falsified yet. So there is no such thing as Like this is true. It's just that this has not been proven to be false so far. Like even if you think about like scientific claims and things of nature. And you can apply to sort of problems in general. So I think it has this very, yeah, this very interesting perspective on the world that I think is beneficial.

Yeah. ⁓ very engineering mindset of moving forward.

Yeah, yeah, yeah, yeah. But I think as you go through it, can kind of see analogies even to like non-engineering problems. I feel like it's a useful perspective to have to just sort of the world and life in general.

Listening to you explaining it reminds me of your agentic loop. Keep looping!

Yeah. But so that's one. ⁓ And the second one is, and this might be one that ⁓ maybe a large part of the audience already knows, but this book called Scout Mindset, ⁓ where the idea is, ⁓ and actually no pun intended, but I just realized, but like no pun intended when it scouts the product. ⁓ But the idea here is that you can have

Yeah. Yeah.

Like the book talks about two different minds. This is Julia Galeff as the author. There's two different mindsets, scout mindset and soldier mindset. So in a soldier mindset, you're sort of on a mission. You're not questioning things along the way. Your goal is to sort of succeed and get there ⁓ sort of no matter what kind of a thing. And in a scout mindset, you are not on a mission. Your goal is to look at the landscape and come back with an accurate map of what reality is. Right. And so it's not. It's not that you're seeking a certain outcome, you're just seeking ground truth. You're just seeking what the reality is. And the book talks about how there are certain situations in which you need a soldier mindset. And there are certain situations in which you need a scout mindset. It also argues that most of us tend to have the soldier mindset even when we should have the scout mindset, where we really should be just seeking out what that reality is. And I think if you can figure out when to have the scout mindset and when to have the soldier mindset. ⁓

Mm-hmm.

And even just being aware of this difference, like recognizing in yourself when you're approaching this with like a, this is my goal and like no matter what, I'm just going to try and like, I just want to be right, right? Like in an argument, for example, I'm just going to keep pushing on whatever my agenda is versus when I should really have a scout mindset and just be like, what is the reality here? What is the right thing to do? I think recognizing that in yourself can be quite powerful.

Yeah.

So going back to age, don't know, maybe scout mindset for a 20 year old and maybe beginning of infinity for a 40 year old. But if you haven't read.

It's all age friendly. I felt this deep wisdom, right? The first thought that comes to my mind was parenting. Like if I'm yelling at my kids, am I being a soldier? ⁓ Well, I probably should hold a scout mindset. So recognizing these patterns. Debbie, this was one of my favorite conversations. Thank you so much.

Yeah.

You know, what strikes me most about this conversation is how much you've thought about, you know, not just the technology and thanks for doing the deep dive with me, with my naive approach, but what it means for people. And you're building agents to give people mental space back. And that's a very beautiful mission. Well, thank you for being so generous with your time and your thinking today.

Thank you. Thank you for having me. I really enjoyed this as well.

Thank you so much.