The Retrieval Loss Problem: Why Your AI Memory System Fails at Scale

You know, the thing that was really interesting is like everyone's trying to engineer context, but be predicted.

Can you tell me more about. This prediction,

you need

the right context for the right task.

How do you do that

If you're building a voice? Agent speed is really critical. The recommended speed for voice experiences is to respond back to you in four to 500 milliseconds.

If you want to personalize your voice experience and a rack pipeline, it's gonna take you rack pipeline, 500 milliseconds.

There's been recently like an explosion of like multiple agents working together. Then it's really like an exciting opportunity. One of the core problems is when you're stitching together these agents.

These agents need shared memory, but also you need to optimize the right set of context that you're giving for each agent. And with hallucination, it becomes a trickling effect.

A lot of people are, uh, talking about retrieval accuracy, and you're talking about retrieval loss. Can you explain a little bit more about what is that?

Hey everyone. Welcome back To Set ai. today I'm sitting down with Shaka [00:01:00] Kabar, CEO of paper. He is building what he calls is the predictive memory for ai.

Shaka, thanks for joining us.

Before we dive into your predictive memory, could you tell us a little bit about yourself and what you're building?

Yeah, so I am Michelle Ka. Worked really a lot on machine learning, AI at like big tech companies, Microsoft, apple, Facebook. we're building paper. Think of it as really the memory layer for AI agents. So if you're a developer and you're building an agent, your agent needs to remember like we remember, and that's what we've built.

And the unique thing that we do is we do predictive memory. Basically for agents.

Nice. That's something new. we'll dive into more about what this predictive memory means, but before we go into that, there are a lot of jargons in this space. So like contact engineering, memory systems, maybe retrieval

can you help me to de decode what some of these actually means? I found a really good code from your blog post and you said something like, most AI systems today rely on vector [00:02:00] search. For retrieval. things like rag systems , right?

And you said that it has a critical blind spot that it finds fragments and not context. Can you help me define what is this context

Yeah, that's a really good question.

So when you think of context, you know the easiest example I give is how we use context as humans. So when we are having a conversation, you probably wanna have context on who you're interviewing, for example, or I wanna have context on like the conversation that I'm having with a person.

It could be like personal context, it'd be context about the project. If we're working like in a team, in a company.

So context is king. Really, context is like so important at work. And as leaders, leading teams or in organizations, you need to really make sure that your team has the right context so that they're empowered to do the best work that they can do.

So now if you take an example of AI agents, when we're building agents, we need to give agents context then. There's really popular terms there that we like, which is called context engineering. It's actually started by [00:03:00] prompt engineering because initially when you have an LLE, you need a prompt. You need to like engineer the prompt so that you can get the right information or context to the agent.

but then this really evolved into a context engineering because. What we learned is you need to, normally, when you're stitching multiple tasks together, you need to laptop to engineer the context window to make sure it doesn't go beyond like the maximum limit to make sure that you're really optimizing the right context and you're retrieving the right information to help your agent do work.

But one of the things that we learned as we went through this journey is that the more data you give your agent to remember, the worse memory gets over time.

So this is like a term that we, that some coined as like context dropped. So a lot of times when you try to bring a lot of data and you give that to an agent, like thousands of or millions of tokens, it doesn't really work well.

And then a lot of times when you try to save a lot of data, like in the rack pipeline,

the more data

use for and save the worst retrieval gets over time. [00:04:00] we'll talk about this in a little bit, but that's basically one of the core problems that we've we're trying to tackle.

Does it have to do with this concept of loss in the middle problem? Because you the how the LMS works and you prefer Yeah,

yeah, exactly. This is what happens when you like, bring in like millions of let's say thousands of documents and give it into the context window of Gemini or any of these like models.

You have this problem where you have a needle in a haystack. It's like you put a needle in the middle and you like can't find a needle With more information, it becomes harder. But I'd say like when we work, we are not looking for like needles in information. Like we're looking for different snippets of information or context that we wanna synthesize, bring it from multiple sources and come back with an answer.

So actually real world queries are more sophisticated than like the needle in a haystack, like evaluation.

But I'd say one of the, the thing that was really interesting is like everyone's trying to engineer context, but we predict it. So rather than trying to engineer the context window, [00:05:00] we've built prediction layers or prediction models that help us predict the type of context that you're gonna need before you actually do it.

And we realize that this helps us. Overcome the challenges of context drop because with more data, our prediction models get better over time and then we can give you better results.

That's very interesting. I've never seen something like that on the market just yet,

before we go more detail into the predictions, can you define. Memory for ai. 'cause we are, a lot of people are talking about memory for ai.

the way that I think about it, we talked about context and how you engineer and hopefully create context in the context window, but when I think of memory, I really go back to like our brains and memory that we have because, the way you think about it, when we as people, when we see things, when we grow up and we learn, we accumulate a lot of information.

A lot of this information is stored in our brains as memory. Some of it is like working set memory, which is basically when you're trying to complete a task, there's a few [00:06:00] pieces of context or information that you need like that we put in your brain that you need to complete a task. Think of a simple to-do list.

This is like the working set memory. Then you have like short term memory, which normally is like episodic. In nature or it's like a form of memory that when you like go to sleep and you wake up, your brain tries to predict actually what information you need during the day and tries to optimize during the things that are important, like near.

And you have like long term memory. This is like an experience that you had a while back and you might remember it or you might not remember it, but. Yeah,if we use that as an inspiration, when I think of memory, it goes beyond like the context window and it's like everything that you put, it could be the chat conversations that you're having with your agent.

It could be all of the rag data or your rack pipeline,multimodal, like video documents, quotes, snippets that you store. It could be shared memory, so it's anything that you store that then you wanna bring into the context window so that your agent has context to do their tasks. So I think if memory is like a broader thing [00:07:00] that you store everything in that can help you then optimize it.

Like work with the context window.

the generic memory is how we as humans work is the total storage of everything that happens.

Exactly.

Information we have. How when we are working on something or making decisions, we don't use all of it. We use a selective part of the memories, we retrieve different parts of the memory that can help the current.

exactly. In that memory that you're talking about, What's inside of the context window that's what's important.

Exactly. You need the right context for the right task.

Right, Exactly. When you think about, your product and how you're selling it, do you feel you are selling a memory solution or a contact solution?

do you have a specific way

of seeing it? Yeah, a memory. Memory solution.

I'm curious, when you explain paper to your users, what makes. People immediately say, oh, I get it.

Yeah, So I think there are like two use cases. [00:08:00] The, two type of customers that we talk to very often that get excited about what we're building.

One is voice agents, and if you're building a voice agent. Speed is really critical and I experienced that when I was at Apple building, like sir, and voice experiences before. Actually the recommended speed for voice experiences for your agent end to end, to respond back to you in four to 500 milliseconds.

That's normally like the way that humans pause and how fast we speak. That's like the recommended experience. But the problem is if you want to personalize your voice experience, you really, if you put your data in a rack pipeline, it's gonna take your rack pipeline 500 more seconds, then you're gonna add a bunch of more to do speech to text and text to speech or even audio.

So that's one to two seconds and that's not. Like doable? Not acceptable.

Yeah,

not acceptable. Yeah. it just doesn't work.

So with paper, because we predict like the type of context that you need and we put that on device and we cache it. Mm-hmm. For voice agents, it becomes a seamless experience.

We can actually retrieve the right [00:09:00] context in less than a hundred milliseconds. That's gonna be the next launch. So we haven't launched that yet, but that's gonna be the next major launch that we do is bringing this context and like the predictions that we have on device.

Hmm.

So this really unlocks use cases that weren't possible before for voice.

so voice is really like critical and important, and people get that.

And the other set of customers is B2B. If you are building like an internal like knowledge agent inside of an enterprise company, that the agent needs access to a lot of maybe terabytes of data. Like all of the Slack conversations that you have, access to all of the documents, like a notion, for example, all of the meetings on Zoom, maybe your code.

imagine giving that agent 360 view of all of the information that, that matters to you to help you do your work. That's a lot of information and a lot of solutions today if you give it all of this information. Retrieval doesn't work. It like breaks. It's not reliable. it just breaks basically.

ours work really well at scale and with more data it gets better. So those are like the two cases.

That's an aha moment. Yeah. So basically if [00:10:00] in order to have good results, you need to manage the context window really well. And what you do is you reduce the amount of data, you put the best set of information.

For the context

exactly. Exactly.

you know right now on the market, there are a lot of people talking about different ways of engineering context, and you mentioned that

Engineer context is actually very brittle and it can be very expensive and it can be unsustainable. Can you explain that a little bit more?

Yeah, for sure. there's been recently like an explosion of like multiple agents working together to help you do things and it's really like an exciting opportunity.

To increase the throughput or output of agents to help you do work. One of the core problems is when you're stitching together these agents need shared memory, but also you need to optimize the right set of context that you're giving for each agent. So imagine you have an orchestrator and you have six or seven agents of the bottom and each of them are trying to do a task.

And let's say they're trying to build software, they wanna build like an app that you ask them to build. There are a [00:11:00] bunch of like challenges today.

But one of the core problems that, that, ends up happening, which is like more manual, is you need to, when you have all of the memory or all of the data, you wanna make sure you're retrieving the right information and you're then passing the right information to each agent.

Whether it's shared memories or memories that are only accessible for to specific agents. if you give a lot of data and you don't have a lot of retrieval, doesn't work really well, then you get into a hallucination problem. And with hallucination it becomes a trickling effect. If you have the agent, the main orchestrators, hallucinating, then all of the subagent end up hallucinating and that becomes like a contextual problem.

So it's like one of the bigger problems that we see is like how you manage hallucination. And there's like different techniques to help you do that. But I think the first step is getting retrieval, working well and reducing hallucinations with that, helps with that experience basically. And then there's like other things that people do to make sure that you're optimizing the type of prompt that you have that you're sending to each agent because.

Depending on the type of prompt that you [00:12:00] have, it could get your agent to do work really well or not do work well. So when you're engineering context, you're thinking about like the prompting and how you engineer that. You're also thinking about what you're retrieving and reducing hallucinations. And finally you're thinking about like cost, because your cost is gonna bloat if you pass in a million tokens and send that to each like agent.

the prompting engineering part, we still do prompt engineering. at the end of the day it is the heart and the For sure. you're thinking of this when you say like scalability of this type of systems, right?

You also created, I remember seeing something called retrieval loss and you were talking about Yeah.

So a lot of people are talking about retrieval accuracy, and you're talking about retrieval loss. Can you explain a little bit more about what is that?

Exactly. So basically retrieval loss, we built retrieval loss to really help us measure like a few core metrics at scale for memory. So one of them is accuracy. Are we able to retrieve the right relevant context? Two is speed. Can we retrieve the right context fast? And then three is cost. We don't wanna like [00:13:00] shove or add like millions of tokens into the context window 'cause that will like impact cost.

So if you look at these three core metrics, we put that into the bucket of retrieval loss. And with retrieval loss, the higher it gets, the worse retrieval is. The lower it gets, the better retrieval is.

What we discovered is for architectures that are search based, that means that when you put all of the memory into a database,whether it's vector or graph or like a regular database, and then you try to retrieve data, actually retrieval loss ends up getting worse with more memories.

So the more memories you add, the worse it gets

you're talking about by just purely increasing the context window.

Exactly, so a way to think about it is if you add like a hundred million tokens to memory, a billion tokens, 10 billion tokens, the more tokens you're adding and storing in memory.

If you try to retrieve it, if it's a search based like retrieval system or architecture, it ends up getting worse with time. And actually Google had this really very valuable research paper that talks about the physical limitations of dimensions of [00:14:00] your embedding. So for example, if you have an embedding that has a thousand dimensions.

there's a theoretical physical limit for accuracy for retrieval. So if you end up having like more than a million or a million and a half documents, let's say, stored in memory, you uh, post that your retrieval starts to become like unusable. So this is like the example of the physical limits that you end up hitting as you add more data to rack pipeline.

It gets worse over time.

One of the core things we discovered, which was really interesting is if we treat this as a prediction problem. The more data we get, the better our prediction models, the better than we our retrieval gets. what we discovered is actually retrieval stays flat or even gets lower with time, which means that with more data, we get better at connecting things together and giving you the right context over time.

The traditional approach, it's just top K using ranking, re-ranking methods. Finding what's most relevant. And that's still based on similarities in the [00:15:00] vector space.

you're saying you're doing some sort of prediction. Can you tell me more about How are you predicting.

Yeah, that's great. Can I share the architecture diagram that we have? Oh, yeah,

yeah. please. I would love to see it.

Okay.

Yeah. Awesome. So you're treating it as a machine learning problem basically?

We are, exactly. Yeah, exactly. what's

your objective then?

Yeah, the, the objective is to get the right context at the right time for the user. So the best case is we can predict the context you need before you ask the question.

How do you do that? and do you have to label your results such as is this relevant?

Is it not relevant? Are you doing some annotation?

So we use different like techniques for prediction. It's not only like one. and I can give you a few examples to make this like a lot more concrete, but like one simple example is if we build a good understanding of the goals for users and then the tasks, we can actually predict the next task.

And if we can predict the next task, maybe we can actually do a good job at predicting the context or the [00:16:00] questions. That you need to ask to help you complete that task. So that's one simple example of how you can think about this. So a lot of the models like that we have we don't train on like the customer data, but what we look is we look at the basically the, the questions and the answers and we try to build prediction models that help us get to the right answer quickly.

So just to quickly maybe walk you through, like the architecture that we have, it's really inspired by how our brain works. We actually have four years. So yeah, it's basically like a brain inspired. Architecture and predictive memory. So like in the human brain you have like the working set memory, which is in the prefrontal cortex.

You have the episodic memory, you have long term, and then you have neural pathways. That actually, neural pathways is really interesting because what happens is like when we are having this discussion, brain is like taking snippets of things that are important and storing it in your brain. But it's not only doing that, it's connecting like [00:17:00] neural pathways with this information.

that you just captured with other things that are relevant that it's gonna predict that you're gonna need in the future. So we do something like that. And if you think about the architecture, it's pretty simple. We have the application layer. Where you can have,AI agents, tools that need like intelligence or ai, AI embedded inside of them, hopefully in the future.

Robots so that they have memory and we can share their memory securely with them. but that's more like the application layer. And then we have SDKs that are on device. We have an MCP server. We actually also have a CLI tool. We have APIs and we have this durable execution flow that we can build in, like on device.

And our SDKs. if you're building an agent and you use like our SDKs on device. It has a, like an intelligence layer, and this is where we have a lot of the prediction models that we have. What happens in the background is like before you start work, we actually try to predict the type of questions that you need in the working set memory.

These could be goals [00:18:00] that we think you care about, tasks that we think you're gonna do in next, but also important context and memories that come from your like long-term memory that we think are gonna be important. That you've used before or like are things that you, we think you're gonna need.

And this becomes like really fast. So if you come and ask a question, like your agent will already know, like the goals of the tasks that you're gonna work on, and then can quickly get access to key memories that are important to help you complete a task.

Episodic memory is a little bit more interesting.

It expands on this notion, but it doesn't look at a filter of goals or like predicted goals. think of it as more of a cache or citation cache. So this is where if you've used context or a memory before and we've, the agent cited it in an answer and you frequently used it, maybe you're gonna need it later.

So we try to build this cache tier or prediction tier of key memories that we think you're gonna need based on the things that you've done in the past.

So you're predicting the cache as well,

Exactly, exactly. it's like a cash based prediction, [00:19:00] but it uses not only cache hits, but also. More important signals like user feedback. If you came back and you said, this is a good answer, or a bad answer, or even a citation. So if the agent came and picked one of the memories and cited it in an answer, the agent can automatically give us feedback and say, Hey, we cited this memory.

So we use that feedback loop to then increase like the citation. if that makes sense.

do you take feedback from the user in a live conversation? when we're chatting, when you are talking, you are sparkling my memories all the time.

You reminded me of building machine learning models, Things like that. when I'm responding to you, you are also getting feedback. Exactly, you tracking these conversation turns and getting feedbacks on the fly.

Yeah, exactly. we don't build models on the fly, but it's a good idea. But we do get that feedback right, because when you come back and. And,send your message back, like as we're talking, we're able to capture that. And then if that's stored in [00:20:00] memory, we get that feedback and we have other signals also, like if you give a thumbs up or, imagine like other signals if you're building an agent, you might wanna get feedback from your users.

We give you the ability as a developer to give us feedback and a feedback API as well. So we do get that feedback and that helps us improve the prediction models that we have as well.

Sounds like you're predicting my intent.

meaning that, after we talk about one topic of on a few turns, and you probably gonna know what I'm about to ask next, you probably know exactly right. Do you do that?

We do that as well. Yeah. we do that for the purpose of trying to get the right context to help you answer the question or complete the task.

because every time when you're answering my question, you're retrieving different memories, different data in your brain to answer my next question exactly. Oh, awesome. I,

yeah. I really love how you're taking the brain and inspiring me to even build a better product. I think just,

I think we want more people to use this brain inspired architecture and [00:21:00] predict predictive architecture.

'cause I think this is really how we're gonna be able to scale, with memory and with ai. Yeah.

The last point that I just wanted to mention, is we also have a graph. We are using Neo four J and what's really cool is we give developers the ability to pass in your own ontology, your custom ontology.

We can then, when we add the memory, we add it to these tiers, but then we also build the graph ontology using the ontology that, that you gave us. And then when a query comes back, if it doesn't hit any of these tiers. We do an agent to graph search and then can give you back, for example, like for example, list of tasks that were in a meeting or if one example question is show me all of the tasks that my customers or feedback that my customers gave on key problems across my a hundred customer conversations.

you can do that with one shot, with a graph, you build the graph and you can get all of the insights or the problems across all customers. So it's pretty cool. And then we, these are like the tiers and then we use [00:22:00] like a storage backend. So we use MongoDB Quadrant and Neo four J and that's like the core storage that we have quadrant we think is the fastest.

And we love it.it helps us be very fast and Neo four J is really good 'cause it helps us build like the launch.

So a new four J is for more relationship and connection building, the graph architecture has a lot of its own merit and the developer can choose to give you my own ontology, right?

My own graph.

Exactly. Yeah. You do

build graph like automatically. Maybe for some of the chat we

automatically Yeah, exactly. we, as a developer, it's really easy. You just give us the, an apology. Your like schema, right? And then when you set up the schema, when you add the memory, we automatically add it to all the tiers and we automatically take the unstructured context.

Let's say if it was a meeting recording, we automatically structure it based on theology that you gave us. And save it to the graph. and then when you wanna retrieve, you can just use natural language and we can find that context in these [00:23:00] tiers. And if it hits the graph, we use the graph search to go and get the right context and get you the result basically.

how's your agent Just make the decision to go, which database? To retrieve the information, either graph or the vector database.

Yeah, so the first step is tiers. so we have a model that predicts tiers. and then basically if it's, if it's the vector one, it hits, for example, quadrant.

And then we also do a lot of chunking and we do query expansion and we use MongoDB for that. And Mongo was great for that. and then we also, when we store data in the graph. For example, in a meeting you might have customer, you might have people, you might have like tasks, you might have insights from a customer, as an example, we actually take all of these store your ontology in the graph, but we also store it as classes in MongoDB, which is pretty cool.

And then basically if hits the agent to graph it, it can go to the graph and get the right context.

do you somehow take user feedback as a signal to. Guide you which route to go.

[00:24:00] Exactly. exactly. So you're doing that as well. So you're having a lot of prediction in your system. We do.

And if you notice here we have eight, like eight, eight prediction models. So we're doing like a bunch of things to stitch this together.

Yeah. That's crazy. But it's makes sense

you mention about when you up in the morning. You might be predicting what you're gonna do and I realize maybe I'm unconscious doing that, just I'm just not aware. Yeah.

I think it's very funny. Today I was reading this research paper on like neuroscience and one of the things that was really interesting that I didn't realize is like when we sleep and we dream.

Our brain is actually creating different, like scenario simulations. I mean depending, there's like different types of dreams, but there are this case where like your brain is like putting future pathways and dreams and trying to predict the future and sometimes it does a good job at predicting your future, but then it uses that dream state to then help optimize what information is needed when you pick up.

So

there's some

[00:25:00] research on that, which I think is really fascinating.

It is fascinating. Maybe part of dreaming is also a realization, a representation of the graph in our brain for the, all the past memories and maybe the memory of the day, and then we're just representing it in a vivid, dream.

Yeah.

In a vivid state. Yeah.

well, there are so many things in our brain how it works that can guide how we design our architecture. I think what you're doing is really interesting. What do you think of the, how Do we balance, the short-term memories and the long-term memories?

Like the retrieval or, or you think prediction can solve all of it.

Yeah, I think prediction can solve most of it. if you notice we have like edge about, so edge is tier zero and tier one, and this is where we have like a smaller working set memory and then episodic memory.

So think of these being like, like short term. And then long-term memory is everything that you put in the cloud, which could be like vector or graph based. So if we can predict the tiers, we can predict the information in the tiers, then we can [00:26:00] try over time with more data. We can improve or optimize what we are putting in these.

Do you think this is how our brains are function? Behind the scene.

I think some of it is inspired by that. Yeah, for sure. we need to build more knowledge about ourselves. I think we need to learn more about how we work, so I think, I'm sure there's like more ways to improve this, but it's really inspired by how we think and how our brain works.

one side of me think that we're efficient In terms of me memorizing and forgetting. and some part of me, I don't know if we're actually that efficient, right?

when I think of long-term memory, I think there's is a component that the modern transformer architecture is not thinking about is emotion.

Because a lot of the times, what triggers my long-term memory is emotion, like a smell of something. And a familiarity like deja vu brings me some old memories, and that's not something that I can imagine. Machine learning or these things, these design architecture can handle, What do you [00:27:00] think?

Yeah, it's a good question. Yeah,I'm not sure. I don't know. I think emotions are really hard. you can,even in the latest like video models, like they look really good, but you still don't feel the emotion coming from the person that's talking. You can kind of feel it and it's really hard.

It's not something that you touch, something that you feel, So it's a little bit different. But I do think like the point that you made is interesting and it is like, what? What triggers really like neural pathways to connect together to get you to remember something. When you're putting tons of things, like for us, emotion becomes something that's important because our body believes that It's like something important to remember.

Whether it's something that's happened in the past, then it was like emotional. It ends up like being triggered really easily. But for when we are building like, agents, what are the right triggers? What are the right neural pathways that you wanna build connections with? What's the dopamine version of basically like the connections here.

if we can predict things that are important for you in the future, maybe that can act like a dopamine [00:28:00] hit and can help you build things together or connect things together to then make it easy to rie, make it the right context for task ask.

I feel the way you are thinking is philosophically one step ahead of.

If you think about the attention type of models, What drive attention is based on the end result of whether it's translation or whatever it is, you're building connection between A to B and then because you already have the end result. That's how we build the large language models, right?

right?

in a sense, we are still focusing on goal oriented building.

I think emotion is another level. maybe it is. Yeah,I'm looking forward to new architecture that can actually factor in how that triggers the neuro pathway of forming and how we forget and remember things and then make it more. I think maybe that's when the true, consciousness is happening in ai.

What do you think?

Yeah, it's very philosophical. I like this discussion. it's getting me thinking about like how we can even further, add more value [00:29:00] into this architecture. I really like that the emotion piece. And I can imagine, for example, in the future when we are working with robots, hopefully we can work with robots safely.

and we can safely share with consent, like our memories with them so we can collaborate with them. But I can imagine like an experience where if like you have a really great experience with a robot and their emotion gets triggered positively. In the future, when you come and they see you, they, because you triggered an emotion for them, right?

It's easier for them to remember you versus someone that they just saw randomly walking. So I think there's values in emotion for sure. I like that. Yeah.

Yeah, Once we can quantify it and measure that exactly the way we're realizing it is not solving you know how we are made.

Yeah. I wanna ask a little bit about your market positioning, and your go to target. my understanding is that your target audience is developers. do you target PMs as well? You mentioned B2B?

Yeah, it's a good question. Like we are targeting like developers and vibe coders who are building.

[00:30:00] agents. So I think in my mind there's two buckets of people.

One is developer and you're building like let's say the next cursor or you're working in cursor, you can come talk to us. It would be like, one example or if you're building a voice agent, so your developer inside of your coding rather than you going and trying to stitch together and build direct pipeline.

You can easily with one line of code use us. we are also, announcing it here that we're gonna go open source. So excited to share the architecture diagram with you guys and we're planning to open source our code retrieval. So that's one bucket. I think the other piece that's really interesting is like we're seeing a lot of users that are also vibe building apps.

You don't really need to be a developer. You can be a PM and we're seeing a lot of generalists. Basically so you can do anything right now with ai. So if you want to build an agent, even if you don't know how to code, you should use us because we have an MCP server.

Mm-hmm.

You can

actually easily, if you create your agent, you can easily use our MCP server and bring, add context and take it everywhere.

so that's like another good [00:31:00] example. If you're using like any of these tools to do a five coding, you can use us. And then for me personally, I'm using paper as an end user. So I have like over 22,000 memories. All of my customer conversations are recorded. All of you know our documents are code.

can you explain to. Let's say whoever have heard about this concept about MCP, but don't know exactly what it is. Can you explain in a layman term,

That's a really good

question. MCP stands for model context protocol. And the simple way that I like to describe it is it's helps your, it helps you connect with apps. So if you have some data in, like tasks in linear, if and you want to get that context, you can maybe use the linear MCP server to help.

you connect with these apps. So a simpler way to think about it is it's like a technology or a framework that helps you connect with other apps and get context from these apps.

So basically getting data or, you [00:32:00] can pass your data to some other apps.

Exactly you can do that or you can complete actions.

So one example is if you want your agent to mark a task done in your task management tool, you can connect an MCP server and then your agent will learn how to add a task in your task management tool. So at gets data, it can get data, but can also do things in the app. Like on, on your behalf with your consent.

cause you said you're selling memory solutions, when you're marketing your product, Do you feel this concept of memory, is still new to most of your potential users? Users? Is this still a new idea? Yeah.

I think it's a new concept, but it's easy for people to understand.

Like for developers who are building agents, they quickly get it because they recognize their agent. Agents need to remember and they need to get the right context. And if you're an end user and if you've used cloud or any of chat, GP, any of the tools, it's part clear that you need to get your agent to remember, your like AI to remember, [00:33:00] especially at the beginning, like the first year.

That was like a big pain point. So I think it's easy to explain to people easier than like context engineering or some of these other terms. But I do like the context engineering term because it's a good term for the problem that it's focused on, which is like helping you get the right context,

right.

Do things right. when you're talking to your user, do you feel you're focusing more on explaining what memory is? Or you're trying to explainwhy paper? It's more why,

why paper? Yeah. It's more like I get what memory is.

Okay, tell me why this is better, for example, or what are you doing that's different?

and you have some competitors. I saw the LAMA index and linein, they have land graph that sounds similar to some of your features,

Yeah, actually.

when you think of like LAMA Index and land graph, we see them more as partners because they're like an orchestration framework. We can actually plug into any of these orchestration frameworks and be like the state or the memory layer. They also have their own memory solutions as well.

like [00:34:00] we can plug into their systems. I think that there's a few people that are focused on memory. I think MEM Zero is one. Zap is another one. Those are like memory focused platforms. or basically.

So you're more on par with those guys.

Exactly. Yeah. It's the same way to think about it.

we're, our architecture is different 'cause we are building predictive memory. but yeah, we're in the same space.

let's say today I wanna build an AI application. What do you think should be in my AI stack?,?

obviously I could include you, but what do you think of as the minimum viable version of it?

I'd say three things. One is you need some orchestration layer and I think one key product that's pretty hot right now, I think it's temporal.

And so I'd recommend like that you look into temporal and see if it can help you drive durable execution. 'cause that's really important. What it does is it helps you guarantee that when you're orchestrating, and giving a task out to a subagent, that subagent always does the work that you're asking them to do.

There's failures. There's like state where something might break and you want to come back and retrieve that state. [00:35:00] So I think one thing to think about is like when you're orchestrating, how do you make sure that you have durable execution? temporal is one example, and we can plug into temporal.

So you could have paper be the memory layer and temporal will be the layer that guarantees your task execution. the second piece beyond orchestration is you need like to choose the LLM layer. So are you using like, you know, chat, g PT, cloud, and tropic? You wanna build an architecture in a way that gives you the ability to use these differently.

You might have different use cases that are optimized for different lms and sometimes you wanna switch and swap these, so you want, you wanna make it really easily swappable.

And then you have the memory layer and the memory layer. You wanna make sure that layer is continuous. So regardless of how many agents you've spin.

Or different LLMs you choose, you wanna make sure that your memory is continuous and you can use the same context and pass it to any of the other, like LLMs or agents. So I'd say these would be like the three core layers. Orchestration, the l LMS that you're using, and then basically the memory layer, which [00:36:00] includes storage.

And database. What about database?

Yeah, I bucket this like really under like memory. 'cause it's includes storage and database basically. Yeah.

Yeah. Okay. Cool. and when you are pitching, how do you make people remember Paper?

I show them a demo. Should I show you a demo?

why not? Yeah.

Okay.

Let's see if this is gonna work. let's see.

so I have a few demos actually that we can go walk through. We've built a few apps.

Actually, one of the apps that we built is this chat app and it's open source. So if you're building, if you wanna get a feel for how memory works, you can try this. You can also like easily put this in versal. And with one button and you can easily give it a try and see how memory works.

But over here you have a typical chat experience. You can add context. These are previous artifacts that you've created that you can add as context, but then you have MAM enabled. You can enable or disable search and you can pick any of the LLM models to try. I'm just using Chachi PT here. And one of the cool things that we did recently with this, and I want to give it a try here, is like audio.[00:37:00]

One of the problems with audio, that we talked about previously is it takes a lot of time to get or retrieve context. So I'm gonna try this right now with you and see if this is gonna work. Can you, I'm assuming Angelina, that you can hear, but what I'm interested in is help me understand what Paper's predictive memory is and why it can scale unlike other architectures.

Okay, I'm just gonna pause it here. Notice how fast that was in smooth. We automatically bring context and it has information. I wanted to give you an, maybe another quick demo here. So this is we talked about retrieval loss. I'm gonna use Opus right now for this example, and I'm just gonna ask a question of show me like the retrieval loss.

And what it's doing now is it's doing some reasoning to go through and do searches. And it'll come back with, in a few seconds,some more information from memory. Here we go. So this came back and it's showing like some more information. So we'll go through this, wait for it to finish, and then I can quickly go through this.

I have another question about the graph part of the tiers, if I don't have my own ontology, you'll automatically [00:38:00] generate that

or, yeah, we do. Is it

something like the user turns on and off?

Yeah, I think the user can basically, when you set up, like the, when you use the SDK and when you set it up, what happens is you can set the schema that you want for the tenant that you want, and then when you do that,we always use that sche.

When we build the graph for you or when we build the ontology.

Makes sense? Yeah.

Yeah. So anyways, I'm not gonna go through all of this, but it gives you a kind of an example of it's bringing all of the context, putting all of the sources, episodic memory, what did you say?

so this looks like the episodic memory that you're, the memory, yeah.

I have so many things here, so it grabs a lot of it and it puts it here in context. Lemme try to see how I can go back to the screen here. Okay. Yeah. I can maybe stop sharing.

No, and that's okay. Yeah. the one you just showed me is that retrieval from your 20,000 like exactly.

Okay. That's why I look so many.

Yeah. Cool. do you think you're in a very crowded space?

[00:39:00] Well,I'd say that it depends how you define the space. if you define the space as ai, it's super crowded. But if you define the space as predictive memory, we're the only ones there.

But I think, yeah.

which one is your space? I think it's either

Yeah. we are in the AI space, but we are. Building this predictive memory and we're the only ones there. But obviously there's other solutions. you can go and you can build your own rack pipeline. you can go and use a solution out there.

So there, there's like multiple choices for a developer. For sure.

What are other choices for developers?

Yeah. I say that they're like three choices.

One is you can build your own rack pipeline as a developer. Like you can choose to go and build your own vector, embedding, build your own graph, try to build predictive memory if you think this is the right architecture.

So that's one option, is you can build it yourself. obviously if it's, so it's select one and

then two is you can basically use like third party solution. So we would be, an example would be like, use M zero Zap or Paper or any of the other solutions. There's Cockney as well and a bunch of others there.

So those are like, the two [00:40:00] options is you can either build it yourself or you can choose to like, use an existing memory layer or memory solution.

What's the third one?

I think the third one is you can choose not to do either, you can pick up Gemini and you can say, okay, I'm not gonna worry about taxing anything.

I'm just gonna put everything into the context window. And I'd say it depends on the use case. if you are building something that. Has like very small amount of data and you don't need to have like personalization, for the user over time. You can, if think about if it's a, like multiple paragraphs or whatever, you can just put it in the context window and that works you don't need to build like a sophisticated memory,rack solution

But if you're

building a voice agent and you need to personalize your voice agent, you need to do something there. Or if you're building like a B2B agent that needs a lot of information, from within the company. You need like a memory layer there.

so Hear me out?

I have a sense of what is your true competitor? if you say you're predicting memory, then your new category. 'cause you're the only one if your [00:41:00] AI is just too way too broad So what about the positioning is more about better rack systems?

Like you're competing with people who wanna build a better rack.

Yeah.

Does that sound about right?

Yeah. I think that sounds right. I wouldn't say we're competing, with people building better rack systems, but we are like, if you wanna build a rack system, you can use us and you can build the best rack system ever.

Okay. Okay. That's the tagline, right? that's the pain point, I think. Exactly. Okay. I really enjoyed the conversation with you. learned a lot from you, and I think I really like the idea of using predictive modeling and machine learning in this architecture.

and I think it also make sense, and I realized when you were talking, I realized I am always kind of predicting, not predicting. I'm just anticipating I could anticipate what you're gonna say

Yeah, makes a lot of sense. I also learned a lot from this, so really appreciated this discussion and it was really great talking to you.

Yeah. Thank you so much for joining me today.

Sounds good. See you next time.