The One Skill That Survives The AI Shift

You wrote the book while people have been saying that RAG is dead right for the past two or three years

There's been like RAG is dead claims every two months people mostly trying to market something new but I would say if you think about RAG it's three components is retrieval augmented generation and retrieval is still extremely important even with agents there was a time when people did like put a one PDF or two PDFs with RAG connected to an LLM for generation and it felt magical look mom I can talk to my PDF. But that is very different than millions of documents, all kinds of formats. Doing at the scale is very very different. Imagine a training video. I'm the instructor and I'm saying, "Okay, so in order to avoid catastrophe, never press this button. If you just transcribe it, you lose the meaning completely." It's called like the red button problem. You want to have visual understanding of what's in the image and or in the sequence of images in the video as well as correlated with the audio in order to get the complete meaning. What I'm concerned about is as a society what we do with this. So I think there's a way where this is actually benefiting everyone.

But in the meantime there could be a lot of pain. We're going to end up in engineering and data science being directors as opposed to actors. The coding agents will write the code. And I think what remains a really critical challenge is to understand what we need to build and guide it in the places where the coding agents are not doing good job. A bunch of high schoolers they looking at me and say like did we make the most incredibly stupidest mistake ever because they all got accepted to their computer science program. I said no you can graduate from college and have the capability of a senior engineer today. Use the next four years to learn how to be that.

I have Ofer Mendelevitch here today. He's a person who can teach us how to actually build RAG systems that work in production and he also recently published an O'Reilly book on exactly this topic. Welcome, Ofer.

Thanks Angelina. Great to be here.

I don't want to give our audience a LinkedIn intro about you. How about you share with us something about yourself that's not on LinkedIn.

Oh, what's not on LinkedIn? And I've seen a lot of cycles. I've started using what we now call AI, data science, machine learning is what it used to be called back in the day, a long time ago. One fun fact is when I started this, I transitioned from engineering to data science. This might sound really weird, but the best most successful algorithm for doing any type of prediction was naive Bayes. If anybody's heard of that, it's a very basic, very simple algorithm, but it was really hard to do. In those days you know CPUs there were no GPUs at the time so I've been in this space for a while

Yeah I know I wonder if people are still using it I naive Bayes

I don't think so I think it's it's not as strong as some of the things so we've evolved from naive Bayes type algorithms to the era of the XGBoost and you know those kinds of algorithm decision trees and now today we live in the in the world of you know LLMs that we all like go crazy over. I got lucky enough to bump into the world of LLMs quite early on. So I used GPT-2 actually GPT-1 for maybe a month or two until it became GPT-2 was released back in 2019 I think it was something like that and so I got I got the chance to use it quite early on before it became ChatGPT that we all know and care about today.

Yeah, 2019 doesn't sound, you know, too far away, you know, but it felt like a decade.

Yeah. And think about it. It's like pre-pandemic first of all, right? Or just before. But yeah, it's so long because the world especially the world of technology has transformed so much in the last I mean since ChatGPT came out and like changed everything is now using LLMs and agents and RAG and all that stuff. So yeah, it's changing so rapidly.

Let's talk about your book. Your book's name is Hands-On RAG for Production. I read the whole book yesterday.

Thank you.

I want to ask why did you write this book? I think you wrote the book. Well, people have been saying that RAG is dead, right, for the past two or three years.

Yeah. So, when I started, it was about a year and a half, a little bit more than a year and a half ago when I started writing this book. And at that point in time, actually, people didn't quite say RAG's dead so much. It was it was pretty much the center of RAG. Everybody was doing RAG. Everybody was experimenting with RAG. And at the time, you know, at Vectara, we were like a RAG platform. We're in the middle of all of this storm of RAG. And essentially, I've learned so much by helping our customers implement our system. And also looking at the open source community. There's a lot of other options to do that like in LangChain and LlamaIndex and a bunch of other open source projects. And I've learned so much about anywhere from ingest to querying, getting accurate results, reducing hallucinations, things like that I think the idea was to really try to distill all this knowledge in a way that's helpful for people. Especially because a lot of the stuff was just prototypes just initial demos and then it is very helpful to build system with this.

But how do you when you scale them to production a lot of other things starts to break and the demo is really simple. There was a time when people did like you know it was like hey I put a one PDF or two PDFs with RAG connected to an LLM for generation and kind of look mom my hands are free I can talk to my PDF right that was the typical demo that people had and it was really fun it felt so powerful you could like have a long PDF with hundreds of pages and just ask questions and get answers grounded in that PDF it felt magical a lot of people but that is very different than you know millions of documents all kinds of formats, powerpoints, documents, different sources. Doing it the scale is very very different and that was the goal of the book is to really convey everything that I know and I've learned over those years in that space.

How about how about we do this? I have a simple use case. I actually have this use case. Maybe we can work together to build a conceptual RAG system.

Yeah, I love it. Let's do it.

Let me share with you my use case. I simply want to verify you know LLM outputs are real or not. So I want to reduce hallucination in any AI outputs. It could be a research report or a blog post right. So what happens is I typically generate references with it. Claude does, ChatGPT, they all have a version of it. But when I do a manual check right we often find that it's partially correct. There are a lot of issues. This process is very tedious and I want to set up a system that actually automate this whole thing and I have three goals. I need to get evidence from the source material. I need to know whether those evidence actually prove my claim is correct. I need to verify the numbers. So any any numbers and statistics quoted in the claim, right?

They need to be correct. If the source material says something like 90% of the POCs are failed in production and in my in my generated result it says let's say 85%. That's a mismatch but RAG system might actually turn out to be you know saying this is a match right because most of the words in this sentence are the same. Can you help me to think of way to set us up correctly?

Fantastic fantastic use.

So what I'll say is let let's go through it step by step but what I'll promise to do is sort of explain or try to explain here how to solve these challenges not all of them necessarily are solved by RAG in the traditional sense but I would say this up front and this is part of the book too is we make the connection between RAG and agents which are what's really common these days and one one thing I could sort of present is that you know you mentioned earlier RAG is dead there's been like RAG is dead claim since the last two years every every two months people mostly trying to market something new which is fine but I would say if you think about RAG it's three components is retrieval augmented generation and the R-A-G, and so the R is for retrieval and retrieval is still extremely important even with agents a lot of the book talks about how to set up proper retrieval generation the difference is that generation and in RAG is just a one shot you ask a of the information and you get a response back and then you send it to the user whereas agent has a loop in there and can ask questions again and again and the reason I'm mentioning this is because in the use case you mentioned right you could for example look at the whole document and ask a question of RAG to evaluate all the stuff that you mentioned the three requirements you could also do it do step one is give me all the things that might need review as a list and then go through the list in the agent step by step and review each of them separately right for validation.

So we'll talk about this but let's start with the requirements. So the input here is a document. Let's say it's a PDF for simplicity. And you have some some source documents that constitute the source of truth for you, right? So is that 10 documents or 100 documents or a million documents, right? So they exist somewhere and then so RAG is all about grounding in some source of truth that you decide, right? That you set up. And so let's assume that it's there. And I'll assume it's let's say a 100 documents for now, but we can talk later about like again going to production. What happens when it's a million documents? It becomes much more complicated. But the first step in RAG is of course you take these documents and you they call ingest. You ingest them into the RAG system. The reason you have to do something with them is because you want them to be ready for retrieval for accurate retrieval.

And there's something called semantic search which is people know as the vector database kind of a mechanism which finds semantically similar information in those documents. And I'm I'm explaining this in detail. Many people know this just to emphasize to your point Angelina if you look for 85 and 95 differences they might not show up in semantic search. It's not really a thing. So either you find if you say like the number of people failing production is if you if you have that word of number of people failing something if that text exists somewhere in the document and it's correct then you'll pull that information you'll be able to correct the number but if it doesn't say it in those words and it just is the number mentioned somewhere it doesn't exist you will not be able to find it. So the ability to get the right information during retrieval is extremely extremely important, right? And so I'll talk in a minute about three different ways to do that or a couple different ways to do that are helping the raw RAG.

Let's get back to your question. So first you ingest your 100 documents. And to do that you go through the regular RAG pipeline steps. You extract the text from the documents. You do what's called chunking which takes the text. It could be 100 pages. So you split it into smaller subsentences or paragraphs and then you vectorize them which means you embed them sorry encode them with an embedding model and you store them in a some sort of a vector database and then they're ready to retrieve. Of course you store not just the vector you store the text itself because you need to retrieve it at some point. And then there's something sometimes people call hybrid search that we talk about too in the book which is the combination.

So semantic search is the new thing with the vector database but we all all had TF-IDF and BM25 you know the traditional search algorithms and so hybrid search is combining those two and sometimes that's important when you have when you look for some things that does don't have a semantic meaning like if I look for a certain product SKU for example doesn't mean anything it's just a number or the word iPhone or something it's just one word so yeah so first I think you do that does that make sense so far in terms of what you want to do for your use case.

I take the reference out and then I will retrieve the whole document as input document and I'll embed all of those and in the meantime you we want to know what we're verifying this one single article I need to make sure it's correct right so I will look for maybe the reference point like the where maybe two to three sentences right before the reference number so that's that's the statement I'm trying to verify

So during ingest you actually want to when you break it into chunks you want to mark each chunk where is it which page and which part of the page so that later on in the query we can sort of reference back to show where it was at 100%.

So we have our in input data ready now.

Yes. So now when you build a RAG pipeline essentially and actually let me use a maybe I can use a diagram here I think will be

You have a diagram.

Can we see this? Yeah. Okay. Is it big enough? Okay. So what you see here is kind of the flow we just talked about taking data on the left side here and moving it through through to the vector store right here right so you extract text you do the chunking I didn't mention table processing right which is sometimes your PDF might have a table or even an image right so if you want to answer questions based on that for example your numbers might be not stated in a text but you might see a diagram says statistics about failure of RAG implementations and it could be like some some five numbers from different customers.

So you want to be able to understand that information as well and this is called multimodal data in RAG and then you encode it in embedding model and you store it in the vector database. Now during query time, so now comes the time when you ask a question and about the whole document. Let's start with that simple case. You give it the whole document and say please review this whole document and tell me you know what things I need to fix, right? What are things and what claims are cited correctly. So this is where this becomes complicated. But you send a query in the query gets embedded too and it starts it starts it looks to find the most similar parts of the text out of the vector store. So if your if your query has a lot of interesting pieces to it including I want the citations to be accurate and I want every piece to be correct.

My guess is it probably finds some of these things here but it's only based on this one match right and that's why I said earlier I think the better implementation of this is with agents so let's talk about this in a minute. But at a high level agents can then break down your query that can read the document the agent can read the document first the document you want to validate break it down into different queries and run five or six or 100 of these queries against the vector store and get the results. But just to finish up this story, right? So imagine that there's that you just want to do one one query.

So you get the top let's say five or 10 results that are relevant for the query and that can be just a vector search or it could be like I said hybrid search with BM25 or TF-IDF or something and there's sometimes a step called reranking which helps get even more accurate results and then you take a prompt and you say to the LLM which sits here you say hey I have this user query and I have this document and here's some results that might be relevant to help the user can answer the user query based on this document and based on these results and then the LLM generates the results and send this back to you. But you can see Angelina here how I have a feeling that if the if the number of things is more than three and the document is pretty long this probably doesn't work in it in its current form in its one-shot form, right?

So what I wanted to show next is like how agents work and there's a chapter on agents which exactly tackles this problem which is agents are you know this is a sort of a rough diagram of how works but agents have these things called tools right and so in this case you would and they're controlled by an LLM so there's an LLM here which is part of the agent and it sort of reasons about your query plans what to do and then executes it by calling one or more tools and so the tools could be calling an API running a query in the database but a lot of times it's actually calling the RAG pipeline that we've seen just before right so it calls or a retrieval part of the RAG pipeline that we've seen before right so you can you can call RAG in this case maybe you have 10 different parts of the document that you kind of want to verify the LLM should find all these parts on its own you don't have to tell what it is and it will probably call the tool 10 times.

Each time it'll ask is this statement correct? And it'll give the results this correct and it'll take all this information back and review it together and give you a formal answer about those that are correct, what are the citations and again the RAG tool will return that the citations correctly and then those that are incorrect will highlight those and ask you to sort of figure out how to fix them. Or suggest a fix if it exists in your document set.

I can see why the agent might work better here because the verification process can be very nuanced.

Yeah, the whole the whole idea of the basic RAG is this strict like a matching of a query to a set of documents. And in this case, the query you're going to send outside is like I want to fix all the problems, right? That in itself wouldn't match any particular document part, right? So it, you have to have either run it as multiple queries which is possible too or that's what the agent will do for you essentially automatically. So that's why I kind of jumped there.

The RAG similarity kind of matching is always not performing well on like exact matches or like numbers. Temporal data like you know time dimension is also really a complex complex data complex questions is also not going to perform very well and

Multihop questions for example. So exactly. Yeah. So that's a good point and I wanted to actually let's talk about this a little bit because I think it's important and it's a big part of the book too. Right. So

Give me an example.

So first of all so I gave you an example of let's start from beginning. So vector search which is where everybody starts when they learn about RAG is I just want to give it enough enough focus right it is important it's very powerful it does this semantic search in multiple languages too right you can ask a question in English and get get a match for a document in Korean or something right or Spanish or whatever so there's a lot of power to it but it is how the name sounds it's semantic search so it looks for semantic similarity now if you if I look for the word iPhone, I probably want to get iPhone, not necessarily Android.

I want to get like or if I if I look for a certain code or something, there's no semantic meaning per se for this. So, this is where you want to add hybrid search or other more traditional search. But I want to talk about two other things that I kind of mentioned in passing and I think are important. One is the idea of multimodal search. Okay. And again it's relevant for RAG and it's relevant for agents which are an extension of RAG. So when you do retrieval on multimodal data right it could be tables in a document it could be images in a document. It can also be audio and video.

And so video could be like to give you an example, it could be a lot of companies, enterprises have huge amounts of maybe recorded meetings or training material, right? And you want to be able to search on that. And so in order to do that and integrate it into RAG, you have to do some extra work. And again, if it's in production, you have, you know, hundreds of thousands of those, it becomes even more maybe taxing from a performance perspective. But generally there is there is ways to look at I think tables are probably the most prominent and to give you a sense why that can become problematic because tables could sometimes be reversed like this.

I've seen this a lot in medical journals or there could be multiple page tables and the traditional way that you process text is called chunking. We mentioned earlier essentially breaking it down into text. But you can imagine a text that has column names at the top and 100 rows. So the first chunk will have column names and five rows and then the second chunk will not have column names. So now the LLM doesn't know what it means. So you got to like treat the table as a first class citizen and have it been treated properly. So the best way I know of to do this is to identify the table as a whole table and store it separately from the chunk. Just store it as a table and then in query time you actually retrieve the table itself and you give it to the LLM as a full table and then it can figure out whatever it wants from that. And the second thing is images. Again, images could be really really powerful as the saying goes, right? An image can convey better than a thousand words. In many cases, it is, but you have to know what you want from that, right?

So you usually want to again store the image as a first class citizen not text and then be able to both use it in retrieval. So this image is relevant for this query and then maybe even show it as part of your result from let's say if it's a chatbot or something right show it as part of the result as an image not describe what you see right users can enjoy that as well. Now audio and video become even more complicated. One of the examples I like to talk about to demonstrate why video is so critical is so audio, you know, you can transcribe it if it's if it's somebody speaking like you and I, but if it's music, unclear how you think if some music is relevant for something. So, usually people ignore that as far as I've seen so far. But video is like a sequence of images with audio, right? So, it's like the most complicated of all. Yep. And it's really important to understand both. So for example, imagine a training video, okay? And I'm standing there. I'm the instructor and I'm saying, "Okay, so in order to avoid catastrophe, never press this button."

Now, if you look at the text, you don't know what this is. If you just transcribe it, you lose the meaning completely, right? This is called like the red button problem, right? Like so you want to you want to have visual understanding of what's in the image and or in the sequence of images in the video as well as correlated with the audio in order to get the complete meaning of what's going on there. Right?

So how do we solve this issue that this is a data processing issue like using a different type of embedding model like a vision language type.

Yeah. Well that's a great question. It's actually quite complex and there's a couple ways to do it. There's I would say there's two types of approaches today. One is still to try to translate it into text, right? So you take the video, you transcribe what you can, but you also in parallel to this transcription, you have a visual model like I mean a lot of the frontier models today can understand video too. And so if they can tell you like you take a chunk of 3 seconds, you transcribe it, you also say what is happening here?

You sort of correlate them so you get a textual description of kind of what that piece of 3 seconds means and you use that is a you know high processing on the sort of on the ingest side but if you solve it correctly then it kind of works it makes search you know and retrieval really fast. There are starting to be these really good embedding models for video too that you can just give it the whole video and say like tell me where somebody is clicking a red button saying something or something like this. They're not as good yet. So they're they're kind of in research. They're being progressing really fast. So my feeling is from what I know is they're not necessarily good yet. Then I haven't seen enough benchmarks to convince me that they are good enough.

Are companies using it though?

I've not seen anybody using it. I think people are intrigued by it because it supposedly is the right way to solve it. But you have to make sure that the models are good. And there's also visual has another complication, right? Which is let me give the image as an example, not video just to make it simple, but even in images. So let's say I have this image we see in front of us in a slide, right? If I try to tell a model to summarize what's in the image, it'll probably notice different things different based on my problem. So normal person might say, hey, it's a diagram of a RAG workflow with a bunch of boxes including this and this and this and this and the arrows go from here to here and do summary. But what am I interest what if I'm interested just in the color scheme or something? You don't know what people are interested in images, right? So sometimes specifically to the data set that you ingest and the use case in the enterprise, you might need to use different techniques depending on what the goal is and what the focus is, right?

So I wouldn't say it's a solved problem either. Like it's it's kind of work in progress. It's really hard. It's getting better all the time thanks to a lot of fantastic research in the space in academia and frontier labs. But it's a very interesting topic

In my use case. What I'm seeing is my claim that I need to verify could be very a very complex sentence it by itself and oftentimes when I do call to do verification it can only verify like half of the sentence and it doesn't have the complete understanding of what you know I need this whole thing to be verified. So it's very nuanced and complex.

There's a couple things here. One is again I think the agent the agent and even my maybe an extension of agent is now we call this harness right harness engineering so I think harness is like a better agent it's agent harness it's the same thing it's just a better implementation of it right and so I think good harnesses of agents good loops of a agentic loops sort of really help you they're really good at identifying the pieces of your question breaking it down to different subtasks and this is what you I think for your use case and then tackling each one as use case if you got if everybody in the audience here is using of course Claude Code or Codex you see that they do that all the time they even tell you here's my five tasks I'm going to do and do them one one by one right so I think that's that's becoming the way that these types of problems are solved and I think that's why I'm kind of gravitating with you here in the conversation towards this but I want also to mention another interesting part that we didn't talk about which is knowledge graphs.

Okay. And it's not directly to answer your question. I think the agentic harnesses are where this is this will help a lot. But I want to talk a little bit about what are knowledge graphs and why they're important. There's another technique to do better retrieval. So there's there's a lot of work that has been done on building knowledge graphs in the last I don't know decade or so. Lots of companies in the space that you can have knowledge databases with. And for those you maybe who's not familiar, it's essentially think of it as a graph of nodes and edges and you build a relationship between them. So for example, the sort of common typical use case to demonstrate is movies, right? We have nodes could be movies, they could be actors, they could be directors, they could be all kinds of other participants.

And there's arrows between like, oh, in this movie this was the director and this was the actors that are participated etc. And you can start asking questions that are really interesting like for example in the movie Inception sorry like let's say what other movies the director of Inception directed. So to answer this question you got to go from the movie to the director and then look at the other nodes that are related to that director and answer that question. In semantic search that will probably usually not work right. It's very hard to make that work. And so a lot of these these are called the multihop questions when you need to go through a graph and sort of like a relationship. And it's a very powerful tool to do that. Now the problem I've seen is that it's also so very a lot of people get really excited about this. What but they have never done a graph. They haven't built a graph. And so the challenge is that it's really expensive to build and maintain.

So what I usually say is just really think through the ROI. Of course it's good to build a knowledge graph. It's of course it could be helpful, but if it solves like 0.1% of your queries, it gives you a boost of 1% out of a you know 10,000 queries a month. Are you ready to spend you know the team effort and the effort on the infrastructure and maintaining the graph over time? So it's really useful when you have really important use cases that are critical and a significant amount of your queries in your AI system are the type that will be you will benefit from that.

When to use knowledge graph with your RAG system is when you have a high stake use case your data actually fits this pattern of more relationship. Even with such use cases it's still going to be expensive right?

Yeah, it will be expensive to build maintain not just technically but also just the graph itself need needs constant changing right and you usually have to have people do that and so I think it's yeah it's a it's a mix of high impact or high importance queries with a significant amount of them that it matters right

Speaking of maintenance embeddings need to be maintained for instance if I have a let's say e-commerce store and they may have new products right every week or product could change as Well, how do you maintain that system in production?

It's a really important problem and I'll mention two things about this. One is yes, absolutely. Think about even a simple simple use a very common use case is to index all of your Google Drive in the embedding system in a vector database, right? Google Drive documents change every day, right? So, if you have again 10 documents, it's not that much of a problem. You just reingest them when they change. But when you go to production, it becomes a really difficult thing because you don't want to ingest everything every time, right? So you want to keep this refresh algorithm that sort of knows what's inside a vector store and only triggers on the chain. So there's two ways to do that are typical in mission critical production systems that I've seen. One is you could store in your RAG pipeline sort of a hash of the documents and when you reingest let's say daily and just go through all the documents calculate the hash and if only do the ones that change that's still a reindexing but a fast one because you skip like 99% of them.

A better way I think is but it depends on whether the source system supports it is to have a I think it's called a kind of trigger base right so let's say that Google Drive if we take that example or it could be S3 or whatever other blob storage right could tell you every day send you a like an REST API call to some process that sort of says hey these five files have changed go reindex them right that is probably a much more efficient way to do but not all source source systems support that. So sometimes you have to do the push, sometimes you have to do the pull.

I'm building this system which component is the most expensive.

Well, there's expensive from kind of let's say API cost. It's expensive from a systems perspective like a machines or expensive from a building it as an engineer or data scientist, right? Did you have any particular one in mind or do you want me to cover all three?

Anything that you've seen very common in production that people aren't paying attention to like when you're when they're building the prototype right

So I would say you know let's start with the retrieval pipeline here right so vector store has become pretty you know pretty commoditized in a lot of ways there's a lot of options it's super easy to do right now again if you have a big vector store in production you have to remember that you have to maintain it reindex it once in a while so but I don't usually see see that as a thing that pops out in people's head is like that's my main pain point. Now if you want to build hybrid search if you or reranker that becomes a engineering effort or machine learning team effort to build there are existing ranking models you can use even the embedding models you keep changing so a lot of the effort here is around keeping those up to date there's more newer ones and that you can utilize and part of maintaining a system like this is being aware of the new ones and upgrading and then you have to test everything and evaluate that nothing has changed and everything became better and not worse. So we haven't touched evaluation but we should maybe touch on it in a in a minute. Yeah.

But part of this is not so much about cost. Even the GPUs you have to run these models are not that expensive. You can also run those on remote like an API like the LLM. So depending on your setting those those are not super expensive. The ingest itself could be really expensive. Right? If you want to let's say ingest a million documents that are all really complicated PDFs with tables and images and maybe videos that could that could be really expensive and when you think about it in production the first ingest is usually the most expensive one right makes sense if you do refreshes and it's not that much it's you know 1% of the documents every every day or week it's not that much but you have to think about this in terms of I'll give you an example You can think of some some companies have documents that are 15,000 pages PDF.

Okay, real documents that they need to ingest, right? So you need to think about and this is the difference between a prototype and a production system like so you can build a pipeline and then page you know 4,000 it goes out of memory and you have to restart it and it doesn't work again. So you have to think about and this is the cost here is a lot of engineering smart engineering and how to build it so that it really can handle messy data lots of data long data that and whatever pipeline you do to extract the text to chunk to do all the processing that you need to do has to be very scalable and part of it is also adding a lot of instrumentation so you can start seeing what the problems are and solve for them it's hard to predict up front. Every company have so much types of data and it's all messy and you always find these edge cases. Oh, this type of file with this type of image always fails and I have to fix it. So it's it's a continuous journey.

My recommendation is really to build with flexibility here initially. So you and a lot of instrumentation so you can start scaling your data and see what happens. Now on cost then you go to the generative part right the generative part is well that's that's a cost you know right if you call OpenAI or Anthropic or Gemini or something else you know what the cost is per API cost and usually the cost here could could grow if you have a lot of queries every month if you call embedding and reranking also in an external one that's the cost. You can also use the open the open open models, right, and host it yourself, right? You can take Gemma 4 or DeepSeek or you know Qwen or one of those GLM 5.2 is known to be like a good one today, but then you have to host it yourself on a GPU machine or maybe a big GPU machine. So that could be a an also a big cost. These models are awesome, but self-hosting them is not always easy and cost you. You have to have a machine.

Many people do prefer that because if you depending on your you can do the math and see like if I do this for a year and I buy a machine where does you know and the cost of the API call is this and I have so many queries a month what does it come down to right so that's where a lot of the costs come in I think in terms of engineering this is this part is pretty easy today just you know call to an API or if you host it you just have to know how to use products like vLLM and things like this part is more complicated from a team perspective like you have to really understand search retrieval pretty well. I think that's that's what I would say about cost. We didn't talk about hallucination detection and maybe we can we can touch on that just briefly in this in this point is I think of cost also as the cost maybe more holistically of you work so hard you pay so much money to vendors to build a system. You want your users to be happy, right? You want the results to be accurate.

You want Angelina to know that the 95 is fixed to an 85 whatever it may be right so I think investing in quality across all of this pipeline including detecting hallucinations using best in breed LLM using the best models here will help your users look at this and say yeah it's good as opposed to it's a new system we have you know 50 engineers working on it for 6 months and launching it to the company of 10,000 people and everybody's like I don't care it's not really helping me. Right? So you want to make sure that you do all this work and you really worry about quality end to end including whatever and if you need a knowledge graph here you can embed a knowledge graph whatever you need to do to make it really valuable to the end users.

So this is the part when we want to have a good a robust eval system.

Yeah. So eval is a whole other topic. So let's jump into that because I think it's important. So, and I want to I want to talk about that and then maybe we can talk about evaluation of agents too a little bit because it's also an interesting but you know starting with RAG right what you want to do is you want to know that the result that gets sent here back to the user is a good result right and there's a couple of couple aspects to that so the first one usually you want to check on the retrieval did I actually get the right chunks of information if you didn't it might be that your search is not good enough.

Maybe the vector search is the only thing and you don't you needed rerankers or something. It could also be another things you may have used if you just use semantic search. When you choose embedding models, you can choose different sizes, how many what the size of the vector. There's a lot of different things that could affect this. So you want to you want to first check your output of the retrieval pipeline to see if you actually got for every query the right set of relevant documents that you expect. To have been for this type of query right in your case in your use case you know if I ask something about one of the one parts of the document did I get the relevant parts that had if I asked about the 85% did I get the 95% or whatever it may be right when you think about it this is a really hard problem in production to do so typically how people solve is they have a bunch of queries let's say a thousand queries that you have as your eval data set and you have to specify what are the right chunks of data? So, it's not a document, but it's the chunks of data that would return.

It's really hard to sit down and write that up. If you think about it, like you said at the beginning, like I don't want to go to all these documents and sort of figure out where the right places are. That's what's required here, right? If you think about it. So, there are good metrics for this that people can see. But in evaluation chapter, we mention all of them. It could be precision or recall traditional metrics like this. But the real problem is not the metrics is how do you collect the data for this.

Now imagine that you also have a million documents not 100 and then the documents always change. So how do you know like tomorrow the documents change and the chunks returned as a top chunk is different. So how do you keep that loop intact? Yeah, it's a really hard problem and I'll I'll mention a solution in a minute, but let me just finish the overall here. The next step is of course you want to look at the generative output like what was the output response and you want to say like is this response correct? Does it use the ranked results properly? Because the LLM could go off the rails too, right? You could you could get five results that are relevant and not use them or use them or add its own stuff, right?

And you want to also ask sort of the user like is this is this answering your question like sort of like a human feedback here. You usually have a query and a kind of golden response so to speak that somebody writes and companies I've I've seen I've talked to a few companies who just they sit down two or three people for a few weeks and have them write the sort of curated results. That's that's good. It's it's you know it's not bad to do that. Of course, it's not perfect, right? Because there's probably five ways to write the same kind of gist of the response in different words in different ways. And but assuming that you have that, you can compare the generated result to the curated result and see how close they are and score it that way.

And so one of the things we talk about in the book is something that was a bunch of papers with a fantastic group of people from University of Waterloo in collaboration with the work I've done with Victor with many other people in the company is this idea of reference-free evaluation of RAG. So this is Jimmy Lin's lab and many of his very talented students. And so the idea is to they have a couple of metrics that actually do not require the golden answer or not require the golden chunk in the retrieval part and still have the ability to give you a really good score of whether the retrieved results or these are actually useful. So those are really really useful techniques. They're not perfect. They have somewhat of their own downsides.

What's the design thinking behind it?

So it uses so I'll explain one of them just for time but like one of them is called the UMBRELA. It's the retrieval part and so what UMBRELA does is you say you take each one of those chunks that you get out. Let's say you get 10 chunks that are most relevant and you look at the chunk and you have an LLM call to score this between zero and three. Zero being this chunk has nothing to do with the query at all. One is it's got something to do with the query. Two is like it's it's quite relevant but it doesn't answer the whole question. And three is like it's a perfect answer, right? Just that chunk. Now again LLM are not perfect but with today's LLM this would be a pretty good rating system. And then and then what you do after that is you take you take these ratings and that's what you use to score.

Now the important thing about this research this is nothing really transformational right you can this called LLM as a judge you can always judge things with a prompt. The nice thing about this technique and the work that they've done at University of Waterloo is that the team have done over many years they've done research and have asked human beings to also rank these these chunks on a number of data sets and they've shown that the correlation of that particular prompting technique and the particular way they've scored is one 0 to three highly correlates with the human answers of the same thing. So that's the power of it. It's not like everybody can invent a system of scoring like this but the trick is to show that it's actually effective because otherwise you could have biases of LLMs and all kinds of different things. So that's really the power of this and the other one this technique used for generation is a little bit different but it also uses LLMs to automate in a way that's been tested with human correlation. That's that's really the main trick there.

When do you think a company should think about building their own RAG system for my use case? Maybe I'm just verifying a few blog posts I'm generating. So it's pretty small. I can build it myself when I can to my harness to my taste. But what if like at a scale it's maybe a marketing agency, let's say they're maybe doing a million blog posts per per months or something like that. Do you think they should build it themselves or what do you think the solution people?

So, you know, so there's there's first of all, there's build it yourself with tools like LangChain or LlamaIndex was traditionally a year ago and there's a couple other tools out there that are open source and you can build it yourself and if you have expertise in you know data science and AI engineering you can certainly do that. I think you have to think about whether that's your business, right? Is that something that you want to keep building, maintaining? It also depends on how complex it is, right? Do you have multimodal data? Do you need a knowledge graph? All these things compound to complexity of engineering and maintaining a team of experts in your company to do that instead of spending the same resources on your actual business, right? Your actual use case.

And that's why companies like the company I work for Vectara and other companies similar to that have come up with what's called RAG as a service and also agent as a service which says we're going to build all these components for you. I want to give you an API as a service. So you outsource in a way all of the complexity of building and maintaining all these components, all this complexity and upgrading it to the latest models to us and that's why you pay us essentially. And you don't have to be an expert in RAG. You just implement a RAG with the APIs on top of the platform. So that's that's the kind of buy versus build that happens in RAG and happens in agents in all the other areas.

What about buying an off-the-shelf RAG system solving my use case comparing with hiring a consulting firm build it, Vectara build it for me versus like buying some domain tool already exist.

So there's a lot that's a great question because there are a lot of vertical domain tools like this available and I would say this like so when you look let's take legal as an example right there's many many tools in the legal space that are vertical and the question is what do they do they do what you need or not right if and I think that's the core question right a lot of them do some use cases not use cases you want but I think if they do what you want they're very easy to use they're usually some kind of monthly subscription and you don't have to do anything and you can always stop the monthly subscription if you need something else. So I would I would think that's an easy decision to make when it does what you need. The problems come up with one when they're missing key features that you need or key like let's say legal review could be a review of a of an agreement, right? And redlining it.

But what if you need also to suggest improvements and it doesn't do that. It just highlights that the changes needed or something and they're going to do it but maybe in 3 years from now you need it today. So you just you can move forward. So sometimes you have to do it yourself. The other thing is sometimes the data that you need to send to it is not like they're they're SaaS companies and they make you send the data to them and sometimes that's just a non-starter, right? So a lot of companies need air-gapped environments and everything has to be on their data center and some of these companies just won't support that yet.

Let let's close out this chapter of discussing your RAG systems. You can close out the screen sharing. I have a I have a question about DevRel for you. If I come to you as a startup let's say I build an agentic framework agentic system company. How important is dev relationship and how should I think about strategizing this piece in the first year or maybe the first three years of go to market strategy?

Yeah. So really good question. So DevRel has become well it's it's been around for a long time more than I've done it. So I've started as a as I said data scientist and machine learning expert and I've sort of worked on the DevRel side maybe the last three three and a half years only. And I love it by the way. I think it's a wonderful career path for whoever is interested in that.

But generally DevRel involves two things in my view like one is and it's it's meant for companies who usually have some sort of API some sort of ways for users to use a product that doesn't require like to call a salesperson and go and kind of implement a full system right the way we used to sell big services in the past right and so when you have an API you kind of want to get developers to use it that's where DevRel comes in right so it's literally relationship building relationships and trust with developers. And so there's two paths to it. One is as a DevRel, I want to convey to the developer community why the product is really helpful, what's it helpful for, and actually demonstrate it. So build demo cases, show show code examples. Basically, it's being a teacher about your product. Here's what the product is and how to use it. And being a good developer helps a lot because you know it's it's your people, you know, you can speak the same language and you can explain how to do it really well. So that's one part. The other part is bringing feedback back to the company, right?

So like you go to conferences and events, you talk you talk to people after your public speaking things, you talk to people at different you know events and things like this and you hear a lot of feedback and people say like you know product is great but this thing doesn't really work really well or this documentation part is not doing well. So I think that's where that feedback is very valuable to the company who sometimes are not aware of it or miss it or something. So bringing that back into the product and engineering group in a company is the second kind of function of it. But to answer your question for startups, there's a lot more startups now that you know I see a huge growth in that in the last maybe half a year to a year in the demand for this because it is a very good growth strategy for a lot of companies where instead of investing in a really expensive sales team that is expensive and needs to justify its its existence very quickly in any startup that's called a sales-led growth.

And so you can invest what's called product-led growth which is you take sort of the product you make it available in a self-served manner to a lot of developers and then you open it up you publicize it you get you know a community growth and so you can have a thousand people using it 10,000 people using it you know 100,000 people using it and so on and what you get in that is you get really good feedback quickly because it's a you don't get a lot of revenue necessarily from those right away right you have it's an investment.

You get a lot of feedback about the product. Sometimes you even change the product dramatically if the feedback is very overwhelming. And then the idea is that over time a lot of developers find it really helpful and then that is a signal of product market fit for the company. And then they call you and they say or they call their boss and they say because developers typically may not have the budget to buy anything but they'll call their boss and say we need to buy that system. Go buy it for us. So if it's a if it's a seat thing, they might just buy two seats and then four seats and then they tell their friends in the company. Eventually you get growth that way and so it's a really good strategy to get this growth engine going.

What's the means of doing this? Is it are you thinking of like hackathon style or is there any other means?

Hackathon is pretty common, especially at the early stages where you want a lot of feedback from people building lots of stuff with your product and seeing where it where it fails, where the docs are not clear and things like this. But there's a lot of other things. So, usually you go to events where so let's say if you target developers, you go to events where developers for your product go to, right? There's a lot of different events, especially the AI, there's there's tons of events all the time. You hopefully get to either you as a company you can sometimes sponsor a booth and get access to all of these these developers or you can do a public you know do a talk in one of these events.

So that's another another big avenue. I think the other thing you do is you just educate. So think think how do you be a teacher right? So you write very good blog posts and post them on a company that really matter. Not and I'll say this with a little bit of kind of like there's a lot of blog posts today that are I think we're losing quality is how I would say this like there's a lot of AI slop that's going on. I think to be good at this I'm I'm a big user of AI and I this is not to say I don't encourage people to use AI in content. I think it's very powerful but make it your own voice. Use AI to help you do it faster if you like, but it's got to be authentic. It's got to be something that developers have very short span of attention. If you bore them, they'll just leave and they'll have a bad impression of you. So create good high quality content that really helps people succeed in their in their role. I again I always think about it as a teacher like if I was a teacher at a high school, I'd want my students to succeed. I think it's the same thing, same mindset.

So write blog post, create demos of how to use your product with very interesting use case that hopefully will fit what people want to build things like that.

How do you plan a budget for a dev relationship for go to market like for year one for a company and how long does it take for you to see some results?

Budget really depends. It's hard to put a number on it, but I would say you should think about you know if you have you have to have first of all a product that could be usable for DevRel. You should have if you self-serve you should have good documentation. If you're earlier than that it's not going to work out. You have to have kind of be at a stage where the product is usable by thousands of people, right? If you open it up, right? Once you do that, I think you have to have some DevRel person or fractional or something to help you sort of with the strategy. But generally, you need a budget for going to events and you have to choose which ones. And I usually recommend if you're starting out, experiment because not all events are good for you, right?

So try small like take a small booth and a couple see how it works. By the way the events don't only only help DevRel they also help sales. If you have if you do founder sales or you do if you have a first salesperson you can do by the way PLG and SLG at the same time. It's very common. So you can do both both of these actions at the same time. And so the events help in other ways but experiment a little bit. Don't think that you know the strategy and go all four fors. Try that different things and then you'll learn really quickly what helps and what doesn't help, what converts and doesn't convert. There's also two modes, right? You can be in awareness building mode where I don't care about customers yet. It's too early. I want to just people to know about me and people to like try my product and I get a lot of feedback. And a little bit later, you say now I'm really trying to get, you know, I'm trying to build a business.

I'm trying to get revenue in the door and I really care more about converting some of these leads and some of these PLG motion into actual sales of enterprise customers and that's a little bit of a different phase both are important just happen in sequence I mean since the day you start probably 3 to six months minimum just because you know you go to some events and it takes time like you can't go to event tomorrow sometimes it's and you go virus right yeah they don't have an event until November or whatever. You go to meetups, you do talks, things like this. I think also the other thing I would mention is I know there's a lot of people who think of DevRel as a how do we measure it? How do we know for forget about how long, but how do we measure it? And I think measurement is really important. But and let me actually say how you measure it, right? Like you can measure by conversions like how many people actually heard about your product and signed up to the product. You can measure by not just if they sign up but do they use it.

So you can measure a lot of stuff in your product but I will say that I like to think about as do as much science as you can but don't overengineer this because a lot of it is hard to attribute to right so the company does usually many things there's there's events that are unrelated to DevRel there are things that are happening there's talks there's press releases about the company and investment rounds and it's sometimes hard to know when it was your direct influence versus somebody else. So I think of it as like yes try but don't spend three months just designing the mechanism to measurement and making it accurate. I'd much better have you spend the time actually doing the work and directionally if you see I did six months of work and I got five customers then yeah it's not working. It's very clear and if you get 10,000 customers you're like oh something is definitely working and then you can fine-tune and sort of dive a little bit deeper.

There's a lot of course mechanisms right like people in mark traditional marketing mechanisms how to measure like if you give a code somewhere you measure where the code came from and stuff like this but don't overthink that too much is my suggestion

I think actually five customer conversion and 10,000 are actually really good results because either you have PMF or you don't the challenge is when you have a sales it right

That's a great question I don't think it's easy to tell right so I think that's where that's why startups are hard. I mean, yeah, forget about DevRel like even as a founder of a company when I run the company for a year and I don't get the amount of revenue that I need is like are my is the people not selling correctly? Is the product not good enough? Is there too much competition? It's it's really hard to tell and it's the same type of problem. So I think what I would suggest in that case is do what you do when you're a founder is the same thing. Just think about it from first principles. Come up with new ideas, try to at some point, you know, founders tend to be you kind of want to have the belief until the end and go all the way and like I'm going to do this. I believe in this like you have this unreasonable belief in your product. That's what drives startups to success again and again. But at some point, but many startups actually that belief is wrong.

And so at some point after enough evidence you also have to have a little bit of a red bulb in your head that says well I got to change something here right so I don't think it's a DevRel thing I think it's a company thing DevRel I mean if you have you know DevRel may not be necessary in a company if you move to full sales-led growth and you say DevRel is just people not interested enough to do it and that's that's fair thing to do um

How do we as engineers or data scientists adapt to this changing era of leveraging AI right there's RAG is dead is one topic and engineers are dead is another topic data scientists are significantly affected as well we don't require that many data DS teams anymore so what do we do

So I do have some thoughts about this first of all I recognize that it's a problem and it's not just a problem it's a pain that impacts real people with real families and but I do believe myself that it's a little bit of an overshoot of the companies, right? They're sort of like they're they're reacting to something they hear and don't they don't even know what's what's happening. I'm talking about the big companies that are sort of, you know, firings of like 5,000 people or 10,000 people or things like this and yes, impacting data scientists too and people are afraid of like are the coding agents going to replace me. So my thought on that is twofold.

One is and I guess my advice is my feeling is that we're going to end up in engineering and data science being you know it's not a word I invented but like directors as opposed to actors right so the coding agents will write the code they're becoming better and better all the time and I think what remains a really critical challenge is to understand what we need to build and guide it in the places where the coding agents are not doing good job like fundamentals things that are not obvious that there's five ways to build something architecture those are where I still there are things still value and this could also transform and change over time but I think to me the best engineer data scientist of the future will be people who essentially augment themselves effectively with the coding agent right in a way I by the way believe this is going to happen in other professions too like any anything think about like I don't know I'm going to pick a weird example like somebody who does a digging in a in a site right for history right and they find something and they say oh I think this is from I don't know Egypt and you know 2000 BC and then they have to go to the library and talk to all their friends and dah dah dah. What if they just shown it to ChatGPT got the response probably mostly accurate it and then did more digging at the same day.

So I think everybody can augment themselves if you figure out the right way. Legal is probably the most incredible example that I think a lot of lawyers are afraid of, but generally like a lot of legal contracts that are I probably wouldn't use a legal LLM to protect me against, god forbid, some some really high impact case that you know of whatever in court. But if I have a contract sent to me by a landlord that I want to review and send feedback, it's probably okay. The impact, the risk is not that high. You know, probably find some things. And you know, LLMs write really really well emails back and very articulate. So it's got a lot of use cases. And so being able to augment yourself in every profession, including in coding and data science and analytics, and just to be more effective, I think that's what we're going to see. We're going to see that employees become that and the ones that are more proficient at that will succeed more in my opinion.

Do you have kids? What do you do?

Well, it's complicated. One of them is actually in this in the profession of building, you know, LL actually video models. So, it's so I think he's worried about other things at this point. But I do tell them kind of the same thing to sort of like figure out if they care. Not everybody cares about this. Not everybody's afraid about it at the same level. But I tell them that yeah, you have to adopt it and embrace it. And I know it's been said many times before, but I do feel it's a little bit like many other technologies that have been created in the past and we as a society adapted to it and our jobs became different. Now that does not mean actually I'll say this way what I'm concerned about is as a society what we do with this. So I think there's a way where this is actually benefiting everyone but because of how finance works and how sort of societal differences in between the poor and the rich and how you know governments around the world react and do deal with this.

I think that's the risk I'm more most concerned about for me and for my kids, which is will this result in more, you know, you could see this result in more layoffs and, you know, companies, you know, abusing this to make more profit. They might lose they might lose talent and it might eventually like not succeed, but in the in the meantime, there could be a lot of pain, right? Yeah. So, yeah, I hope that doesn't happen. I was fortunate to have a bunch of high schoolers invite me to here in the Bay Area. It was a high school and they invited me to their they have a computer science club and it was all people in last year of high school and they all got accepted to their computer science program for the next year in college.

And they were just looking at me. This is about I want to say like nine months ago. They were looking at me and say like, "Did we make the most incredibly stupidest mistake ever?" Right? What are we going to do four years in college? Right? So, I told them something similar. I said, "No, look, companies now equate a jun like a junior engineer with somebody that just graduated from college." That's like one to one, but it doesn't have to be. You can graduate from college and have the capability of what would be like a mid-level or a senior engineer today. So use the next four years to learn how to be that you actually have space to do that during college and learn how to use the AI tools as effectively as you can and make sure that you come out of college being this hybrid computer scientist of the future, right? Or data scientist of the future or whatever you are. I think I think I hope that colleges and universities support fast enough support their students in doing that and change the curriculum to support that.

I don't know how they're going to benchmark themselves to say I'm coming out as what's used to be a mid-tier engineer. I don't know if that equates to medium tier engineer anymore today or maybe in the next 3 to 5 years.

Yeah, I think all those classifications are changing too like who knows like today I mean I think it's still important in universities to learn the fundamentals of computer science those are really going to stay really important fundamentals of statistics machine learning how it works kind of understand it but I think the both the grading in universities and also the acceptance of new employees to a company how are they going to do the testing of write an algorithm to solve Fibonacci and five lines of Python like that's worthless like it's not really valuable So I think the way people evaluate candidates is also going to have to change a lot and I wouldn't be surprised if they'll say hey here's ChatGPT or Anthropic coding agent and we give you some some some exercise and we see how you do with it together and they'll test how much you overcome some challenges perhaps that are sort of in that working together with the coding agent to produce something that's high quality and testable and is the right decision in the process things like that.

I've heard of some of the founders I interview mentioning that they're incorporating AI tools in their interviews already. Thank you so much, Ofer. Thanks for sharing so many insights about building a reliable production grade RAG system and you know giving giving me some thoughts about my system as well. Also really appreciate your insights on getting us prepared as well for this changing world and hopefully we are all seeing a bright future of everybody you know being enhanced rather than you know being replaced.

Exactly. No, thank you, Angelina.