Do You Actually Need a Knowledge Graph for Your RAG System?

Knowledge graphs are having a moment in the RAG world. They’re on every architecture diagram, in every “advanced RAG” talk, and near the top of a lot of roadmaps. And according to Ofer Mendelevitch, most of the people most excited about them have one thing in common: “they’ve never built a graph.”

Ofer is the author of O’Reilly’s Hands-On RAG for Production and an independent AI advisor who led developer relations at Vectara, an enterprise RAG platform. He’s not anti-knowledge-graph — the book covers them, and he’s clear about the specific queries they unlock that nothing else can. What he pushes on is the reflex to add one because it sounds sophisticated, without doing the arithmetic on what it costs to build and keep alive. His framing turns a hype decision into a math problem, which is exactly what a production team needs.

What a knowledge graph actually buys you

Start with the thing they genuinely solve, because it’s real. A knowledge graph is “a graph of nodes and edges” where you encode relationships explicitly. His example is movies: nodes for movies, actors, directors, with edges connecting who directed and starred in what.

That structure answers questions plain retrieval can’t. “What other movies did the director of Inception direct? To answer this question, you go from the movie to the director and then look at the other nodes related to that director.” These are multi-hop questions — you have to traverse relationships to get the answer. “In semantic search, that will probably usually not work.” If your users genuinely need to reason across chains of relationships, a graph is the tool, and no amount of better embeddings substitutes for it.

So the capability is not in question. The question is whether your workload needs it enough to justify the price.

The ROI test

This is the part worth writing down. Ofer’s decision rule is a cost-benefit calculation, and he states the failure case in numbers:

“If it solves 0.1% of your queries — it gives you a boost of 1% out of 10,000 queries a month — are you ready to spend the team effort, the infrastructure, and maintaining the graph over time?”

Run that math for a second. A graph that improves a tiny slice of your traffic by a tiny margin is not free — it’s a standing tax on your engineering team. So the test isn’t “would a knowledge graph help?” (almost always yes, marginally). The test is whether the queries it helps are both important and frequent enough to earn the cost.

In his words, it’s worth it when you have “a mix of high impact or high importance queries with a significant amount of them that it matters.” Angelina compressed it well on the show: use a knowledge graph when the use case is high-stakes, your tolerance for being wrong is low, and your data genuinely fits a relationship pattern. Miss any of those and you’re building infrastructure to solve a rounding error.

The cost people forget: maintenance

The build cost is visible. The maintenance cost is the one that ambushes teams. A knowledge graph isn’t a static asset you construct once — “the graph itself needs constant changing, and you usually have to have people do that.” Relationships shift, entities appear and merge, and someone has to keep the structure accurate over time. “It will be expensive to build and maintain — not just technically.”

This is the same trap that catches embeddings, by the way. Any retrieval layer you stand up becomes something you own forever. An e-commerce catalog with new products every week means re-embedding on a schedule; a knowledge graph means continuous curation of nodes and edges. The mistake isn’t underestimating the first build — it’s forgetting that you signed up for perpetual upkeep.

Ofer’s underlying principle generalizes past graphs: match the sophistication of your retrieval to the actual shape and stakes of your queries, and price in the ongoing maintenance before you commit. A knowledge graph on a workload that doesn’t need multi-hop reasoning is the RAG equivalent of premature optimization — impressive on the diagram, a drag on the team. Do the ROI math first. If the important, frequent queries in your system need relationship traversal, build it and mean it. If they don’t, the calm answer is no.

FAQ

When should you use a knowledge graph in a RAG system?

Use one when a meaningful share of your queries are high-stakes, need low tolerance for error, and genuinely require reasoning across relationships — multi-hop questions that traverse connected entities. If the queries a graph would help are rare or low-impact, the build and maintenance cost outweighs the benefit, and standard retrieval is the better choice.

Why do knowledge graphs get overhyped in RAG?

Because they sound sophisticated and appear on every advanced-RAG diagram, teams add them by reflex. Ofer’s observation is that many enthusiasts have never actually built one, so they underestimate the cost. A graph almost always helps marginally — the real question is whether it helps enough of your important queries to justify the ongoing expense.

What is a multi-hop question and why does semantic search fail at it?

A multi-hop question requires traversing relationships — for example, “what other movies did the director of Inception direct?” You have to go from movie to director to other movies. Semantic search matches on similarity of meaning, not on relationship chains, so it usually can’t answer these. A knowledge graph encodes those connections explicitly.

How expensive is it to maintain a knowledge graph?

More than the initial build. The graph needs constant updating as entities and relationships change, and that usually requires people to curate it, not just technical infrastructure. Teams often budget for construction and forget the perpetual maintenance, which is the cost that makes graphs a bad fit for low-value query workloads.

What’s the ROI test for adding a knowledge graph?

Ask what share of queries it improves and by how much. Ofer’s example: if it helps 0.1% of queries with a 1% boost across 10,000 queries a month, the team effort, infrastructure, and maintenance aren’t worth it. Build it only when the queries it improves are both high-importance and high-frequency.

Do you need a knowledge graph if you already have vector search?

Not usually. Vector and hybrid search cover most retrieval needs. A knowledge graph adds value specifically for relationship-traversal queries that similarity search can’t answer. If your workload rarely asks multi-hop questions, adding a graph is infrastructure overhead that solves a small fraction of your traffic.

How is maintaining a knowledge graph different from maintaining embeddings?

Both require ongoing upkeep. Embeddings need re-indexing when documents change — for example, an e-commerce catalog with new products weekly. A knowledge graph needs continuous curation of its nodes and edges as relationships and entities evolve, which tends to require human effort rather than an automated refresh, making it the heavier long-term commitment.

Can agents reduce the need for a knowledge graph?

Agentic retrieval helps with complex, multi-part questions by decomposing them into sub-queries and running each against retrieval. That handles a lot of what people reach for graphs to do. Knowledge graphs remain distinctly useful for genuine relationship traversal, but for many multi-step problems a well-built agent loop is a lighter alternative worth trying first.