Why your RAG can’t tell you which docs are out of date

A team we spoke with recently has a release process worth describing, because the thing that goes wrong in it goes wrong in a great many places.

When they ship a feature, a QA engineer opens a ticket in an internal release tool. The ticket describes what the feature does and carries a link to the Jira story behind it. The internal tool then calls a retrieval system the team built themselves — one that has indexed every article in their knowledge base — and asks it a perfectly reasonable question:

Given this feature, which knowledge base articles need to be written or updated?

The suggestions come back, and they are not good. They tend to be articles that sound like the feature rather than articles the feature actually broke. The team’s own read was that their RAG implementation needed work, and they were looking for a better one.

I think the more useful read is that they asked a question their index could not answer, and that no amount of tuning would have changed that. It is worth walking through why, because the same shape shows up far outside documentation.

The question is not really about the documents

To decide whether an article is out of date, you need to know two separate things: what the article currently claims, and what recently changed. A search across the knowledge base can tell you the first. Nothing in the knowledge base can tell you the second.

Everything that describes the change lives somewhere else. The intent is in the Jira story, which explains what the team meant to build and why. The actual change is in the merge requests, which show which files moved and what the new behaviour is. Some of it is only in the code, where a default value quietly flipped in a way nobody wrote a sentence about.

So the retrieval system was handed a question about the relationship between two bodies of knowledge, and given access to one of them.

Figure 1. The knowledge base holds what the documentation claims. Whether any of it is still true is decided entirely by systems outside the index.

“But the ticket links to the Jira story”

This is the first objection people raise, and it is a fair one. The release ticket really does carry a link to the Jira story. The story really does reference the merge requests that implemented it. The chain is sitting right there in the data. Why can the system not simply follow it?

There are three answers, and the third is the one that decides the architecture.

1. A link is not content. Indexing the release ticket stores the ticket’s text, and that text contains a URL. What it does not contain is anything the URL points at. Embedding the string /browse/PROJ-1234 puts those characters in the index; it does not put the story’s description, its acceptance criteria, or its list of merge requests anywhere near it. Retrieval can match the URL as text. It cannot dereference it. Following a link is a fetch, and fetching is not something a similarity search does.

2. Following it at query time is a different system. You could certainly build something that reads the ticket, extracts the URL, calls the Jira API, reads the story, extracts the merge request links, calls the code host, and keeps going. That is a perfectly reasonable thing to build — but it is an agent with credentials and tools, not a retriever with an index. It needs authenticated access to every system as the person asking, it has to decide how deep to go and when to stop, and it pays the latency of every hop on every single query. Teams often discover they have accidentally started building it, one special case at a time.

3. And the last hop is not a link at all. This is the part that catches people, and it is worth sitting with.

   
   

Grant a perfect link-follower with every credential it needs. The ticket points to the story. The story points to the merge requests. The merge requests point to the changed files. And then the chain simply stops. (Figure 3, further down, traces the whole path and marks where it breaks.) Nothing points from a changed file to the documentation that describes it. Nobody writes that link, because keeping it accurate would be more work than keeping the documentation accurate, which is the problem we started with.

So the final hop — from a piece of behaviour that just changed to the pages that describe how it used to work — is the one the entire question rests on, and it is the one hop that cannot be followed. It has to be inferred, by comparing what changed against what the documentation claims.

That inference is a retrieval problem, and making it requires the changed behaviour to be sitting in the index next to the documentation, so the two can be compared at all. Which brings us back to scope, by a longer road.

Similarity and staleness are different properties

This is the part that makes the failure so persistent, and so easy to misdiagnose as a tuning problem.

Suppose a release adds support for authenticating with a service account, where previously only a personal user token worked. Rank the knowledge base by similarity to that description and the top results are the articles that are visibly about authentication: the authentication overview, the OAuth setup guide, the page on rotating credentials. They share vocabulary with the feature. They rank well.

Now ask a different question. Which articles are actually wrong now?

The troubleshooting page that says if you see a 401, check that your user token has not expired is now incomplete advice, and it will generate support tickets within a week. The quickstart whose second step is “create a personal token” now describes the harder of two paths as though it were the only one. A comparison table two levels deep in the integration docs is missing a row.

None of those pages talk about the feature. Some of them do not contain the phrase “service account” at all, which is precisely why they are wrong — they were written before the concept existed. Similarity ranks them low for exactly the reason that makes them urgent.

Figure 2. Staleness is not correlated with similarity. The page most likely to generate a support ticket is the one that never mentions the feature, because it was written before the feature existed.

The system was measuring one property and being graded on a different one.

Why the usual fixes don’t move the needle

When retrieval underperforms, there is a standard escalation ladder, and this team had climbed most of it. Better embeddings. A reranker on top of the candidate set. Larger chunks, then smaller chunks with more overlap. A longer context window so more of the knowledge base could be passed to the model at once.

Every one of those changes improves how well a system finds the right thing among the things it has. None of them changes what it has. The Jira story was not slightly too far down the ranking; it was never a candidate. The merge request was not chunked badly; it was never ingested. You cannot rerank a document that is not in the index, and a context window of any size does nothing for information that was never retrieved.

This is worth stating plainly, because the two problems feel identical from the outside and respond to completely different work:

• Retrieval quality is about ordering. The evidence is present and ranked badly. Embeddings, rerankers and chunking strategy all help here, and the ladder above is the right ladder.

• Retrieval scope is about presence. The evidence is absent. Nothing on that ladder reaches it, and effort spent there produces small gains that never compound into a working feature.

  
  

A scope problem wearing a quality problem’s clothes can absorb months. The tell is that every improvement helps a little and nothing ever fixes it.

The real answer is a traversal

Look at what a competent human does when handed this task, because it is nothing like a search.

They open the release ticket and follow the link to the Jira story. From the story they find the merge requests that implemented it. They skim the diffs to see which parts of the product actually moved. Then, holding that in their head, they go looking for documentation that describes those parts — and crucially, they search for the old behaviour, not the new feature, because pages describing behaviour that no longer exists are exactly the pages that need editing.

That is a chain of four hops, and three of them cross from one tool into another.

Figure 3. The first three hops are links that exist and could be followed. The fourth does not exist in any system and has to be inferred, which is why the chain cannot be walked by link-following alone.

Notice also that the final hop inverts the search. You are not looking for documents similar to the new feature; you are looking for documents describing the behaviour the merge request just replaced. That query cannot even be written down until the first three hops have happened, which is why the inference has to come last and why it needs everything the earlier hops produced.

What actually helps

The first move is unglamorous and it is most of the win: put the other systems in the index. Not a second retrieval system for engineering data alongside the one for documentation, but one index that spans the issue tracker, the code host and the knowledge base together, so that a single query can draw evidence from all of them at once.

Doing that properly is harder than it sounds, and the difficulty is almost entirely about permissions. The moment your index spans Jira and a code host, retrieval can surface things a given person is not entitled to see, and a system that leaks a private repository into a documentation suggestion is worse than one that suggests nothing. Permissions have to be modelled per source and enforced at query time, per user, or the unified index becomes a liability the day it starts working.

This is the part we have spent most of our engineering effort on at PipesHub, and it is why the connectors carry their source’s permission model rather than flattening everything into one collection. A retrieval result should be shaped by who is asking.

The second move is resolution across that scope, and this is where the interesting engineering sits.

When a reference appears in something the system has already retrieved — a Jira key, a ticket URL, an external id — it can be resolved to the record it names. The important detail is the search scope: resolution runs across every connector the person asking has access to, not just the source the text came from. A Jira key sitting inside a Confluence page resolves to the actual Jira issue even when the session is filtered to Confluence alone. This is a lookup rather than a guess: the identifier is matched against records that are already indexed, and each candidate is permission-checked before it is returned. No model is asked to infer the connection, and nothing reaches out to a live Jira or GitLab API in the middle of composing an answer. Each candidate is permission-checked before it is returned, so resolution can never surface a record the asker was not entitled to see. From there, traversal follows typed relationships across the whole organisation’s graph rather than within one tool.

Entities that share an identifier are correlated as well. People are the clearest case: the same email address resolves to one identity whether it arrived on a Jira ticket, a Linear issue, a GitLab issue or a Slack message, and records attach to it with typed roles — who created a thing, who it was assigned to, who reported it. So what has this engineer been working on across our tools this week is answerable without any link between the records themselves.

Put those together and the release question becomes answerable. The QA engineer’s ticket is indexed alongside the Jira story it links to, the merge requests that implemented that story, and the knowledge base itself. Asked which articles the feature affects, the system resolves the ticket’s Jira reference to the real story, follows it to the merge requests, and reads what actually changed. Only then does it go looking through the documentation — and it searches for pages describing the behaviour that just moved, rather than for pages that sound like the feature.

That final step is still an inference rather than a link, for the reason given earlier: nothing connects a changed file to the page that documents it. But it is now an inference made with the diff in hand, instead of a guess made from the feature description alone. That is the whole difference between the two rankings in Figure 2.

Every step is checked against the permissions of the person asking, and the answer comes back with citations to the specific records it drew on — so a technical writer can see why each article was flagged and go and verify it, rather than being handed a list to trust.

The honest summary is that unified scope gets you most of the way, because the overwhelming majority of these failures are a system reasoning over evidence it never had. Traversal is what takes you the rest of the way, and it is a real engineering problem rather than a configuration flag.

The general shape

Documentation drift is a good example because it is easy to picture, but the pattern is much wider than docs. Any question of the form what does this change affect has it. Which dashboards break if this column is renamed. Which customers are on the code path this patch touches. Which runbooks reference a service that no longer exists.

Each of those is a join across systems that were indexed separately, if they were indexed at all. And each will fail in the same quiet way — plausible answers, drawn from the one corpus the system could see, with no signal that the evidence which mattered was never in the room.

Before reaching for a better retriever, it is worth asking a cheaper question first: if a careful colleague had to answer this by hand, which systems would they open? If the answer is more than one, and your index covers one, the retriever was never the problem.

We work on this at PipesHub, an open-source context layer that connects your workplace tools and makes them searchable together, with each person’s permissions enforced at query time rather than bolted on afterwards.