Do AI Agents Need a Company Index? We Tested It

AI assistants such as Claude can now plug straight into the apps a company uses (Slack, Jira, GitHub, Google Drive) and look things up when you ask a question. So is that enough? Or does an assistant still answer better when it has a search index: one searchable copy of everything the company knows, built ahead of time?

We tested it. We gave one AI agent the same 100 everyday work questions twice: once letting it search each app directly, and once giving it a single index of all of them. Then we repeated the whole test with three AI models of different sizes.

The short version: with the index, the same agent got more of the answers right, answered about a third faster, and cost nearly a third less, with every model we tried.

Times are the typical (median) answer and costs the average per answer. The percentages are the average change measured question by question, so they can differ slightly from the ratio of the rounded figures shown.

Key findings

• More accurate at every model size. With a company index, Claude Haiku found 80% of the facts a correct answer needs (up from 55% searching each app directly), Claude Sonnet 89% (up from 65%) and Claude Opus 92% (up from 85%).

• Faster and cheaper at every model size. Answers took 27% to 45% less time and cost 27% to 32% less, because the agent made fewer lookups and read about half as much text.

• A mid-size model with an index beat the top model without one. Sonnet with the index (89%, $0.16, 22 s per answer) beat Opus searching each app directly (85%, $0.58, 55 s).

• Answers became easier to check. With the index, agents linked the exact source record far more often and made up to two-fifths fewer wrong statements.

• The smaller the model, the bigger the gain. Haiku and Sonnet gained about 25 points of accuracy; Opus gained 8, so for top models the case rests more on speed, cost and checkable sources.

• Summaries over time are still the weak spot, where the index and direct connectors were about even for Opus.

  
  

Figure 1. The same agent, with and without a company index: share of the right facts found, time per answer and cost per answer, for Claude Haiku, Sonnet and Opus (100 questions, 2 runs each, 19 Oct 2026).

A few words we use throughout:

• Agent: an AI model that can take steps on its own before it answers, such as running a search, opening a document or reading a thread.

• Connector: the official add-on that lets an agent use one app, for example Slack's or GitHub's own connector for Claude.

• Index: a searchable copy of all the apps' content in one place, kept up to date in the background. Searching it is like searching one library instead of five separate shelves.

• Facts found: for every question we wrote down the specific facts a correct answer must contain. "Facts found" is the share of those facts that appear in the agent's answer.

The index in this test is PipesHub, which we build and which is open source. But most of what we learned applies to any AI agent that works with company data, and that is what most of this post is about.

How did we test it?

The data. We used our own company's real working data. We are a small software company, and our work runs through five tools: about 13,000 Slack messages, 3,800 Jira tickets (our task tracker), 2,400 GitHub pull requests (proposed code changes and their discussion), plus pages from Confluence (our wiki) and files in Google Drive.

The questions. We wrote 100 questions of the kind people ask at work. Some can be answered from one tool. Many need two or three tools at once, for example "Which ticket did this Slack thread lead to, and did the code change for it go in?" Others ask for a detail buried in a long thread, a summary of what happened over several months, or whether something has changed since a given date.

The two setups. The agent answered every question twice, once in each setup:

• Searching each app directly: the agent uses each tool's own official connector, the same Slack, Atlassian (Jira and Confluence) and GitHub connectors anyone can add to Claude today.

• Using the index: the agent searches one index of all five tools and opens the records it finds. PipesHub's own chat assistant was switched off, so in both setups the same agent does all the thinking. Only where it looks things up changes.

Everything else stayed the same: the model, its instructions, the questions, and the day the test ran.

How answers were scored. For each question we wrote the correct answer by hand and listed the facts it must contain. Two different AI models graded every answer against that list, and a third settled any disagreement.

What did we find?

The index came out ahead on all four things we measured, with every model. We compared the two setups question by question. For each of the first four rows below, the result holds up statistically: we are 95% confident the real improvement is above zero.

Times are the typical (median) answer and costs the average per answer. The percentages are the average change measured question by question, so they can differ slightly from the ratio of the rounded figures shown.

Three things stand out:

• The smaller the model, the bigger the accuracy gain. Haiku and Sonnet gained 25 and 24 points. Opus, which is already good at searching, gained 8.

• Every model got faster and cheaper, including Opus. Opus found most answers on its own, but with the index it needed fewer lookups and read far less text to get there.

• Answers became easier to check. With the index, the agent linked the exact message, ticket or file it used far more often. With Sonnet and Opus it also made about two-fifths fewer wrong statements.

Six takeaways for anyone building with AI agents

These hold whether or not you use PipesHub.

1. Fix what the agent can see before you pay for a bigger model. Sonnet with the index (89% of facts, $0.16, 22 s) beat Opus without it (85%, $0.58, 55 s), at about a quarter of the cost and in under half the time. If answers are weak, check what information the agent can reach before you upgrade to a more expensive model.

2. A slow agent is usually busy searching, not thinking. The time we saved came from fewer lookups and less text to read, not from a faster model. Each direct lookup returns a lot of loosely matching text, and the model has to read all of it. If your agent feels slow, look at how many lookups it makes before anything else.

3. Questions that span several tools, or dig through chat history, are where agents struggle most. Searching each app directly, Sonnet found 72% of the facts for questions that needed two or three tools; with the index it found 90%. For questions that depend on Slack history, it went from 41% to 89%. Each app's own search is designed for a person browsing that one app, not for an agent pulling facts together across several.

4. Make answers checkable. Ask your agent to link the exact record it used. That is easier when every message, ticket and file has a stable link. An answer with a source can be trusted or corrected quickly; an answer without one has to be checked from scratch.

5. Measure the whole task, not one part of it. A week earlier, this same comparison showed the index was more accurate but slower. The slowness came from the connections around the search (how the agent called it and how results came back), not from the index itself. We only noticed because we timed complete answers from start to finish. Measuring search quality alone would have missed it.

6. Token efficiency matters more as agent usage grows. Frontier models are worth their price for hard reasoning, but most of an agent's work is looking things up, and that doesn't need a frontier model. With the PipesHub index, Claude Haiku linked the right source records more often than Claude Opus searching each app directly (80% against 72%), in about a third of the time (19 s against 55 s) and at about 1/60th of the cost per answer ($0.009 against $0.58). Opus still got more of the facts right overall, so the split this points to is: use a frontier model for the reasoning, and let a smaller model with the index fetch the context it needs.

When doesn't an index help?

For the strongest model the accuracy gain is small, and summaries over time are still the weak spot.

• Summaries and "what changed" questions. With Opus, the two setups tied on summaries at 80%, and were about even on whether something had changed (91% with the index, 93% without). With Haiku, the index reached only 56% on summaries, its weakest type of question. A summary needs the whole history of a project, while a search returns the few best-matching moments. This is what we are improving next.

• Top models are catching up on accuracy. Opus gained 8 points, against 24 to 25 for Haiku and Sonnet. As models get better at searching on their own, the case for an index rests more on speed, cost and checkable sources, and less on accuracy alone.

• An index is a copy. It is only as fresh as its last update from each app. For "what changed in the last five minutes", the live app is the source of truth.

What made it fast: five lessons for teams building tools for AI agents

A week before these results, the index was more accurate but slower than searching each app directly. Five changes in one week turned that around. None was clever; each removed waste between the agent and the data.

The rows for searching each app barely moved, which shows the test itself was stable. The change is in the index rows: 40–60% faster, cheaper, and more accurate.

1. Search for what the agent asked, as it asked it. Our search used to send every query through an AI model to reword it and add variations, which took about 4 seconds. But an agent already writes a precise query. Searching it exactly as written made Haiku's answers 27% faster and 21% cheaper, with fewer wrong statements.

2. Tell the agent which tools only read. Agent apps such as Claude Code run several lookups at the same time only if each tool declares that it changes nothing. Our search tool didn't declare it, so searches ran one at a time and everything else queued behind them. A lookup that takes half a second was averaging 9.5 seconds, nearly all of it waiting in line.

3. Return the part of a document that matters. Instead of sending a whole document, send the passages that match the question, and let the agent ask for more if it needs it. Together with a search that steps through a period of time month by month, this cut the cost of PipesHub's own assistant by a quarter, with the same accuracy.

4. Send status and dates with every result. Details like "merged on 2 Sep" or "resolved on 3 Jan" were missing from search results. Once we added them, the share of date facts the agent got right (with Sonnet) rose from 50% to 87%.

5. Look up exact phrases in the index, not by scanning files. Searching for an exact phrase was scanning gigabytes of stored files, which took 10 to 30 seconds. The index already held the same words and answers in a fraction of a second.

Limitations

• One company, 100 questions. This is our own data: a small company whose work runs through five tools. Larger companies, and work that is less technical, may behave differently. We are repeating the test on public data from larger organisations and will publish that too.

• AI graders. Two different AI models graded every answer against hand-written answer keys, and agreed on 95% of the facts. A third model settled the rest. People checked the answer keys, not every individual grade.

• We corrected some answer keys. When we audited the questions both setups found hardest, some of our answer keys turned out to be out of date, to combine two points into one, or to accept only one of two valid answers: 16 in a first round and 13 more in a second. We fixed them and regraded both setups, before and after. Every number here uses the corrected keys. The fixes moved each setup by a few points, and the index's lead by no more than 2.

• Connectors change. We used the official connectors available on 19 Oct 2026. Their makers keep improving them, so results for searching each app directly may get better.

• We build PipesHub, and the questions are private. The questions are about our internal work, so we can't publish them. That is why our next round runs on public data, with the questions, answer keys and test code published so you can check the numbers yourself.

  
  

Frequently asked questions

Do AI agents still need a search index if they can use MCP connectors?
In our test, yes. With each app's official connector (Model Context Protocol, the standard way AI agents plug into tools), the agent found 55% to 85% of the right facts depending on the model. With one index of all the apps, it found 80% to 92%, and it was faster and cheaper with every model.

How much faster and cheaper is an agent with a company index?
Answers took 27% to 45% less time and cost 27% to 32% less in model usage. Most of the saving came from fewer lookups (about 7 to 11 per answer instead of 12 to 16) and roughly half as much text for the model to read.

Which AI models benefit most from a company index?
Smaller models. Claude Haiku and Claude Sonnet each gained about 25 points of accuracy; Claude Opus gained 8. Every model got faster and cheaper.

Can a smaller AI model match a bigger one?

With a company index, yes, one size up: Haiku with PipesHub beat Sonnet without it, and Sonnet with PipesHub beat Opus without it, at a quarter of the cost or less.

Where does searching each app directly still do as well?
Summaries over long periods and "what changed" questions were about even for Claude Opus. An index is also a copy, so for changes in the last few minutes the live app is the source of truth.

What is PipesHub?
PipesHub is an open-source context layer for agentic AI. It indexes tools such as Slack, Jira, Confluence, GitHub and Google Drive into one permission-aware index, and any agent that supports MCP can search it.

Can I reproduce these results?
Not on this data: the questions are about our own internal work. We are repeating the test on public data from larger organisations and will publish the questions, answer keys and test code with it.

Try it yourself

Measure your own agent first. Whatever you use, a quick check tells you whether what the agent can see is what's holding it back:

1. Write 20 to 30 questions your team actually asks, each with its answer and the document, message or ticket the answer comes from.

2. Run your agent on them and record every lookup it makes, the time per answer and how much text it reads.

3. Look at the misses. If the agent found the right document but missed the detail, or needed many lookups to find it, giving it better context will help more than a bigger model.

   
   

To try PipesHub: it is open source and runs on your own servers. Install it from GitHub, connect your tools, and point any agent that supports MCP (Claude Code, Cursor, Claude.ai and others) at your installation's /mcp address. The MCP server's README has the setup steps for each app.