How Accurate Is Your Code Graph? A Benchmark of 4 Tools

An AI coding agent needs more than a list of files. Before it changes a function, it needs to know what depends on it. A code graph records those relationships: which function calls which, which file imports which, and which class extends which. It only helps if it is accurate. We benchmarked four code graph tools, PipesHub and three open-source alternatives, on nine open-source repositories. We measured how accurately each captures those relationships, how fast it builds a large repository, and whether it stays correct as the code changes.

This benchmark was run by the PipesHub team and includes PipesHub. We describe the setup, the metrics and the limits so you can judge the results in context.

What we found

Why code graph accuracy matters for AI coding agents

In a small project you remember who calls what. In a codebase of thousands of files nobody does, so engineers search, and more and more they hand the question to an AI coding agent.

Some AI coding systems use a code graph to represent relationships between functions, files, and classes. When those relationships are accurate, they can help an agent understand which parts of a codebase may be affected by a change. If the graph misses a caller, the agent may change a function and break code it never looked at. If it invents one, the agent spends its time and its context reading code that has nothing to do with the task.

How we benchmarked four code graph tools

The four tools

Most code graphs for agents are built the same way. A parser such as tree-sitter reads each file and finds the definitions and calls in it. The hard part is the next step: working out which definition each call refers to. A language’s compiler or type checker can resolve many of these relationships, but those relationships are not always available in a form that an AI coding agent can easily query and use. A code graph has to work them out without building the project, from names, imports and whatever types it can see. That step is where tools differ, and where edges go missing or go wrong.

All four tools below give an agent this map through MCP, the protocol coding agents such as Claude Code and Cursor use to call tools.

Repositories and languages

Why these repositories. They cover the four languages PipesHub links across files, with a small and a large repository in each, and every one builds under its language’s SCIP indexer, so it has an answer key. Each also stresses something specific:

• requests: small, idiomatic Python, a baseline.

  
  

• django: large, dynamic Python that calls through package re-exports (models.ForeignKey).

  
  

• excalidraw: a TypeScript monorepo whose packages import each other through path aliases.

  
  

• gin and spring-petclinic: small, conventional Go and Java.

  
  

• kubernetes/client-go: Go that is mostly generated code.

  
  

• guava: large Java with deep class hierarchies and overloaded methods.

  
  

• PipesHub backend and frontend: our own code, in production. PipesHub was developed against it, so read those two rows as home ground.

The set is hand-picked to cover these cases, not a random sample.

How we measured precision, recall and F1

We built a reference for each repository with SCIP indexers, one per language: scip-python, scip-typescript, scip-go and scip-java. SCIP (the Source Code Intelligence Protocol) records which definition each name in the code refers to, resolved the way the language’s own type checker resolves it. From that we took four kinds of relationships: function calls, imports of a module, imports of a specific symbol, and inheritance. Every tool’s output was mapped to the same names and scored against this reference by one scorer.

• Precision is the share of the relationships a tool reports that are in the reference.

  
  

• Recall is the share of the reference’s relationships the tool found.

  
  

• F1 combines the two, and is high only when both are.

An illustrative example, not a benchmark result: a reference holds 100 function calls, and a tool reports 80 calls, 72 of them correct. Its precision is 90% (72 of 80), its recall 72% (72 of 100) and its F1 80%.

Benchmark setup

• Hand-written code. The main results leave out files PipesHub skips on purpose (generated code, vendored dependencies, credentials), for every tool and on both sides of the comparison.

  
  

• Without tests. A second view also leaves out test files, by one path rule for every tool.

  
  

• Each tool at its best. Graphify 0.9.92 with its most exact export; codebase-memory-mcp 0.12.0 in full mode with its type-aware resolver; CodeGraphContext 0.6.13 with inheritance resolution on. We checked each tool’s documentation and source for settings that change accuracy, and checked that our readers lost none of its edges. CodeGraphContext’s SCIP mode was left out, because it uses the same indexers that build the reference.

  
  

• Fair counting. Java annotations (@Override) are not counted as calls, and a constructor has one name on both sides.

Reproducibility: The benchmark code and supporting artifacts are available in the PipesHub code graph benchmarks repository.

Code graph accuracy results across nine repositories

What we learned:

• When PipesHub draws an edge, it is almost always right. Its precision was the highest of the four, or within a point of it, on every repository (92.0–99.7%). When several functions share a name and the call could mean any of them, it leaves the edge out rather than guess. For an agent, a missing edge costs a search; a wrong one sends it down the wrong path.

  
  

• The bigger and more deeply typed the code, the wider the lead. On guava, with its deep class hierarchies, PipesHub scored 79.0 F1 against 55.9 for the next tool; on django, 83.9 against 76.9. Following a call through inheritance and imports is where guessing by name breaks down.

  
  

• Its one loss comes from a modelling choice. On excalidraw, a TypeScript monorepo, Graphify finds more edges (83.2% recall against 80.1%), mostly imports of exported constants, which PipesHub does not treat as symbols. PipesHub stays more precise there (95.4% against 92.0%) and trails by 0.3 points of F1.

  
  

• Generated code is left out on purpose. On kubernetes/client-go, 2,010 of the repository’s 2,551 files are generated. Counting them, PipesHub’s recall drops to 26.5% while Graphify, which indexes them, scores 82.0 F1. We lead with hand-written code because that is the code people and agents read and change.

  
  

• Most of what PipesHub misses is in tests. Leaving test files out lifts its recall on django from 77.2% to 85.6%, and on gin from 76.5% to 83.8%.

The numbers: F1 on hand-written code for each repository, where higher is better and the best score is in bold.

B1, hand-written code, 9 repositories, SCIP truth.

*It could not finish within the memory limit every tool ran under (90% of the machine).

Indexing speed, scale and memory usage

PipesHub indexed home-assistant, a 29,195-file Python monorepo with 19,626 code files, into Neo4j in 4.5 minutes, building 474,000 code edges within Neo4j’s shipped 3 GB memory limit.

PipesHub is the only one of the three that builds into a database server. Graphify and codebase-memory-mcp each write a local file for one developer; PipesHub writes a graph that a whole organization queries, with permissions on every record, and on the largest repository it still finished about 7× sooner than Graphify. codebase-memory-mcp, a compiled tool writing to SQLite, is the fastest builder. Read back from Neo4j, PipesHub’s graph scores exactly what it scores in memory, so the database adds no loss.

On Python and Go codebases PipesHub is as fast or faster: it built the 3,300-file PipesHub backend in 118 s against Graphify’s 187 s and kubernetes/client-go in 9 s against 15 s, and matched it on django (51 s and 52 s). Graphify is quicker on guava (39 s against 64 s) and on small repositories, where every tool finishes in seconds and PipesHub spends a little on each file it writes to the database. Timings: Graphify 0.9.92, the version used for accuracy; codebase-memory-mcp 0.11.0, current when the scale run was made; PipesHub as its indexing service runs, eight files in flight.

The agent-facing tools answer in 4–14 ms at the median and under 0.25 s at the 95th percentile on every repository, including finding a call path between two symbols on the largest one (63 ms at the 95th percentile).

Do code graphs stay accurate after code changes?

A code graph has to keep up with the code. Rebuilding it from scratch on every commit is slow, so tools patch only what changed. If a patch misses something, such as a deleted function or a moved import, the graph quietly goes stale and the agent acts on code that no longer exists.

We evaluated two things after each code change: how closely the updated graph matched a fresh build of the same tool, and how accurately the graph represented the reference relationships.

PipesHub’s updated graph matched its fresh build at every step and scored 94.2 F1 after the seven changes. codebase-memory-mcp also had no edges missing relative to its fresh build, but its F1 score against the reference was 80.9. Graphify’s updates missed up to 29 edges its fresh build finds, and CodeGraphContext’s up to 474. These results highlight an important distinction: update consistency and relationship accuracy measure different things.

We took a 488-file test repository through seven real changes: a rename across 15 files, a deleted file, a moved file and its importers, a new 19-file feature, a removed method and its callers, an emptied file, and a move that leaves a re-export behind. After each change, every tool updated its graph the way it does in use, and we scored it against that step’s answer key and against a fresh full build of the same code.

The same held live. Synced from GitHub through PipesHub’s connector, the first sync of the repository was ready in 150 seconds, and after each later push the graph was right and marked ready in 6–25 seconds. Across 25 measurements it held no wrong edge.

We then killed services mid-sync: the connector, the indexing service, Neo4j, and the whole application container. In every case the graph came back with no wrong edge, correct and ready again in 42–123 seconds.

Do code graphs improve AI coding agent performance?

Graph accuracy and agent performance are different things. We tested the second in three ways, and the answers differ, so we report them separately.

Ranking the file to fix, without an LLM

Given a GitHub issue, the task is to name the file that has to change. We added graph signals to PipesHub’s hybrid search (BM25 keyword search plus embeddings): the symbols and file paths an issue mentions, weighted by how it mentions them. The weights were chosen on SWE-bench Lite, then scored once on held-out data.

• SWE-bench Verified, 407 issues not in Lite: the correct-file-first rate (Acc@1, the file to fix ranked first) rose from 49.6% to 54.3% (p = 0.004).

  
  

• ContextBench, 500 tasks in 8 languages: with the same weights, the share of needed files found in the top five rose from 29.4% to 31.6% (p < 0.001). The gain was mostly in Python; TypeScript, JavaScript and Go were flat.

PipesHub’s search beside published retrievers

Without any graph signal, PipesHub’s hybrid search (BM25 plus text-embedding-3-small) ranked the right file first on 54.3% of SWE-bench Lite issues, ahead of the retrievers published with the LocAgent paper (Chen et al., 2025). Acc@5 means the file to fix is in the top five.

Published rows are on the paper’s 274-issue subset of Lite, ours on all 300 issues; on a reconstruction of that subset ours move by under 1.5 points. CodeRankEmbed stays ahead at Acc@5. This 54.3% and the Verified 54.3% above are a coincidence: different issues, a different baseline, and only the Verified figure includes the graph.

With an agent

A current model with text search and file reads already finds the file to fix on 89.7% of 300 SWE-bench Lite issues. In this experiment, adding graph tools did not produce a statistically significant improvement in agent performance: the score increased from 89.7% to 90.3% (p = 0.80). The graph-enabled setup also used 44% more prompt tokens and took 51% more time. These results do not establish that graph tools are never useful; they show that we did not measure a significant improvement in this particular setup. On ContextBench the graph tools added 0.4–1.6 points on every measure, none significant, at 33% more prompt tokens.

When the issue all but names the file, text search may already be enough. A graph should earn its keep on questions text search cannot answer: who calls a function, what breaks if it changes, or how one part of the code reaches another. That is what our next post measures.

Limitations and reproducibility

Nine repositories in four languages is a sample, not a census. Keep these limits in mind when reading the numbers.

• We chose the repositories and ran the benchmark. The selection criteria are above; two of the nine are PipesHub’s own code.

  
  

• The reference has blind spots. SCIP resolves what a type checker can see. Calls made through reflection, dynamic dispatch on untyped values or names built from strings are missing from it, for every tool alike.

  
  

• One machine. Every tool ran under the same limits: at most 90% of CPU and memory, and two hours per run. CodeGraphContext could not finish the PipesHub backend within the memory limit, so it has no score there.

  
  

• Tool versions. Graphify 0.9.92, codebase-memory-mcp 0.12.0 and CodeGraphContext 0.6.13, current in October 2026. All three are moving quickly.

  
  

• Our own test repository. The change-tracking test runs on a repository we built, so that every relationship is known after every change. The live GitHub runs use the same repository.

  
  

• One model. The agent results use one model. A weaker model may lean on the graph more.

  
  

• Graph quality is not agent performance. A more accurate graph did not, on its own, make the agent better at finding the file to fix.

Reproducing it. github.com/pipeshub-ai/codegraph-bench

What these results mean for AI coding agents

Code graph quality can be measured, on accuracy, on indexing at scale and on staying correct as code changes. In our nine-repository evaluation, PipesHub had the highest F1 score on eight repositories, and its graph matched a fresh build after each of seven changes.

These findings hold for the repositories, tool versions and settings we tested. They do not show that a more accurate graph by itself makes a better coding agent: on finding the file to fix, adding graph tools did not produce a statistically significant improvement in our setup (89.7% to 90.3%, p = 0.80). Where a graph should matter is on questions about relationships, and that is what our next post measures.

Want a code graph for your own codebase? PipesHub is open source on GitHub.

Frequently asked questions

What is a code graph?

A code graph records the relationships between parts of a codebase, such as which function calls which, which file imports which, and which class extends which. AI coding agents can use it to find the code a change affects and to see how parts of a codebase connect.

How do you measure code graph accuracy?

In this benchmark, we compared the relationships each tool produced with a reference built by language-specific SCIP indexers, which resolve names the way each language’s type checker does. We measured precision, recall and F1 for function calls, imports and inheritance.

Does a more accurate code graph make an AI coding agent better?

Not by itself. Graph accuracy and agent performance are different measures. Without an LLM, graph signals made search rank the right file first more often (49.6% to 54.3% on 407 SWE-bench Verified issues, p = 0.004). With an agent, adding code-graph tools did not make a significant difference on finding the file to fix (89.7% against 90.3%, p = 0.80).

How do code graphs stay up to date when code changes?

Most tools update only the parts of the graph that changed rather than rebuilding it. After each of seven changes we checked two things: whether each tool’s updated graph matched its own fresh build, and how accurately it represented the reference relationships. PipesHub’s and codebase-memory-mcp’s updated graphs both matched their fresh builds at every step, but they differ in accuracy: 94.2 F1 for PipesHub and 80.9 for codebase-memory-mcp.