In brief

Four AI models from four companies each worked on a single service. Asked how the wider system fits together, they got about a third of the answer right. The models were not the problem. Nobody had told them what the rest of the system looks like.

Abstract

People say AI assistants guess when they cannot see the whole system. We wanted a number for it. So we asked twenty questions about one system of sixty repositories: which services call which, who publishes and reads each event, and what stops working if a service goes down. We asked them five different ways, on four models from four companies, and marked 3,300 answers against a key we wrote by hand.

With only its own service and the ones next to it, each model found between 33% and 37% of the right answer. Connected to the same system over MCP, the same models found 90% to 97%. We did not expect the four low numbers to land so close together. That is why we are publishing this.

Giving the assistant all sixty repositories is the obvious fix. It helps, but not enough (73–84%), and it costs more to run.

1 Method

The system under test

One system of about sixty repositories. Services talk to each other over HTTP, over gRPC, and through message topics. It is not our own product and we do not name it here. We picked it because it is hard, not because it is easy: several services have almost the same name, and many calls between them never appear as plain text in any file.

The questions

Twenty questions in six groups: who depends on a service, what a service calls, who publishes and reads an event, what exists and where it lives, which service exposes a given API, and two trick questions. All twenty are in the appendix, each with its score.

How each question was asked

We asked each question twice, in two ways, because they show different things. On its own means a fresh start: one question, nothing read before it. That is the fair test of what an assistant knows when it arrives. After the other nineteen means all twenty in one sitting, which is closer to how people work. By then the model has read a lot and can read it again cheaply.

We report both. Accuracy hardly changes between them. Cost does. So §4 uses the figures for questions asked on their own, and says where the other way disagrees.

Scoring

Every question has an answer key. We wrote it by reading the system, not by asking a model. Each answer has to end with one line listing only service names, so marking is just comparing two lists. No judge model, no scoring rubric, no room to be generous. Recall is how much of the key the answer found. Wrong services are the names it added that are not in the key.

Controls

Five repetitions of everything. Each question was asked five times in each setup on each model, and the reported figure is the mean.
We turned off web search, and cut the shell it runs commands in off from the internet. Without that, one model answered a question by searching the web and quoting a marketing page.
We checked every run for reading outside its own folder. A setup meant to see five repositories that reaches sixty has leaked, and the answer no longer measures anything. Those runs were thrown away, not marked.
Failed calls are gaps, not zeros. When a provider ran out of quota, the question never reached the model. Marking those zero would have helped our own result, because the slowest setup is the one without the MCP. We left them out: 13 of 3,313.

2 Results

Table 1. Recall by information setup and model. Each cell is the mean of 100 scored answers to a question asked on its own, with nothing before it. Recall is the share of the correct answer the model named.

What the assistant could see	Claude Sonnet	Claude Opus 4.8	Gemini 2.5 Pro	OpenAI gpt-5.5
MCP only, no source codenothing checked out; the MCP server is the only source	98%	96%	88%	—
Its own service and immediate neighboursthe everyday case: five repositories on disk	34%	33%	36%	37%
Its own service, and the MCPsame five repositories, MCP connected	97%	94%	90%	92%
Every repository in the organisationall sixty on disk, no MCP	76%	73%	84%	81%
Every repository, and the MCPall sixty on disk, MCP connected	100%	95%	91%	94%

The two shaded rows are what this page is about. Only one thing changes between them: whether the MCP server was connected. The question, the model, the repositories on disk and the number of tries all stay the same.

Why not just clone everything?

It is the first thing people suggest, so it deserves a measurement and not an opinion. Here is every option, in the order people bring them up, with what each one scored and how much it had to read.

Table 2. The alternatives, measured. Tokens are the mean read to produce one answer. Answer density is the two combined: how much of a correct answer you get per 100,000 tokens.

Approach	Recall	Tokens read	Answer density
Search the code it hasgrep and read, five repositories	35%	145k	24
Clone all sixty repositoriesthe intuitive fix	78%	193k	40
Retrieval (RAG) over the codenot measured	—	—	—
Ask over MCPfive repositories, MCP connected	94%	99k	94

Look at the second row. Cloning everything loses twice over: it reads 33% more than working from one service, and still ends 16 points below the MCP. You pay more and get less. More text was never the missing piece. The assistant has plenty of text. What it lacks is something that says which text matters.

Density puts both halves in one number: 24 without the MCP, 94 with it. That is 3.9 times more. If you only remember one figure, use this one. You can score well on accuracy by reading everything, and well on cost by reading almost nothing. Neither trick works here.

3 The half recall cannot see

Recall on its own is easy to cheat. Name every service in the company and you have named all the right ones too. Recall says 100% and the answer is useless. So we counted the other direction as well.

A wrong service is one the assistant listed that does not belong in the answer. Say you ask which services call a component. The right answer is ten. The assistant names twelve: eight correct, four made up. Those four are wrong services. An engineer who acts on one of them wastes real time, checking a service that was never affected or warning a team that was never downstream.

Table 3. Wrong services per 100 answers — services named that do not belong in the answer. All three models pooled, each question asked on its own. Lower is better.

What the assistant could see	Wrong services per 100 answers
Its own service and neighbours	40
Its own service, and the MCP	8
Every repository	25
Every repository, and the MCP	10

Both numbers move together, and that is what makes the result worth something. The setup that found far more of the right services also made up far fewer wrong ones: 40 per 100 answers down to 8. Two of the twenty questions exist only to test this. They ask about things the system does not do, and the right answer is to say so.

4 Cost

Searching a codebase is not free. The assistant lists folders, opens files, and pays for every token it reads. Asking over MCP is a single call. When each question was asked on its own, the MCP came out cheaper for every model, in both comparisons.

Table 4. Cost of one answer with the MCP, as a multiple of the same answer without it. Below 1.00 is cheaper. Each question asked on its own.

Comparison	Claude Sonnet	Claude Opus 4.8	Gemini 2.5 Pro	OpenAI gpt-5.5
Own service → own service + MCP	0.69×	0.79×	0.82×	0.94×
All repositories → all repositories + MCP	0.63×	0.74×	0.54×	0.80×

We put this second because it is the weaker of the two results. When the questions run one after another, the model has already read a lot and can read it again cheaply. There the picture is mixed: one model costs slightly more with the MCP connected. Accuracy moved the same way for every model in every setup. Cost mostly did.

5 What this does not show

A result is only worth as much as what it admits.

One system. Every number here comes from one codebase of sixty repositories. A system with clearer naming would show a smaller gap. We chose the questions to be answerable rather than flattering, but we still chose them.
Cost does not always go our way. We say this in §4 instead of leaving it out. When questions run one after another, one of the three models we tested that way costs slightly more with the MCP connected.
One model per company. Each column is a single model, named in the table. Another model from the same company could land somewhere else.
We built the MCP server we are measuring. That is the obvious objection and we cannot argue it away. All we can do is show our method: we wrote the answer key from the system, the marking is mechanical, we measured the baselines instead of assuming them, and we printed the results that went against us.

A Every question, and what it scored

An average can hide almost anything, so here are all twenty questions under all five setups. Service, topic and repository names are replaced by letters, and the same letter always means the same thing. Each figure is the average of every answer behind it. The line above the table counts how many answers your current filters cover.

Table 5. Recall per question and setup. Bold marks the best setup for that question; a dash is a combination that was not run. Slice it by model, by how the question was asked, or by category, and switch the measure to wrong services.

Model

How it was asked

Where this leaves us

None of this says an assistant should stop reading code. It says that structure is the part it cannot reach by reading one repository: who calls whom, who reads what, what breaks. That is also the part it will guess at with confidence. The Go question in the appendix is the counter-example, and it is in there for the same reason as everything else.

See it on your own system How the connection works