writing · benchmarks
is the mcp worth installing?
someone on reddit said they could just ask claude. so i measured it — eleven questions with no tools, then the same questions against live hm land registry.
written with the assistance of ai.
the question
i can just ask claude that
that is a direct quote from r/mcp, and it is the right question to ask about any mcp server. if the model already knows the answer, the server is ceremony. so rather than argue, i measured it — and the answer turns out to be "yes, for about half of what the server does", which is more interesting than either side of the argument.
the test
how it was run
eleven questions of the kind the server exists to answer. i wrote answers from memory first — no tool calls, one attempt, no retries — into a file on disk. only then did i derive the truth from the live endpoints. the predictions were committed before anything was queried, so the ordering is auditable rather than asserted.
one model, one run. n=1. read the shape, not the decimal places — and note that the subject is the same model family being argued about, which is the point.
the result
perfect on direction, nothing on numbers
the objection is simply correct where the answer is a direction. which of two areas is dearer, whether prices fell in the crash, which authority a postcode sits in, whether scotland is in scope — memory got all five, and installing anything to answer those is waste. asked for a specific figure it missed all six, with a median error of 11.3%.
the worst miss
the index that moved under me
asked for cornwall’s house price index in july 2022 i said 135. it is 97.5 — off by 38%. but that is not a number slightly wrong, it is the wrong base year: the uk house price index is rebased so that 2023 = 100, and january 2023 comes back as exactly 100.0. i answered on the old 2015 base, which used to be right.
the failure mode is not ignorance. it is a superseded convention held confidently — a stale fact that still parses. 135 looks like a perfectly plausible house price index, so nothing about the answer invites a second look.
the actual argument
the error is unsignalled, not just large
all six numeric answers carried the same stated confidence: low. inside that one band the errors ran from 2.7% to 38.5%. the model could not rank its own answers, which is the real case for the tool — not that memory is bad, but that its error is unbounded and comes with no signal. you cannot tell the 2.7% from the 38.5% without checking, and once you are checking you have done the work the server does.
there is also a class of question no model can answer by construction. what 52 coates avenue sold for in july 2022 is one row of thirty million; it was never memorised by anything. for those the comparison is not accuracy, it is availability.
the other question
what eighteen tools cost to keep installed
someone else in the thread made a sharper point: eighteen tools plus three prompts is real context on every turn, and the cost is the full schemas, not the names. correct, and worth measuring rather than guessing. dumping tools/list over stdio and counting:
| tokens | |
|---|---|
| 18 tool schemas | 13,964 |
| 3 prompts | 230 |
| total, every turn | 14,194 |
| share of a 200k window | 7.1% |
| names alone, if schemas were lazy | 124 |
the guess was that a couple of tools with big nested filter objects would quietly eat more than the rest combined. they do not. the largest is 1,544 tokens — 11% of the total — and even the top four come to less than the remaining fourteen. no schema nests deeper than three.
the split is the surprise: descriptions are 52% of the cost and schemas only 35%. the lever is prose, not flattening. the same thread suggested saying in each description which tools need a key — that was already true of all eight that do.
the honest part
when not to install it
the measurement cuts both ways, so here is the case against, which is stronger than i expected:
- you only need the direction. five of eleven questions needed no tool at all, and most casual questions are of that kind.
- you have one address and one question. the gov.uk page answers it in a browser in thirty seconds. the server earns its keep on volume and on joins, not on single lookups.
- you know the area. the edge is entirely in places you have never walked. ten minutes on the pavement beats every indicator here, and no dataset in the stack can represent what you learn.
- your decision is finer than the data. if you are choosing between two streets and every figure is an authority or trust average, the tool cannot tell them apart and will imply that it can.
- you would act the same either way. this is the test that settles most cases — if the answer does not change what you do, do not build the pipe.
what is left after all that is narrow but real: a specific figure, on a convention you would get wrong, over more places or more sources than you will open tabs for, where you need the derivation and not just the answer. that is the server. it is a smaller claim than "ask it anything about property", and it is the one the numbers actually support.
try it
reproduce it
re-derive all ten evaluation answers from the live endpoints
npx uk-land-registry-mcp
npm run verify-evalthe evaluation set ships in the repo as evaluation.xml, and verify-eval re-derives every expected answer from hm land registry rather than trusting the file. it passed on all ten the morning this was written — which matters, because the index gets revised for months after publication and a late registration can still move a historical record.