tinthe.dev

OpenAPI vs. MCP: Ten Percent is not worth Maintenance

Case study - OpenAPI vs MCP; Seventy agent runs, five models, one tie.

The question on optimizing for LLMs comes often, in my circles. People want to accommodate possible customers, and more and more often the answer's in MCPs. I've had a hunch that's not quite correct, but reading benchmarks did not settle it.

The result was a tie. I built the same admin surface as a documented REST API and an MCP server, then ran seventy isolated agent workflows across five models.

Each arm scored 35/35 on the corrected task pack and 33/35 on the first pass. MCP was about ten percent cheaper on a few soft measures. Real, but nowhere near enough to justify shoehorning it into ordinary functionality so a product can wear an "AI-native" hat.

MCP failed to earn a second interface.

The decision, before the code

One report I nearly cited compared typed MCP tools with a single undocumented RPC endpoint, then called the latter a "raw API". Typed beat untyped. Shocking.

A week before the first run, I published the hypothesis: good OpenAPI should win or tie for coding-time integration; MCP's distinct value should be discovery and auth across hosts, not task completion. If the numbers disagreed, I would say so.

The test subject was Quillet, the newsletter engine running this site. (You are reading this through it.) A real small product, not a fake weather API: posts have draft, published and sent states; test sends must not touch the list; duplicate full sends return a conflict.

Quillet had a thin JSON API. I filled its list, get, update and test-send gaps, wrote an OpenAPI spec with examples and lifecycle rules, then put an MCP server over the same eight curated operations.

The implementation, including the transport correction, is public.

Making it a fair fight

Both arms used the same process, database, Basic Auth, business logic and description quality. REST routes and MCP tools called the same in-process functions. The OpenAPI agent got a base URL and the spec, with curl and short scripts allowed. The MCP agent got the tool server and no HTTP fallback.

Every cell started a new local agent in an empty directory, without ambient MCP configuration or memory. The database reset before every task, email used a no-op backend and nothing touched production. A separate oracle inspected the database after every run; the agent's closing paragraph did not count as proof.

The seven tasks went from reads and draft edits to chained publish/test-send workflows. One tested a duplicate full-send and its 409. The last was editorial: choose the most publish-ready draft and send one preview without touching the list.

Composer 2.5, Grok 4.5, Claude Sonnet 5, GPT-5.6 Terra and GLM 5.2 ran every task through both interfaces. Cheap-to-mid models, deliberately. (GLM stood in for DeepSeek, which my account does not carry.) If an integration only behaves once an expensive thinking model babysits it, that is a weak default for a normal business.

The harness failed first

My first MCP implementation was wrong: a local stdio sidecar which called Quillet's HTTP API. That would have measured "curl directly" against "curl hidden behind a tool" - a benchmark of my own plumbing. I caught it before the matrix, threw it out and mounted MCP over Streamable HTTP on Quillet itself. Same server, same state, two honest interfaces.

The first matrix then scored 66/70 - 33/35 on each arm.

Three misses came from a bad assumption in task four. The database reset between cells, but the prompt assumed a post created in task three still existed. Most agents improvised and created it; two models did not, costing three cells between them. Both interfaces got hit - two MCP, one OpenAPI.

The fourth miss was more interesting. On the OpenAPI arm, Composer found two eligible unsent posts, sent one, got confused by the shell response, pivoted to the second and sent that too. Then it reported success. The database oracle disagreed.

Three runs, same failure. The tempting story writes itself: structured tool results kept the agent honest, curl did not. Two things spoil it. Composer passed the same task on MCP, but Grok passed it on OpenAPI - so neither the interface nor the model explains the miss on its own. And my harness never logged the actual HTTP payloads, so "misread the response" is inference from the agent's prose plus the database, not a receipt.

A narrow shell-UX finding. Not proof that REST is unsafe for agents.

I tightened the fixture to one eligible post, made "retry the same slug" explicit, fixed the task-four dependency, and reran the affected cells. They passed. The append-only record still keeps the original 66/70; the corrected latest cells score 70/70.

Both corrections pushed toward the answer I had already published. That deserves saying out loud. The task-six fix helped the OpenAPI arm specifically, since the same model had passed on MCP under the original fixture, and two of the three task-four misses were MCP cells. If you want to distrust the 70/70, that is the crack to push on. The 66/70 stays in the record so you can.

The numbers

Across the 35 corrected task-by-model pairs where both arms passed:

MCP often finished a bit faster and cheaper. Likely because a structured tool result is tidier than fetching a specification, composing curl and parsing shell output.

These are non-deterministic agents, mostly one run per cell, with no claim of statistical significance. The preference changes by model and task. Call the edge roughly ten percent.

Ten percent with no success-rate advantage is a footnote, not a product strategy.

The small-business decision

This spike measured agent execution, not engineering cost. I will not pretend the token table priced development. But the cost direction is clear for a small or mid-size business which already owns an HTTP API.

The OpenAPI file runs 511 lines and documents one contract - human developers, generated clients, validation, SDK work, agents. The MCP server took 241 lines and added a second contract over the same eight operations: same curation, same semantics, plus its own auth, its own transport, its own test path.

The 241 lines sound tempting, but you'll quickly find yourself reimplementing bits that OpenAPI gives out of the box. The raw spec is shorter, but you'll much sooner feel an API spec missing than you'd feel the MCP missing. And if you've got both? Then you're keeping two contracts honest with each other. That cost does not end.

There are real reasons to pay it:

This test measured none of those. It measured whether agents finish the work, and they did, either way.

"Agents need to use us" is not on that list.

For most smaller businesses, improve the OpenAPI first. It is usually cheaper because it repairs an asset you already need, and better because humans, scripts, SDKs and agents all benefit from the same contract. If a real distribution problem later demands MCP, generate or maintain the wrapper from that source of truth.

Shoe-horning MCP into CRUD does not make the product more capable. It just sounds nice for advertising.

What the spike was worth

The decision came out in four parts. Keep the improved API - that work stood on its own. Do not expect MCP to lift coding-agent success. Add it when a named host, discovery or auth need shows up, and not before. Revisit if real customer tasks show a material gap.

What the spike left behind: usable code, a repeatable harness, transcripts, and a written record of where the test itself went wrong.

This is the shape of the research work I sell: one fixed question, enough production-adjacent code to test it, independent checks, and a decision with conditions for changing it. Not a large build disguised as discovery. Not forty pages of architecture theatre.

A second interface has to earn its maintenance. Ten percent did not make the cut.

Book a discovery call