Coding harness study finds higher bills, uncertain gains

Changing the software around a coding model can substantially increase its bill without producing a clear improvement in completed tasks. That is the central finding of a Shandong University study reported by the-decoder.com, which compared Claude Code, OpenCode and mini-SWE-agent while holding models constant.

The researchers tested five models, using the 45 hardest tasks from SWE-bench Verified and a larger pool of 447 tasks. On the larger pool, Claude Code and mini-SWE-agent finished less than two percentage points apart for both models tested there. OpenCode was the exception: it trailed by up to nine points. These are the study’s reported results, not independently verified performance measurements.

For teams choosing a harness, the useful distinction is between overhead per step and the number of steps needed to finish. Claude Code’s system prompt and tool definitions contain roughly 16,600 tokens, compared with roughly 7,000 for OpenCode and fewer than 1,000 for mini-SWE-agent. The report says these instructions are sent again at each step. Illustrative calculation: assuming ten calls that each resend those instructions, Claude Code would submit approximately 166,000 instruction tokens, versus approximately 70,000 for OpenCode. The difference is approximately 96,000 tokens before task content, tool results or generated output. This is hypothetical token accounting, not a measured bill; caching and token prices affect the charge.

That overhead does not establish which harness is cheapest to complete useful work. In the study, Claude Code cost nearly twice as much as mini-SWE-agent with GLM-5.3-Flash, but the two cost roughly the same with Deepseek-V4-Flash because the leaner harness took more steps. The combined implication is concrete: shortlist a harness with the model you intend to deploy, and compare total spending against completed tasks. A smaller prompt alone is insufficient grounds for switching. A cheaper attempt is also less useful if its chance of completion falls, as OpenCode’s reported deficit illustrates.

Small claimed accuracy gains deserve a separate acceptance rule. The median task outcome changed on 13 percent of tasks between reruns, the same rate seen when switching harnesses. Before approving a migration on accuracy grounds, require a comparison that includes reruns of the existing setup and records both completion and cost. Keep reasoning settings consistent: disabling thinking hurt some model and harness combinations substantially. These results mainly concern one benchmark, and Opus 5 ran only in Claude Code, so they do not establish that harness choice is immaterial across models or everyday development work.