When an agent fails to finish useful work, adding planning, tools, or a reviewer is an understandable response. Those additions may help. Their presence alone, however, does not establish an improvement in success rate. Start by identifying where work stops and what changing that point would cost.
Two recent AI Times reports are useful to read together: a comparison of harness costs and Microsoft's Agensh collaboration experiment. One covers research that found little clear performance gain from changing harnesses. The other describes higher test-pass rates with more agents. They changed different variables.
For a builder, that creates separate questions. Does the execution environment around the same model need more machinery? Is there enough independent work to benefit from concurrent workers? The former calls for examining context and failure recovery. The latter calls for examining task boundaries and integration work.
Harness complexity and agent count are separate dimensions
A harness is the execution environment around a model: system instructions, tool definitions, delivery of tool results, conversation management, and recovery after interruptions. Collaboration introduces further decisions about ownership, sharing findings, verification, and integration.
You can keep each worker's tools and context small while running several workers. A single worker can also receive long instructions, many tools, and several review stages. The number of agents on a screen is an incomplete description of execution cost or operating difficulty.
| Question | Harness comparison | Agensh research |
|---|---|---|
| What changed? | The harness around the same model | The number of workers using the same model |
| What was the work? | Fixing issues in existing repositories | Rebuilding reference program behavior |
| What was observed? | Run variation, success rate, and token cost | Test-pass rates and collaboration within a time limit |
| What should builders examine? | Whether extra context and recovery help | Whether parallel progress justifies coordination work |
These distinctions follow the studies' harness experiment design and Agensh evaluation. Putting them on a single leaderboard would obscure what each comparison measures.
What changed when the harness changed?
What Does a Harness Buy? Tokens, Mostly compared Claude Code, mini-SWE-agent, and OpenCode. It divided 492 offline-compatible SWE-bench Verified tasks into a hard subset of 45 and a remaining pool of 447. Five models ran on the hard subset; two locally served Qwen models ran on the larger pool. Opus was an anchor inside Claude Code only, rather than part of a complete comparison across harnesses.
On the larger pool, Claude Code and mini-SWE-agent met the study's equivalence margin of five percentage points. On the hard subset, switching harnesses and rerunning an unchanged configuration both produced a median task outcome flip rate of about 13%. That makes repeat-run variation relevant to interpreting the comparison. It does not establish that all harnesses always perform alike. See the abstract and results for the scope.
For the same model and tasks, applying a common price list to token usage produced a cost spread of up to roughly 3×. The monetary analysis covered DeepSeek-V4-Flash and GLM-5.3-Flash. It did not turn local-model tokens and subscription usage into equivalent actual invoices. The authors provide data and analysis code.
The budget also needs qualification. Each trial allowed 300 model calls, but wall-clock limits differed between harnesses on the local Qwen runs. Versions and default behaviors were tied to the experiment. Check those conditions before applying the result to a current release or another workload. The setup and Appendix A document them.
Cost accumulates between the first and last call
The cost analysis separates initial context, transcript growth, and total calls. System instructions and tool schemas consume context before a worker reads a task file. Repeatedly sending retained history adds another term. A lightweight harness can also take more steps on a particular model. The cost section explains why the smallest initial context does not guarantee the lowest total cost.
Use that breakdown in your own evaluation. Record the first input as system instructions, tool descriptions, and task content. Then track what file contents, tool outputs, and model records add. Expensive tool definitions and repeated file reads require different remedies.
Keep uncached input, cached input, and output separately. A single input-token total makes repricing difficult. Also distinguish provider-reported usage from actual billing. A long cached transcript and a short uncached transcript cannot be compared reliably through token totals alone.
Apply the same discipline to planning and review. Did more planning reduce rework? Did a reviewer identify a concrete defect? How much context was sent again to request that review? Give each stage an explicit failure it is meant to reduce, so an increase in cost has a purpose you can evaluate.
Examine cheap, short runs as well. Some finish efficiently; others abandon the task early. The existence of an output file is a weak completion criterion. Record an independent check of expected behavior alongside the reason execution stopped.
When is a simple configuration sufficient?
Start by writing down the work boundary. Clear inputs, a narrow change scope, and a reproducible completion check make a small execution loop a useful baseline: read, change, verify, and revise after failure. Verification belongs in that baseline.
Suppose a team is evaluating small bug fixes. Before adding a reviewer, classify existing failures. Did the worker misunderstand the requirement, miss a relevant file, or choose the wrong verification command after making a correct change? Calling every failure a planning problem makes the value of an added stage difficult to assess.
Ask the same question about tools. If a shell and file access already cover the work, what error would a specialized tool prevent? A tool may still be useful when the workflow needs restricted paths or structured errors. Ground the decision in the job it helps complete.
Evaluate recovery against specific stop reasons: output limits, context exhaustion, tool errors, and premature completion. Define which states can be retried, what progress survives, and what must be verified afterward. Sending the same prompt again for every failure can conceal the cause.
Permission checks and approval requirements serve their own purpose. A quality-oriented planning stage and a control on external writes should be evaluated against their respective requirements. Simplify redundant steps whose benefit is unclear while preserving the access and action boundaries the work requires.
What did Agensh show with more agents?
Microsoft's Agensh paper used GPT-5.6-sol at high reasoning effort with the Copilot harness. It evaluated five difficult ProgramBench tasks with no Internet access and a six-hour time budget. Workers reconstructed program behavior from reference execution and bundled documentation.
Under those conditions, mean final test-pass rate rose from 19.31% with one agent to 28.78% with 128. That is 9.47 percentage points, or approximately 49% relative improvement. Test-pass rate is distinct from the share of whole tasks completed. Only Pandoc was extended to 1,024 agents; its score rose from 33.89% to 55.06%. The official repository summary gives the conditions with the numbers.
An equal time budget is not an equal cost budget. More workers require accounting for total model usage and execution resources. Workers were activated gradually, so multiplying head count by the time limit would not establish actual usage either. These results alone do not establish cost efficiency or superiority over central orchestration. See the experiment details.
Agensh adds collaboration above each worker's underlying harness. Shared work, messages, and reusable findings support claiming scope, verifying contributions, and integrating progress. Decentralized allocation still requires coordination infrastructure. The architecture section distinguishes those layers.
The useful adoption question is whether your workload contains enough separable work. Does each step depend on the previous result, or are there parts workers can investigate or implement independently? Without that distinction, adding workers leaves the intended source of parallel progress unclear.
Conditions worth evaluating for multiple workers
A candidate workload should let you describe independent boundaries. Consider separate input formats to investigate, or modules behind an agreed interface. Write completion criteria for each contribution and for the integrated result. Difficulty defining those criteria is a reason to clarify requirements first.
Include shared files and execution resources in the boundaries. Workers editing different code paths may still compete for data files, test ports, or build outputs. Separate concurrency-induced failures from implementation failures in the evaluation. That distinction helps decide between fewer workers and better resource isolation.
Information sharing also deserves measurement. Compare sending every finding on every call with retrieving relevant findings when needed. Record what was observed and the conditions under which it holds. An unverified assumption presented as a shared fact can lead several workers in the wrong direction.
Check integration separately from individual success. Contributions that pass alone can fail together. Record which revisions were verified and what ran after integration. For an adoption decision, the changes present in the final working result matter more directly than the volume of generated code.
Make the time constraint explicit. A workload that can wait may prioritize cost; a fixed deadline may justify spending within an agreed budget to finish earlier. Choose that objective before running the comparison. Switching to whichever axis looks favorable afterward weakens the decision.
A measurement sheet before adoption
The following is a proposed evaluation approach for builders. Use matched inputs and completion criteria to compare candidates, rather than treating a paper's score as a forecast for your service.
| Measure | Record | Decision it supports |
|---|---|---|
| Completion quality | Agreed checks, missing requirements, regressions | Distinguish declared completion from verified completion |
| Usage and cost | Uncached and cached input, output, tool and execution costs | Separate common-list repricing from actual charges |
| Elapsed time | Start through final verification and integration | Distinguish an early response from finished work |
| Failure and recovery | Stop reason, retries, retained progress | Identify the failure a change should address |
| Coordination work | Duplicate work, conflicts, review and integration waits | Locate limits on useful concurrency |
| Human involvement | Time and reasons for corrections, approvals, and recovery | Expose work outside automated execution costs |
Include the cost of failed attempts. Averaging only successful runs removes the retries needed to reach them. When reporting cost per completed task, show the evaluation batch's full usage and define success. If a batch has no successes, report that failure instead of inventing a finite cost per success.
Measure the whole time path. Tool execution, rate-limit waits, review queues, and conflict resolution sit outside response generation. Examine slow runs alongside averages. For human involvement, distinguish changed requirements from repairs to agent mistakes.
Include both everyday work and likely failure cases. Normal inputs alone make recovery hard to evaluate; extreme cases alone can hide routine cost. Give each configuration the same work and fix the completion criteria beforehand. Keep failures visible in the results.
An experiment sequence for adopting a change
Preserve the current small configuration and rerun the same work first. You need some understanding of variation in outcomes and cost before interpreting a new configuration. Pair results by task as well as looking at averages. This helps distinguish repeatable fixes from runs that happen to fail on different tasks.
Next, choose candidates with a clear purpose. Evaluate context reduction, failure recovery, and worker-count changes separately when possible. If model and reasoning settings change too, describe the outcome as a result of that combination. Attribution should match what changed.
Choose task count and repetitions around the difference you need to detect. For a small evaluation with a narrow score gap, show individual outcomes and uncertainty, then decide whether more evaluation is warranted. Neither dismissing every difference nor selecting the best single run provides a dependable baseline.
Set adoption criteria across minimum quality, allowed cost, completion time, and human involvement. Lower cost cannot compensate for failing a mandatory check. If faster completion requires more operator recovery, expose that tradeoff. Link each added stage to the evidence that justifies keeping it.
Keep rollback criteria after adoption. New workloads and model or harness versions may change behavior. Preserve the prior configuration and usage records for comparison. The conditions under which a design was selected are more useful than a score detached from those conditions.
Decide what needs to improve before adding complexity
Identify whether you need fewer execution failures or more capacity for independent work. Excess context may be the problem; integration of verified results may be the bottleneck instead. A shared success criterion, together with cost, elapsed time, and quality after integration, makes the value of an addition easier to assess.
A useful harness has an explainable contribution to finishing the work. Start with a small baseline, choose the failure or waiting time you want to reduce, and see whether the improvement repeats. The conditions under which a design helps are the evidence for the next decision.
Wrapping up
The Hypemarc Builder Community will keep sharing research and practical criteria for building and operating agents. We'll continue gathering useful resources to share with fellow builders.
- Open chat: AI Builders
- Adoption advice and inquiries: Contact form
- Get the next post: Blog RSS · AI resource library