How much measured coding performance comes from the model, and how much comes from the surrounding agent software?
First, the intuition.
A model never interacts with a repository alone. It has a loop that chooses tools, formats actions, manages context, handles errors, and decides when to stop. Small design choices in that loop can amplify or suppress capability.
What the researchers actually did
Pairing Matters systematically crosses six lightweight coding harnesses with three open-source models on 100 SWE-bench Verified tasks. It holds grading and serving conditions relatively consistent and analyzes unsuccessful runs.
What the results suggest
- Within the same model, changing the harness produced differences of up to 44 percentage points in task resolution.
- Harness rankings changed across models: there was no universally winning harness for all tested pairings.
- Charged tokens per solved task varied by more than twentyfold in some comparable loop families.
How convincing is the evidence?
The full-factorial setup contains 1,800 graded runs, making the model–harness interaction itself the object of measurement. Executable task grading is stronger evidence than a self-declared completion message.
What remains uncertain?
This is not a single-versus-multi-agent experiment. The task subset, model versions, and harness implementations limit generality. It does not prove the same ranking will persist after software upgrades.
Connections worth exploring
- This is an experimental confound: a different harness can be mistaken for an architecture gain.
- Protocol design, context compression, and retry policies matter even with frozen weights.
- Reproducible agent evaluations should publish both model and harness versions.
Benchmark the system around the model, not just the model.
Lu Zhao, Christopher M. Poskitt and David Lo · Oct 23, 2026
Our analysis is not an independent replication.