An agent's interface to the computer shaped its results

An interface, not just a model, in the loop

Submitted to arXiv on 6 May 2024, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering by Yang, Jimenez, Wettig and colleagues argues that a language model agent is a new category of end user, and that like a human developer working through an IDE rather than raw system calls, an agent performs better through a purpose-built interface than through direct, unmediated access to a shell. The paper reports SWE-agent reaching 12.5% pass@1 on SWE-bench, the real-world GitHub issue benchmark introduced the previous year, and 87.7% on HumanEvalFix, both described as state of the art among comparable systems at the time of submission.

What the interface actually changed

SWE-bench's own paper found that Claude 2, its best-performing model without an agent scaffold, resolved only 1.96% of the benchmark's 2,294 real issues working from a full repository. SWE-agent's contribution was not a new underlying model but a redesigned set of commands for searching, viewing and editing a codebase and for running tests, replacing raw file and shell operations with actions scoped to what the paper's authors judged a coding agent actually needs. The project's public repository describes this as leaving maximal agency to the underlying model while governing its available actions through a single configuration file, so the model still decides what to do, but the space of things it can directly do is narrower and better matched to the task.

What this does and does not establish

The reported jump from roughly 2% to 12.5% pass@1 is specific to SWE-bench's task distribution, to the models available at submission time, and to SWE-agent's particular set of commands; it does not establish that any given interface change improves any given agent's performance, only that this one measurably did on this benchmark. The repository's later record of newer models reaching higher scores under the same interface, and of a much smaller reimplementation reaching a substantial score with far less code, both suggest that the interface and the underlying model interact rather than one simply dominating the other, though the paper itself only demonstrates the first comparison directly.

Questions to carry into your own evaluation

• Is a coding agent's tool set scoped to the actions a developer would actually take, or left as raw shell access?
• Would the same model score differently on your task through a different set of tools?
• Does an agent's benchmark score reflect the model, the interface, or both together?

A model's capability and an agent's usable capability are not the same figure. SWE-agent's result is evidence that the gap between them can be closed by interface design as much as by a better model.