量化智能体编码评测中的基础设施噪声
Quantifying infrastructure noise in agentic coding evals
Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. These scores are often treated as precise measurements of relative model capability and increasingly inform decisions about which models to deploy. However, we’ve found that infrastructure configuration alone can produce differences that exceed those margins. In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01). Static benchmarks score a model's output directly—the runtime environment doesn’t factor into the result. Agentic coding evals are different: models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns. The runtime is no longer a passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test. Eval developers have begun accounting for this. Terminal-Bench 2.0, for instance, specifies recommended CPU and RAM on a per-task basis in their latest 2.0 release. However, specifying resources isn't the same as enforcing them consistently. Moreover, we discovered that enforcement methodology can change what the benchmark ends up actually measuring.
SWE-bench 和 Terminal-Bench 等智能体编码基准测试常被用来比较前沿模型的软件工程能力——排行榜上名列前茅的模型之间往往仅相差几个百分点。这些分数通常被视为模型相对能力的精确衡量标准，并越来越多地被用于指导模型部署决策。然而，我们发现仅基础设施配置一项就能产生超过这些差距的差异。在内部实验中，Terminal-Bench 2.0 上资源配置最高与最低的设置之间存在 6 个百分点的差距（p < 0.01）。
How we got here
静态基准测试直接对模型的输出进行评分——运行时环境不会影响结果。智能体编码评测则不同：模型被赋予一个完整的环境，在其中编写程序、运行测试、安装依赖，并通过多轮交互进行迭代。运行时不再是一个被动的容器，而是问题解决过程中不可或缺的组成部分。两个拥有不同资源预算和时间限制的智能体实际上参加的并不是同一场测试。
We run Terminal-Bench 2.0 on a Google Kubernetes Engine cluster. While calibrating the setup, we noticed our scores didn't match the benchmark’s official leaderboard, and infra error rates were surprisingly high: as many as 6% of tasks were failing because of pod errors, most of which were unrelated to the model’s ability to solve the tasks. The discrepancy in scores came down to enforcement. Our Kubernetes implementation treated the per-task resource specs as both a floor and a hard ceiling: each container was guaranteed the specified resources but killed the moment it exceeded them. Container runtimes enforce resources via two separate parameters: a guaranteed allocation—the resources reserved up front—and a hard limit at which the container is killed. When these are set to the same value, there's zero headroom for transient spikes: a momentary memory fluctuation can OOM-kill a container that would otherwise have succeeded. To account for this, Terminal-Bench’s leaderboard uses a different sandboxing provider, whose implementation is more lenient, allowing temporary overallocation without terminating the container in order to favor infrastructural stability. This finding raised a larger question: how much does resource configuration impact evaluation scores? To quantify the effect of the scaffold, we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped. Everything else stayed constant: same Claude model, same harness, same task set. In our experiments, success rates increased with resource headroom. This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped. The drop between strict enforcement to 3x headroom (5.8% to 2.1%) was significant at p < 0.001. With more headroom, fewer containers get killed for exceeding their allocation. From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40). Most of the tasks that were crashing at 1x would have failed regardless—which is something that we observed in the data. The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution. Starting around 3x, however, this trend changes: success rates climb faster than infra errors decline. Between 3x to uncapped, infra errors drop an additional 1.6 percentage points, while success jumps almost 4 percentage points. The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. At uncapped resources, the total lift over 1x is +6 percentage points (p < 0.01). At the margins, tasks likerstan-to-pystanandcompile-compcertsignificantly improve their success rates when getting memory headroom. rstan-to-pystan compile-compcert
评测开发者已经开始考虑这一因素。例如，Terminal-Bench 2.0 在其最新的 2.0 版本中为每个任务指定了推荐的 CPU 和 RAM 配置。然而，指定资源并不等同于一致地执行这些限制。此外，我们发现执行方法本身可以改变基准测试最终实际衡量的内容。
How this affects measurement
我们如何发现了这个问题
Up to roughly 3x Terminal-Bench specs, the additional resources fix infrastructure reliability problems, namely transient resource spikes. The sandboxing provider used by the Terminal-Bench maintainers is implicitly doing this behind the scenes; the eval gets more stable without getting easier. Above the 3x mark, however, additional resources start actively helping the agent solve problems it couldn't solve before, which shows that limits can actually change what the eval measures. Tight limits inadvertently reward very efficient strategies, while generous limits are more forgiving and reward agents that can better exploit all available resources. An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret. Onbn-fit-modify, a Terminal-Bench task requiring Bayesian network fitting, some models’ first move is to install the standard Python data science stack:pandas,networkx,scikit-learn,and all their toolchain. Under generous limits, this works. Under tight ones, the pod runs out of memory during installation, before the agent writes a single line of solution code. A leaner strategy exists (implementing the math from scratch using only the standard library), and some models do default to it. Others don’t. Different models have different default approaches, and the resource configuration determines which of those approaches happen to succeed.We replicated the core finding across different Anthropic models. The direction of the effect was consistent, while the magnitude varied. The same trends seem to hold on models other than Claude, but we haven’t rigorously tested them. bn-fit-modify pandas networkx scikit-learn, We also tested whether this pattern holds on evals outside Terminal-Bench by running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each. The same effect held, though the magnitude was smaller: Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x. SWE-bench tasks are less resource-intensive, so a smaller effect is expected, but it shows resource allocation isn't neutral there either.
我们在 Google Kubernetes Engine 集群上运行 Terminal-Bench 2.0。在校准配置的过程中，我们注意到我们的分数与基准测试的官方排行榜不一致，而且基础设施错误率高得令人惊讶：有多达 6% 的任务因 Pod 错误而失败，其中大多数与模型解决任务的能力无关。
Other sources of variance
分数差异归结于执行方式。我们的 Kubernetes 实现将每个任务的资源规格同时作为下限和硬上限：每个容器被保证获得指定的资源，但一旦超出就会被终止。容器运行时通过两个独立的参数来执行资源限制：一个是保证分配（Guaranteed Allocation）——即预先保留的资源——另一个是容器被终止的硬限制。当这两个值设置为相同时，就没有为瞬时峰值留出任何余量：一次短暂的内存波动就可能导致本可成功完成的容器因 OOM（内存不足）而被终止。为了解决这一问题，Terminal-Bench 的排行榜使用了另一个沙箱提供商，其执行方式更为宽松，允许临时性的超额分配而不终止容器，以优先保证基础设施的稳定性。
Resource allocation isn't the only hidden variable. In certain configurations, time limits too start playing a role. In principle, every element of the evaluation setup can influence the final score, from the cluster health to the hardware specs, from the concurrency level to even egress bandwidth. Agentic evals are end-to-end system tests by construction, and any component of that system can act as a confounder. We have observed anecdotally, for instance, that pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents. We have not formally quantified this effect, but it illustrates a larger point: the boundary between "model capability" and "infrastructure behavior" is blurrier than a single benchmark score suggests. A model provider can shield its eval infrastructure from this by dedicating hardware, but external evaluators can't easily do the same. Public benchmarks are typically meant to measure pure model capabilities, but in practice they risk conflating them with infrastructure quirks. Sometimes this may be desirable, as it enables end-to-end testing of the entire stack, but more often it's not. For coding evals meant to be shared publicly, running at multiple times and on multiple days would help average out the noise.
这一发现引出了一个更大的问题：资源配置对评测分数的影响究竟有多大？
What we recommend
为了量化脚手架（Scaffold）的影响，我们在六种资源配置下运行了 Terminal-Bench 2.0，从严格执行每个任务的规格（1 倍，同时作为下限和上限）到完全无上限。其他所有条件保持不变：相同的 Claude 模型、相同的测试框架、相同的任务集。
The ideal scenario is to run each eval under the exact same hardware conditions—both the scaffold running the eval and the inference stack—as it would ensure perfect reproducibility across the board. However, this may not always be practical. Given how container runtimes actually enforce resources—via a guaranteed allocation and a separate hard kill threshold—we recommend that evals specify both parameters per task, not a single pinned value. A single exact spec sets the guaranteed allocation equal to the kill threshold, leaving zero margin: the transient memory spikes we documented at 1x are enough to destabilize the eval. Separating the two parameters lets you give containers enough breathing room to avoid spurious OOM kills, while still enforcing a hard ceiling that prevents score inflation. The band between them should be calibrated so that scores at the floor and ceiling fall within noise of each other. For instance, in Terminal-Bench 2.0, a 3x ceiling over the per-task specs cut infra error rates by roughly two-thirds (5.8% to 2.1%, p < 0.001) while keeping the score lift modest and well within noise (p = 0.40). That's a reasonable tradeoff: the infrastructure confounder is largely neutralized without removing meaningful resource pressure. The exact multiplier will vary by benchmark and task distribution, and should thus be reported, but the empirical calibration principle is general.
在我们的实验中，成功率随着资源余量的增加而上升。这主要是由基础设施错误率在每个步骤单调下降所驱动的——从严格执行时的 5.8% 降至无上限时的 0.5%。从严格执行到 3 倍余量之间的下降（5.8% 到 2.1%）在 p < 0.001 水平上具有统计显著性。余量越大，因超出分配而被终止的容器就越少。
Why we care
从 1 倍到 3 倍，成功分数在噪声范围内波动（p = 0.40）。在 1 倍配置下崩溃的大多数任务即使不出错也会失败——这是我们在数据中观察到的现象。智能体进行探索，触碰到资源限制而被抢占，但它本来就从未走在通往正确解决方案的路径上。
These findings have practical consequences beyond eval infrastructure. Benchmark scores are increasingly used as decision-making inputs, but this increased attention (and reliance) hasn’t always come with corresponding rigor in how they’re run or reported. As things stand today, a 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both. Without published (or standardized) setup configurations, it’s hard to tell from the outside unless interested parties go the extra mile to reproduce objective results under identical conditions. For labs like Anthropic, the implication is that resource configuration for agentic evals should be treated as a first-class experimental variable, documented and controlled with the same rigor as prompt format or sampling temperature. For benchmark maintainers, publishing recommended resource specs (as Terminal-Bench 2.0 does) can go a long way, while specifying enforcement methodology would close the gap we identified. And for anyone consuming benchmark results, the core takeaway is that small score differences on agentic evals carry more uncertainty than the precision of the reported numbers suggests—especially as some confounders are simply too hard to control for. Until resource methodology is standardized, our data suggests that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched. The observed spread across the moderate range of resource configurations in Terminal-Bench is just below 2 percentage points. Naive binomial confidence intervals already span 1-2 percentage points; the infrastructure confounders we document here stack on top of that, not within it. At the extremes of the allocation range, the spread reaches 6. A few-point lead might signal a real capability gap—or it might just be a bigger VM.
然而，从大约 3 倍开始，这一趋势发生了变化：成功率的上升速度快于基础设施错误的下降速度。
Acknowledgements
在 3 倍到无上限之间，基础设施错误额外下降了 1.6 个百分点，而成功分数却跃升了近 4 个百分点。额外的资源使智能体能够尝试只有在资源充裕时才可行的方法，例如拉取大型依赖、启动开销较大的子进程以及运行内存密集型的测试套件。在无上限资源下，相比 1 倍的总提升为 +6 个百分点（p < 0.01）。在边际情况下，像 rstan-to-pystan 和 compile-compcert 这样的任务在获得内存余量时成功率显著提升。
Written by Gian Segato. Special thanks to Nicholas Carlini, Jeremy Hadfield, Mike Merrill, and Alex Shaw for their contributions. This work reflects the collective efforts of several teams working on evaluations for coding agents. Interested candidates who would like to contribute are welcome to apply atanthropic.com/careers.
这对评测衡量意味着什么
在大约 3 倍 Terminal-Bench 规格以下，额外的资源解决的是基础设施可靠性问题，即瞬时资源峰值。Terminal-Bench 维护者使用的沙箱提供商在后台隐式地执行了这一操作；评测变得更稳定，但并没有变得更容易。
然而，在 3 倍以上，额外的资源开始主动帮助智能体解决之前无法解决的问题，这表明资源限制实际上可以改变评测所衡量的内容。严格的限制无意中奖励了非常高效的策略，而宽松的限制则更加宽容，奖励那些能够更好地利用所有可用资源的智能体。
一个编写简洁高效代码且速度很快的智能体在严格约束下会表现出色。一个用重量级工具暴力求解的智能体在宽松条件下会表现出色。两者都是合理的测试对象，但将它们合并为一个分数而不指定资源配置，会使差异——以及现实世界的泛化能力——变得难以解释。
在 bn-fit-modify 这个需要贝叶斯网络拟合的 Terminal-Bench 任务中，某些模型的第一步是安装标准的 Python 数据科学工具栈：pandas、networkx、scikit-learn 及其所有工具链。在宽松限制下，这可以正常工作。在严格限制下，Pod 在安装过程中就耗尽了内存，此时智能体甚至还没有编写一行解决方案代码。一种更精简的策略是存在的（仅使用标准库从头实现数学计算），某些模型确实默认采用了这种策略。其他模型则没有。不同的模型有不同的默认方法，而资源配置决定了哪些方法恰好能够成功。我们在不同的 Anthropic 模型上复现了这一核心发现。影响方向一致，但幅度有所变化。相同趋势似乎在 Claude 以外的模型上也成立，但我们尚未对其进行严格测试。
我们还通过在 SWE-bench 上进行交叉实验，测试了这种模式是否适用于 Terminal-Bench 之外的评测。我们在 227 个问题上（每个问题 10 个样本）将总可用 RAM 变化到基线的 5 倍。同样的效应成立，但幅度较小：分数再次随 RAM 单调上升，但在 5 倍时仅比 1 倍高 1.54 个百分点。SWE-bench 任务的资源密集度较低，因此预期影响较小，但这表明资源分配在那里也不是中性的。
其他方差来源
资源分配并不是唯一的隐藏变量。在某些配置中，时间限制也开始发挥作用。
原则上，评测设置中的每一个元素都可以影响最终分数——从集群健康状况到硬件规格，从并发级别到甚至出口带宽。智能体评测从构造上来说就是端到端的系统测试，该系统中的任何组件都可以充当混杂因素。例如，我们曾轶事性地观察到，通过率会随一天中的时段而波动，这可能是因为 API 延迟随流量模式和故障事件而变化。我们尚未正式量化这一影响，但它说明了一个更大的问题："模型能力"与"基础设施行为"之间的界限比单个基准测试分数所暗示的要模糊得多。模型提供商可以通过专用硬件来隔离其评测基础设施免受此影响，但外部评测者很难做到这一点。
公共基准测试通常旨在衡量纯粹的模型能力，但在实践中它们有可能将其与基础设施的特殊行为混为一谈。有时这可能是可取的，因为它支持对整个技术栈进行端到端测试，但更多时候并非如此。对于旨在公开共享的编码评测，在多个时段和多天运行将有助于平均掉噪声。
我们的建议
理想的方案是在完全相同的硬件条件下运行每次评测——包括运行评测的脚手架和推理栈——这将确保全面的完美可复现性。然而，这可能并不总是切实可行的。
鉴于容器运行时实际执行资源限制的方式——通过保证分配和一个单独的硬终止阈值——我们建议评测为每个任务同时指定这两个参数，而不是单个固定值。单个精确规格将保证分配设置为等同于终止阈值，没有留出任何余量：我们在 1 倍配置下记录的瞬时内存峰值足以使评测变得不稳定。将两个参数分开可以给容器足够的缓冲空间来避免虚假的 OOM 终止，同时仍然执行硬上限以防止分数膨胀。
两者之间的区间应该经过校准，使得下限和上限处的分数落在彼此的噪声范围内。例如，在 Terminal-Bench 2.0 中，将上限设置为每个任务规格的 3 倍，将基础设施错误率降低了大约三分之二（5.8% 到 2.1%，p < 0.001），同时保持分数提升适度且完全在噪声范围内（p = 0.40）。这是一个合理的权衡：基础设施混杂因素在很大程度上被中和了，同时没有消除有意义的资源压力。确切的乘数将因基准测试和任务分布而异，因此应当予以报告，但实证校准原则是通用的。
我们为什么关注这些问题
这些发现具有超越评测基础设施的实际影响。基准测试分数越来越多地被用作决策输入，但这种关注度（和依赖度）的增加并不总是伴随着运行或报告方式上相应严谨性的提升。就目前情况来看，排行榜上 2 分的领先优势可能反映的是真实的能力差异，也可能反映的是某次评测在更强的硬件上运行，甚至是在一天中更幸运的时段运行，或者两者兼而有之。在没有公开（或标准化的）配置设置的情况下，除非相关方不遗余力地在完全相同的条件下复现客观结果，否则外界很难做出判断。
对于像 Anthropic 这样的实验室而言，其启示是智能体评测的资源配置应被视为一等实验变量，以与提示格式或采样温度相同的严谨性进行文档化和控制。对于基准测试维护者而言，发布推荐的资源规格（如 Terminal-Bench 2.0 所做的那样）可以大有帮助，而同时指定执行方法将能弥补我们所发现的差距。对于任何使用基准测试结果的人来说，核心要点是：智能体评测上的小幅分数差异所携带的不确定性比报告数字的精度所暗示的更大——尤其是当某些混杂因素根本难以控制时。
在资源方法标准化之前，我们的数据表明，排行榜上低于 3 个百分点的差距在评测配置被文档化和匹配之前值得存疑。在 Terminal-Bench 中等范围的资源配置下观察到的差距略低于 2 个百分点。朴素的二项式置信区间本身就已跨越 1-2 个百分点；我们在此记录的基础设施混杂因素是叠加在这个区间之上的，而非包含在其中。在分配范围的极端情况下，差距可达 6 个百分点。
几分的领先可能标志着真正的能力差距——也可能只是一台更大的虚拟机。
致谢
由 Gian Segato 撰写。特别感谢 Nicholas Carlini、Jeremy Hadfield、Mike Merrill 和 Alex Shaw 的贡献。这项工作反映了多个团队在编码智能体评测方面的集体努力。有意向贡献的候选人欢迎在 anthropic.com/careers 申请。
SWE-bench 和 Terminal-Bench 等智能体编码基准测试常被用来比较前沿模型的软件工程能力——排行榜上名列前茅的模型之间往往仅相差几个百分点。这些分数通常被视为模型相对能力的精确衡量标准，并越来越多地被用于指导模型部署决策。然而，我们发现仅基础设施配置一项就能产生超过这些差距的差异。在内部实验中，Terminal-Bench 2.0 上资源配置最高与最低的设置之间存在 6 个百分点的差距（p < 0.01）。
静态基准测试直接对模型的输出进行评分——运行时环境不会影响结果。智能体编码评测则不同：模型被赋予一个完整的环境，在其中编写程序、运行测试、安装依赖，并通过多轮交互进行迭代。运行时不再是一个被动的容器，而是问题解决过程中不可或缺的组成部分。两个拥有不同资源预算和时间限制的智能体实际上参加的并不是同一场测试。
评测开发者已经开始考虑这一因素。例如，Terminal-Bench 2.0 在其最新的 2.0 版本中为每个任务指定了推荐的 CPU 和 RAM 配置。然而，指定资源并不等同于一致地执行这些限制。此外，我们发现执行方法本身可以改变基准测试最终实际衡量的内容。
我们如何发现了这个问题
我们在 Google Kubernetes Engine 集群上运行 Terminal-Bench 2.0。在校准配置的过程中，我们注意到我们的分数与基准测试的官方排行榜不一致，而且基础设施错误率高得令人惊讶：有多达 6% 的任务因 Pod 错误而失败，其中大多数与模型解决任务的能力无关。
分数差异归结于执行方式。我们的 Kubernetes 实现将每个任务的资源规格同时作为下限和硬上限：每个容器被保证获得指定的资源，但一旦超出就会被终止。容器运行时通过两个独立的参数来执行资源限制：一个是保证分配（Guaranteed Allocation）——即预先保留的资源——另一个是容器被终止的硬限制。当这两个值设置为相同时，就没有为瞬时峰值留出任何余量：一次短暂的内存波动就可能导致本可成功完成的容器因 OOM（内存不足）而被终止。为了解决这一问题，Terminal-Bench 的排行榜使用了另一个沙箱提供商，其执行方式更为宽松，允许临时性的超额分配而不终止容器，以优先保证基础设施的稳定性。
这一发现引出了一个更大的问题：资源配置对评测分数的影响究竟有多大？
为了量化脚手架（Scaffold）的影响，我们在六种资源配置下运行了 Terminal-Bench 2.0，从严格执行每个任务的规格（1 倍，同时作为下限和上限）到完全无上限。其他所有条件保持不变：相同的 Claude 模型、相同的测试框架、相同的任务集。
在我们的实验中，成功率随着资源余量的增加而上升。这主要是由基础设施错误率在每个步骤单调下降所驱动的——从严格执行时的 5.8% 降至无上限时的 0.5%。从严格执行到 3 倍余量之间的下降（5.8% 到 2.1%）在 p < 0.001 水平上具有统计显著性。余量越大，因超出分配而被终止的容器就越少。
从 1 倍到 3 倍，成功分数在噪声范围内波动（p = 0.40）。在 1 倍配置下崩溃的大多数任务即使不出错也会失败——这是我们在数据中观察到的现象。智能体进行探索，触碰到资源限制而被抢占，但它本来就从未走在通往正确解决方案的路径上。
然而，从大约 3 倍开始，这一趋势发生了变化：成功率的上升速度快于基础设施错误的下降速度。
在 3 倍到无上限之间，基础设施错误额外下降了 1.6 个百分点，而成功分数却跃升了近 4 个百分点。额外的资源使智能体能够尝试只有在资源充裕时才可行的方法，例如拉取大型依赖、启动开销较大的子进程以及运行内存密集型的测试套件。在无上限资源下，相比 1 倍的总提升为 +6 个百分点（p < 0.01）。在边际情况下，像 rstan-to-pystan 和 compile-compcert 这样的任务在获得内存余量时成功率显著提升。
这对评测衡量意味着什么
在大约 3 倍 Terminal-Bench 规格以下，额外的资源解决的是基础设施可靠性问题，即瞬时资源峰值。Terminal-Bench 维护者使用的沙箱提供商在后台隐式地执行了这一操作；评测变得更稳定，但并没有变得更容易。
然而，在 3 倍以上，额外的资源开始主动帮助智能体解决之前无法解决的问题，这表明资源限制实际上可以改变评测所衡量的内容。严格的限制无意中奖励了非常高效的策略，而宽松的限制则更加宽容，奖励那些能够更好地利用所有可用资源的智能体。
一个编写简洁高效代码且速度很快的智能体在严格约束下会表现出色。一个用重量级工具暴力求解的智能体在宽松条件下会表现出色。两者都是合理的测试对象，但将它们合并为一个分数而不指定资源配置，会使差异——以及现实世界的泛化能力——变得难以解释。
在 bn-fit-modify 这个需要贝叶斯网络拟合的 Terminal-Bench 任务中，某些模型的第一步是安装标准的 Python 数据科学工具栈：pandas、networkx、scikit-learn 及其所有工具链。在宽松限制下，这可以正常工作。在严格限制下，Pod 在安装过程中就耗尽了内存，此时智能体甚至还没有编写一行解决方案代码。一种更精简的策略是存在的（仅使用标准库从头实现数学计算），某些模型确实默认采用了这种策略。其他模型则没有。不同的模型有不同的默认方法，而资源配置决定了哪些方法恰好能够成功。我们在不同的 Anthropic 模型上复现了这一核心发现。影响方向一致，但幅度有所变化。相同趋势似乎在 Claude 以外的模型上也成立，但我们尚未对其进行严格测试。
我们还通过在 SWE-bench 上进行交叉实验，测试了这种模式是否适用于 Terminal-Bench 之外的评测。我们在 227 个问题上（每个问题 10 个样本）将总可用 RAM 变化到基线的 5 倍。同样的效应成立，但幅度较小：分数再次随 RAM 单调上升，但在 5 倍时仅比 1 倍高 1.54 个百分点。SWE-bench 任务的资源密集度较低，因此预期影响较小，但这表明资源分配在那里也不是中性的。
其他方差来源
资源分配并不是唯一的隐藏变量。在某些配置中，时间限制也开始发挥作用。
原则上，评测设置中的每一个元素都可以影响最终分数——从集群健康状况到硬件规格，从并发级别到甚至出口带宽。智能体评测从构造上来说就是端到端的系统测试，该系统中的任何组件都可以充当混杂因素。例如，我们曾轶事性地观察到，通过率会随一天中的时段而波动，这可能是因为 API 延迟随流量模式和故障事件而变化。我们尚未正式量化这一影响，但它说明了一个更大的问题："模型能力"与"基础设施行为"之间的界限比单个基准测试分数所暗示的要模糊得多。模型提供商可以通过专用硬件来隔离其评测基础设施免受此影响，但外部评测者很难做到这一点。
公共基准测试通常旨在衡量纯粹的模型能力，但在实践中它们有可能将其与基础设施的特殊行为混为一谈。有时这可能是可取的，因为它支持对整个技术栈进行端到端测试，但更多时候并非如此。对于旨在公开共享的编码评测，在多个时段和多天运行将有助于平均掉噪声。
我们的建议
理想的方案是在完全相同的硬件条件下运行每次评测——包括运行评测的脚手架和推理栈——这将确保全面的完美可复现性。然而，这可能并不总是切实可行的。
鉴于容器运行时实际执行资源限制的方式——通过保证分配和一个单独的硬终止阈值——我们建议评测为每个任务同时指定这两个参数，而不是单个固定值。单个精确规格将保证分配设置为等同于终止阈值，没有留出任何余量：我们在 1 倍配置下记录的瞬时内存峰值足以使评测变得不稳定。将两个参数分开可以给容器足够的缓冲空间来避免虚假的 OOM 终止，同时仍然执行硬上限以防止分数膨胀。
两者之间的区间应该经过校准，使得下限和上限处的分数落在彼此的噪声范围内。例如，在 Terminal-Bench 2.0 中，将上限设置为每个任务规格的 3 倍，将基础设施错误率降低了大约三分之二（5.8% 到 2.1%，p < 0.001），同时保持分数提升适度且完全在噪声范围内（p = 0.40）。这是一个合理的权衡：基础设施混杂因素在很大程度上被中和了，同时没有消除有意义的资源压力。确切的乘数将因基准测试和任务分布而异，因此应当予以报告，但实证校准原则是通用的。
我们为什么关注这些问题
这些发现具有超越评测基础设施的实际影响。基准测试分数越来越多地被用作决策输入，但这种关注度（和依赖度）的增加并不总是伴随着运行或报告方式上相应严谨性的提升。就目前情况来看，排行榜上 2 分的领先优势可能反映的是真实的能力差异，也可能反映的是某次评测在更强的硬件上运行，甚至是在一天中更幸运的时段运行，或者两者兼而有之。在没有公开（或标准化的）配置设置的情况下，除非相关方不遗余力地在完全相同的条件下复现客观结果，否则外界很难做出判断。
对于像 Anthropic 这样的实验室而言，其启示是智能体评测的资源配置应被视为一等实验变量，以与提示格式或采样温度相同的严谨性进行文档化和控制。对于基准测试维护者而言，发布推荐的资源规格（如 Terminal-Bench 2.0 所做的那样）可以大有帮助，而同时指定执行方法将能弥补我们所发现的差距。对于任何使用基准测试结果的人来说，核心要点是：智能体评测上的小幅分数差异所携带的不确定性比报告数字的精度所暗示的更大——尤其是当某些混杂因素根本难以控制时。
在资源方法标准化之前，我们的数据表明，排行榜上低于 3 个百分点的差距在评测配置被文档化和匹配之前值得存疑。在 Terminal-Bench 中等范围的资源配置下观察到的差距略低于 2 个百分点。朴素的二项式置信区间本身就已跨越 1-2 个百分点；我们在此记录的基础设施混杂因素是叠加在这个区间之上的，而非包含在其中。在分配范围的极端情况下，差距可达 6 个百分点。
几分的领先可能标志着真正的能力差距——也可能只是一台更大的虚拟机。
致谢
由 Gian Segato 撰写。特别感谢 Nicholas Carlini、Jeremy Hadfield、Mike Merrill 和 Alex Shaw 的贡献。这项工作反映了多个团队在编码智能体评测方面的集体努力。有意向贡献的候选人欢迎在 anthropic.com/careers 申请。
Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. These scores are often treated as precise measurements of relative model capability and increasingly inform decisions about which models to deploy. However, we’ve found that infrastructure configuration alone can produce differences that exceed those margins. In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01). Static benchmarks score a model's output directly—the runtime environment doesn’t factor into the result. Agentic coding evals are different: models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns. The runtime is no longer a passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test. Eval developers have begun accounting for this. Terminal-Bench 2.0, for instance, specifies recommended CPU and RAM on a per-task basis in their latest 2.0 release. However, specifying resources isn't the same as enforcing them consistently. Moreover, we discovered that enforcement methodology can change what the benchmark ends up actually measuring.
How we got here
We run Terminal-Bench 2.0 on a Google Kubernetes Engine cluster. While calibrating the setup, we noticed our scores didn't match the benchmark’s official leaderboard, and infra error rates were surprisingly high: as many as 6% of tasks were failing because of pod errors, most of which were unrelated to the model’s ability to solve the tasks. The discrepancy in scores came down to enforcement. Our Kubernetes implementation treated the per-task resource specs as both a floor and a hard ceiling: each container was guaranteed the specified resources but killed the moment it exceeded them. Container runtimes enforce resources via two separate parameters: a guaranteed allocation—the resources reserved up front—and a hard limit at which the container is killed. When these are set to the same value, there's zero headroom for transient spikes: a momentary memory fluctuation can OOM-kill a container that would otherwise have succeeded. To account for this, Terminal-Bench’s leaderboard uses a different sandboxing provider, whose implementation is more lenient, allowing temporary overallocation without terminating the container in order to favor infrastructural stability. This finding raised a larger question: how much does resource configuration impact evaluation scores? To quantify the effect of the scaffold, we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped. Everything else stayed constant: same Claude model, same harness, same task set. In our experiments, success rates increased with resource headroom. This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped. The drop between strict enforcement to 3x headroom (5.8% to 2.1%) was significant at p < 0.001. With more headroom, fewer containers get killed for exceeding their allocation. From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40). Most of the tasks that were crashing at 1x would have failed regardless—which is something that we observed in the data. The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution. Starting around 3x, however, this trend changes: success rates climb faster than infra errors decline. Between 3x to uncapped, infra errors drop an additional 1.6 percentage points, while success jumps almost 4 percentage points. The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. At uncapped resources, the total lift over 1x is +6 percentage points (p < 0.01). At the margins, tasks likerstan-to-pystanandcompile-compcertsignificantly improve their success rates when getting memory headroom. rstan-to-pystan compile-compcert
How this affects measurement
Up to roughly 3x Terminal-Bench specs, the additional resources fix infrastructure reliability problems, namely transient resource spikes. The sandboxing provider used by the Terminal-Bench maintainers is implicitly doing this behind the scenes; the eval gets more stable without getting easier. Above the 3x mark, however, additional resources start actively helping the agent solve problems it couldn't solve before, which shows that limits can actually change what the eval measures. Tight limits inadvertently reward very efficient strategies, while generous limits are more forgiving and reward agents that can better exploit all available resources. An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret. Onbn-fit-modify, a Terminal-Bench task requiring Bayesian network fitting, some models’ first move is to install the standard Python data science stack:pandas,networkx,scikit-learn,and all their toolchain. Under generous limits, this works. Under tight ones, the pod runs out of memory during installation, before the agent writes a single line of solution code. A leaner strategy exists (implementing the math from scratch using only the standard library), and some models do default to it. Others don’t. Different models have different default approaches, and the resource configuration determines which of those approaches happen to succeed.We replicated the core finding across different Anthropic models. The direction of the effect was consistent, while the magnitude varied. The same trends seem to hold on models other than Claude, but we haven’t rigorously tested them. bn-fit-modify pandas networkx scikit-learn, We also tested whether this pattern holds on evals outside Terminal-Bench by running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each. The same effect held, though the magnitude was smaller: Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x. SWE-bench tasks are less resource-intensive, so a smaller effect is expected, but it shows resource allocation isn't neutral there either.
Other sources of variance
Resource allocation isn't the only hidden variable. In certain configurations, time limits too start playing a role. In principle, every element of the evaluation setup can influence the final score, from the cluster health to the hardware specs, from the concurrency level to even egress bandwidth. Agentic evals are end-to-end system tests by construction, and any component of that system can act as a confounder. We have observed anecdotally, for instance, that pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents. We have not formally quantified this effect, but it illustrates a larger point: the boundary between "model capability" and "infrastructure behavior" is blurrier than a single benchmark score suggests. A model provider can shield its eval infrastructure from this by dedicating hardware, but external evaluators can't easily do the same. Public benchmarks are typically meant to measure pure model capabilities, but in practice they risk conflating them with infrastructure quirks. Sometimes this may be desirable, as it enables end-to-end testing of the entire stack, but more often it's not. For coding evals meant to be shared publicly, running at multiple times and on multiple days would help average out the noise.
What we recommend
The ideal scenario is to run each eval under the exact same hardware conditions—both the scaffold running the eval and the inference stack—as it would ensure perfect reproducibility across the board. However, this may not always be practical. Given how container runtimes actually enforce resources—via a guaranteed allocation and a separate hard kill threshold—we recommend that evals specify both parameters per task, not a single pinned value. A single exact spec sets the guaranteed allocation equal to the kill threshold, leaving zero margin: the transient memory spikes we documented at 1x are enough to destabilize the eval. Separating the two parameters lets you give containers enough breathing room to avoid spurious OOM kills, while still enforcing a hard ceiling that prevents score inflation. The band between them should be calibrated so that scores at the floor and ceiling fall within noise of each other. For instance, in Terminal-Bench 2.0, a 3x ceiling over the per-task specs cut infra error rates by roughly two-thirds (5.8% to 2.1%, p < 0.001) while keeping the score lift modest and well within noise (p = 0.40). That's a reasonable tradeoff: the infrastructure confounder is largely neutralized without removing meaningful resource pressure. The exact multiplier will vary by benchmark and task distribution, and should thus be reported, but the empirical calibration principle is general.
Why we care
These findings have practical consequences beyond eval infrastructure. Benchmark scores are increasingly used as decision-making inputs, but this increased attention (and reliance) hasn’t always come with corresponding rigor in how they’re run or reported. As things stand today, a 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both. Without published (or standardized) setup configurations, it’s hard to tell from the outside unless interested parties go the extra mile to reproduce objective results under identical conditions. For labs like Anthropic, the implication is that resource configuration for agentic evals should be treated as a first-class experimental variable, documented and controlled with the same rigor as prompt format or sampling temperature. For benchmark maintainers, publishing recommended resource specs (as Terminal-Bench 2.0 does) can go a long way, while specifying enforcement methodology would close the gap we identified. And for anyone consuming benchmark results, the core takeaway is that small score differences on agentic evals carry more uncertainty than the precision of the reported numbers suggests—especially as some confounders are simply too hard to control for. Until resource methodology is standardized, our data suggests that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched. The observed spread across the moderate range of resource configurations in Terminal-Bench is just below 2 percentage points. Naive binomial confidence intervals already span 1-2 percentage points; the infrastructure confounders we document here stack on top of that, not within it. At the extremes of the allocation range, the spread reaches 6. A few-point lead might signal a real capability gap—or it might just be a bigger VM.
Acknowledgements
Written by Gian Segato. Special thanks to Nicholas Carlini, Jeremy Hadfield, Mike Merrill, and Alex Shaw for their contributions. This work reflects the collective efforts of several teams working on evaluations for coding agents. Interested candidates who would like to contribute are welcome to apply atanthropic.com/careers.