What has actually been measured

The run happens inside the deployment. The agent runs in the object with the instance’s own image as its machine, the model calls go out through the production queue, the grader applies the official test patch and runs the official tests in the same container before the runner releases it, and the container is never offered to the agent to destroy. The three misses are the three instances that ran out of the fifteen-minute budget; two of them left no patch and the third’s patch failed its test. Every solve landed in one file. With nothing to fetch the model works the task itself, which is why instances take four to nineteen minutes and why this number is lower than a number measured with the network open.

The object is billed for 73 % of wall clock — 4,419 s of 6,059, 527 s of it the runner grading. τ² bills the object for 3 %. Same substrate, opposite bill, and the difference is where the waiting happens: a model call leaves the object through the queue (1,103 s of provider time across 292 calls, none of it billed to the object), while a tool call runs inside it, and this task is 370 shell commands in a container. Whether long tool calls should leave the object the way model calls do is the open question below.

97 % of prompt tokens come from cache, so the gross figure overstates the bill by about thirty times: of 7.56 M, roughly 244 k were actually re-read. A cache rate is not a cost measure on its own; the instances that ran out of time re-read the most. The container is alive for 97–99 % of wall clock, and whether stopping it between calls would pay depends on run9’s compute-versus-storage rates, which are not known. run_js is not used in 372 tool calls; the two run calls are run9’s JavaScript inside the container, not the sandbox.

What it measures. The domain is a mounted plugin, the customer is a second model, and the score is the database — whether the agent left the world in the state the annotated solution leaves it in. The agent's own account of what it did is not evidence; the ending of every failed conversation is recorded beside the result so a zero can be attributed rather than guessed at.

It runs inside the deployment. The agent runs in the object, the model call goes out through the same queue production uses, and the transcript stays in the object afterwards. Only the customer is local, because τ²'s user is a second model with no access to the domain.

Two metrics, and they are not the same number. pass^k counts tasks that pass k trials in a row, out of 8; the trial count is the raw score over all 24 attempts. They can differ by ten points in one run, which is why both are reported and the /24 column is not optional.

The result: 21 of 24 trials, and 6 of 8 tasks survive all three. The three failures are tasks 1 and 5; every one ends on the simulated customer, not on a tool or the substrate. There is no Order not found and no tool failure in the run — the agent had working tools throughout.

The failure shape is a conversation that ends without the write. Task 5 fails twice: once the agent asked and the customer ended the call before answering, once it exchanged an item that should have been returned. Task 1 fails once, in a conversation the customer ended. So one of the three losses is the agent choosing the wrong action; two are conversations the customer ended.

Read this as a sample, not a score. The run before this one scored 75 % pass^1 on the same code, twelve points below this result, and the only change between them is a prompt sentence that measured as no change on the behaviour it targeted. Run-to-run variance of the shipping harness is wider than the differences this page could otherwise attribute. One run, including this one, does not fix a figure.

A gap in the agent's grounding, read from the transcripts. The agent tells the customer a refund will arrive “in 5–7 business days” in four of this run's conversations, one of them “per our policy”. The policy grants that timeline for cancellations and payment changes only (policy.md L92, L106); returns and exchanges say the customer receives an email (L126, L136), and no tool result carries the number. The agent holds the whole policy, reads a neighbouring section as backing, and states a commitment the company has not made. A general instruction to say only what the policy backs does not correct a section-level mismatch; what would is presenting the agent the section that governs the action, which is a change to how a policy is mounted rather than a prompt line.

Permissions: nothing is asked that the agent already has. Every customer question in this run, read in place, is either the confirmation τ²'s policy requires before a write or a choice only the customer can make — which variant, which card. That confirmation belongs to the benchmark's company, not to antiproton: the harness imposes no permission step, an absent mount policy allows the write, and a plugin carries one only by configuring it.

An approval-gated mount has no visible pending state. A plugin can configure approval; when it does, the operation is held with no marker that anything is waiting and no expiry. In the product a conversation resumes on the customer's next message, so it is not a dead end — but the gateway's approvals table records tool-level approvals, not “an operation is held and waiting”. Whether an approval-gated mount should show a pending state, and whether it should time out, is open.

Raw identifiers are the only affordance the agent has for a choice. The policy requires listing exact details before a write, and there is no other way to present a variant, so the run's replies quote identifiers directly — 250 table rows and 254 of them across the conversations. The content is right; the way to render a choice is missing.

The transcripts end on the agent's turn by construction, not by truncation. The runner posts a customer message to the object only when it is not the simulator's stop signal, so the final ###STOP###/###TRANSFER### is consumed by the loop and never archived. Every ending is the customer stopping or transferring; none is turn exhaustion. The zeros are the customer's, and the protocol — not the harness — is why the last turn is missing.

Substrate, not the loop, decides the bill: the object is billed for 3 % of wall clock, because a model call leaves the object through the queue. The failures are not the substrate's: no call is lost, no run stalls, no object fails to wake.

run_js is not used in this run, and the sandbox is offered on every turn.

Cloudflare bills a Durable Object for the wall clock during which it is active, with no exemption for waiting on the network. So the question is never how long a run took but how much of it the object was awake for — and a model completion is almost entirely waiting.

Provider latency is billed as Workers CPU in the queue consumer, not as object duration, which is the whole reason the model call leaves the object. What stays inside is the tool call. On τ² the tools are a mounted plugin answering in milliseconds, so the object sleeps through almost the whole run; on SWE-bench the tool is a shell in a container, and the object is awake for as long as the command runs. The loop is the same in both; the bill is a property of what the tools do.

The benchmark waits the way the console waits. Polling would measure the poller — every poll is a request that wakes the object — so the runner listens on the same hibernation-API socket the console uses, which costs the object nothing while it waits.

The benchmark starts one container per instance and hands it back when the run ends. If a release fails it now says so per instance — silence there is how two boxes were once found still running, one of them from an instance that had finished cleanly.