A pass@1 tie is not a price equivalence. It is a routing decision.

On coding-agent benchmarks this month, two frontier models tied on pass@1 — and the cheaper one was the wrong choice for a lot of workloads. Not because the score was wrong. Because pass@1 is the wrong metric to negotiate on.

The data comes from Together AI's DeepSWE comparison of GLM-5.3 and Claude Fable 5. Same 904 rollouts, same agent harness, same pass@1. GLM-5.3 came in at $3.99 per rollout versus $21.63 for Fable 5 — a 5.4× gap. The numbers are not contested; the metric is just undercalibrated for what buyers think they're buying.

Pass@1 measures one attempt. Production runs many.

pass@1 is the fraction of problems a model solves on the first try. pass@k asks whether the model solves it within k tries, with sampling diversity doing the work. The DeepSWE pair is revealing because pass@1 collapsed to identical — and pass@4 showed GLM-5.3 ahead. The same models, the same workload, the same benchmark, and pass@1 hid a divergence that pass@4 surfaced.

For a developer running a single completion per request, the two are interchangeable and GLM-5.3 is obviously cheaper. For a coding agent that scores its own output, retries on test failure, or samples multiple candidates and judges between them, the gap is what they actually pay in extra generations.

A standard reasoning for this is variance. The "weaker on average" model is not weaker on every problem — it is wrong on a different subset. If your retry chain runs the same model twice, you recover some of that variance at twice the cost. If your retry chain runs a different model, you recover more, sometimes disproportionately so.

The cascade is the product, not the GLM discount

Together AI's more interesting result is the cascade they built on top. Route to GLM-5.3 first; escalate to GPT-5.6 Sol only on rejection. The reported figure is 85.9% pass@1 at meaningfully less than the Fable-5 spend. They do not headline "GLM-5.3 is cheaper than Fable 5" — they headline the cascade hitting 85.9% at half the cost of either frontier model alone.

That distinction matters. A two-tier cascade is not a pricing trick; it is a different serving shape. You pay an extra latency tax on every problem that gets escalated, you duplicate the prompt cost on those requests, and your idempotency story for downstream tooling has to survive the retry. Fronting this on the cheap tier only works if the cheap tier produces a verifiable signal on its own output — usually a test pass, a critic model, or a structural check — that correlates with the user's acceptance criterion.

Most production agent stacks do not have that signal in place. They send a completion to the user and hope. For those stacks, "tie on pass@1, cheaper on cost" is the right answer and GLM-5.3 wins. For stacks that do have a self-check or jury in front of the user, the cascade is the optimization and the headline cost figure is the wrong one to anchor on.

Quality index disagrees with the bench

The Artificial Analysis Intelligence Index still ranks the two models differently on aggregate reasoning — Claude Fable 5 sits in a different tier than GLM-5.3 on its published scoring. That is not a contradiction with the DeepSWE result. The Intelligence Index measures the marginal difficulty a model can solve; DeepSWE measures agentic coding, where following tool schemas, sustaining multi-turn context, and avoiding hallucinated imports are themselves part of the score. A model that hits the same pass@1 on a coding-agent benchmark with five times lower cost is, by construction, doing something different from what an aggregate index rewards.

The same logic inverts the day your workload stops looking like DeepSWE. Long-running reasoning where thinking-time is most of the cost, or open-ended generation where a single "best" answer is the point, both favor the higher-tier model in ways DeepSWE does not measure. Cascade architectures can paper over them, but they paper specifically over discrete decision points with a verifiable signal — not over any gap in subjective quality.

What to actually do with this

Three rules of thumb follow from the data, and none of them require believing any single vendor's self-reported figures.

First, treat pass@1 as a floor. If your production path runs more than one attempt, your unit cost is pass@1 price × expected attempts to success, and pass@4 lets you estimate that the chance of one-and-done is not 100% — which it never is at the frontier.

Second, measure your own cascade, not the vendor's. A "GLM-first cascade hits 85.9%" figure is from one model pair on one benchmark with one classifier deciding escalation. Substitute your classifier, your acceptance threshold, and your prompt shape and the number moves. The architecture is general; the headline number is not portable.

Third, the per-rollout cost is the wrong unit to budget against if you are running a multi-turn agent. The right unit is cost per accepted task, and that includes the escalations you would never have known about had you not instrumented them. The Together AI post is unusually honest about this — they report both the rollouts that succeeded at GLM-5.3 and the ones that needed the upstream model. A cheaper tier whose failures you cannot observe is not cheaper; it is a cost you will discover later.

The number that is actually interesting in the Together AI head-to-head is therefore not $3.99 versus $21.63. It is that two models can tie on the metric people compare on and diverge on the metric production actually pays for. The fact that the gap is large, on a benchmark as close to a real coding agent as DeepSWE gets, is the news.

Both projects are on GitHub and PyPI. Install them.