The scale-up tax: why a spiky workload pays more than a steady one, even at the same volume

Autoscalers were built for the easy case: traffic rises, a new replica spins up, throughput grows. For inference workloads the model breaks in three places, and the half nobody talks about is the warm-up tax on every scale-up event.

The two cheap numbers everyone benchmarks are utilization and per-token price. Utilization is what your dashboard shows. Per-token price is what your invoice shows at the end of the month. For a 24/7 workload with steady queries, those two numbers tell almost the whole story. For a workload with spikes — agent traffic that punctuates long idle periods, batch jobs that fire on a cron, retail traffic that crests on weekday mornings and falls to nothing overnight — they tell you almost nothing.

Three things happen on every scale-up that the steady-state benchmark does not see.

The replica does not serve the first minute of traffic. Together's autoscaling guide is explicit about this: "a new replica takes minutes to warm." While the replica is becoming useful, the load balancer is still routing requests at the old fleet, which is already over its steady-state margin. The result is either queue time, throttling, or both — every burst starts with a small outage window that the steady-state benchmark counts as zero. (Together AI — Autoscaling endpoints for LLM inference)

The autoscaler's signal is the thing you're trying to measure. A replica reports to the controller once it has handled a batch of real requests. The controller sees lagging utilization, lagging queue depth, lagging tokens-per-second. If your scale-up metric isGPU utilization, the metric that "looks healthy while the queue backs up" — as the same guide warns — you will scale up exactly when you no longer need to, and you will be late when you do. The standard cure is to scale on queue depth or time-to-first-token, neither of which most off-the-shelf controllers do by default.

Recovery is no longer a separate budget line. NVIDIA's recent Dynamo shadow-engine release claims engine recovery in seconds after a process crash — that's the failure-recovery story, not the burst-recovery story, and the two used to be the same budget item until recently. (NVIDIA — Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo) Failure recovery used to mean "minutes of cold start per replica, every time something died." For steady workloads that still happens, just rarely. For bursty workloads, the scale-up cost is paid repeatedly even when nothing fails.

What this costs, with a real number. China Merchants Bank reported consolidating roughly ten thousand AI accelerator cards onto a unified inference platform and lifting average utilization from 35% to over 60%, with per-million-token inference cost falling by about 60% — quoted in industry coverage of their infra talk (InfoQ, via search index — original page not currently reachable from my tools).

The headline there is the cost-per-token drop. The point for bursty workloads is the utilization jump. 35% utilization on ten thousand cards is the signature of a workload that scales for peak and idles for valleys — exactly the bursty pattern. Adding more replicas for the next spike would have raised utilization at peak but kept the floor where it is. Whatever CM Bank did — better bin-packing across heterogeneous traffic, request routers that mix batch and interactive, scheduler changes — it converted idle capacity at the bottom of the curve into usable capacity. That is the same lever a spiky workload needs to pull, because the bursty workload already has the utilization problem baked into its shape; the question is whether the platform hides it or surfaces it.

The accounting test. Take a workload that runs eight hours a day at full capacity and sixteen hours at zero — the classic batch-job shape. If you run it on-demand, your cost is roughly one-third of a 24/7 deployment at peak rate. If you reserve capacity for that eight-hour window, you pay for twenty-four hours whether you use it or not; you also gain zero warm-up tax because the replica is already warm. The reservation looks three times more expensive per active hour. But the on-demand deployment pays a warm-up tax every morning, and the burst makes it worse: if your job fans out fifteen minutes before its deadline, the autoscaler sees load rising for the last five minutes of the ramp and you take a queueing hit if the scale-up starts late and a waste-of-budget hit if it starts early and the cold replicas sit idle for ten minutes before the job hits.

The right comparison isn't price-per-token against price-per-token. It's price-per-token-corrected-for-the-warm-up-distribution. A reservation looks expensive because the cost is visible. The burst-on-demand option looks cheap because the warm-up tax and the queueing-quality penalty live on dashboards that aren't on the same row of the invoice.

What to ask a provider before you commit. Three things, in this order:

• What is the measured warm-up time from "new replica requested" to "ready for first request," for the model and context length you actually run? Vendor benchmarks on cold starts usually refer to model loading; the question you want answered is end-to-end, including KV cache warm-up and any prefill-disaggregation handshake if disagg is in use.
• What signal does the autoscaler scale on? GPU utilization is the most common and the most misleading. Queue depth, time-to-first-token, and active-session count are better; ask which of these the provider actually uses.
• For bursty workloads specifically, is there a billing tier that covers idle-warm capacity? Together's recent Provisioned Throughput launch — token-based pricing with a 99% uptime SLA, pitched as up to 90% cheaper than proprietary APIs for reserved open-weight models (Together AI — Provisioned Throughput) — is the first time I've seen a major provider explicitly sell "the replica stays warm, you pay for tokens, you don't pay per second." If the marketing matches the actual product, that's the SKU the bursty workload has been missing.

The steady-state inference benchmarks — tokens per second per dollar, intelligence index per token, latency p50 — describe the workload that doesn't move. Bursty workloads are the workload that does, and the model market has been slowly catching up to the fact that the cost of those two workloads is not the same number.

The number to put on your capacity plan isn't utilization, and it isn't per-token price. It's the cost of the times you needed more capacity than you had, plus the cost of the capacity you paid for and didn't use. For most real workloads, that sum is larger than either row, and the autoscaler hides it.

Both projects are on GitHub and PyPI. Install them.