Before you buy more GPUs, check whether the workload is too small for the one you have

AWS’s useful number this week is not a new model score. It is a serving result: an automatic speech recognition workload cut inference costs by 75% by running NVIDIA CUDA Multi-Process Service with NVIDIA Triton Inference Server on Amazon EC2.

That is an actionable finding because it points at a common failure mode in production inference: the model is not too expensive because the GPU is too weak. It is too expensive because each request uses only part of the GPU, and the serving stack leaves the rest idle.

The AWS post is about ASR, not chat completion. That matters. Speech requests can be bursty, uneven in duration, and small enough that one process does not naturally saturate an accelerator. But the shape of the problem is familiar across inference: a workload looks GPU-bound on a bill, while the actual device is underfilled at request time.

The usual response is to shop sideways. Move to a smaller GPU. Quantize. Batch more aggressively. Try a cheaper provider. Those may still be right answers, but the AWS example says there is another question to ask first: can multiple inference processes safely share the same GPU without stepping on each other?

MPS is not magic capacity. It is a way to let CUDA kernels from multiple processes run concurrently on one GPU rather than forcing them through isolated lanes that leave slack unused. Triton supplies the serving layer around that. In the AWS example, the combination is enough to move the cost line sharply, with the headline 75% reduction attached to the ASR setup they measured.

The practical implication is narrower and more useful than “use MPS.” If your inference process is already saturating memory bandwidth, compute, or HBM capacity, concurrency will not make a second GPU appear inside the first one. If your p95 latency is dominated by a single large decode that occupies the device cleanly, MPS may add scheduling complexity without buying much. But if your traces show low duty cycle, short kernels, small batches, or replicas sitting warm for availability rather than throughput, this is exactly the class of serving change that can pay before a model change does.

That distinction is easy to miss because utilization is often reported as a comforting average. A GPU at a respectable average utilization can still be economically bad if the peaks are brief, the queues are uneven, or the replicas are provisioned for latency isolation. Inference is sold by the hour, but users arrive as events. The bill sees the hour. The GPU sees gaps.

This is also why the number should not be generalized carelessly. AWS measured an ASR deployment on EC2 with NVIDIA MPS and Triton; the 75% reduction belongs to that setup. A long-context LLM serving path has different bottlenecks: KV cache pressure, prefill/decode imbalance, memory residency, batching policy, and time-to-first-token targets. The right lesson for LLM operators is not to expect the same percentage. It is to stop treating one model replica per GPU process as the default unit of economics.

That default is already being challenged from several directions. Together AI’s recent work on agentic scheduling claims more than 2x single-node throughput by treating an agent workflow as a schedulable program and reducing KV-cache thrashing. Its autoscaling guidance separately points out that GPU utilization can look healthy while the queue backs up, and that new replicas can take minutes to warm on dedicated inference endpoints (Together AI). NVIDIA is working the failure-recovery side, describing shadow engine recovery in Dynamo as a way to restore LLM inference capacity in seconds rather than waiting through a cold restart that reloads weights into HBM and recompiles kernels (NVIDIA).

Those are different mechanisms, but they rhyme. The serving stack is becoming the economic surface. Not just the model. Not just the chip. The scheduler, warmup path, process model, cache policy, and failure mode decide how much paid silicon turns into delivered tokens or transcripts.

For a team running inference, the action is to run a utilization audit before a procurement debate. Pull per-replica request concurrency, queue time, GPU duty cycle, memory headroom, batch size distribution, cold-start frequency, and p95/p99 latency. Look for the specific pattern AWS’s ASR example exposes: paid accelerator time with unused execution slots. If it is there, test controlled sharing on one instance class before changing the fleet.

The test should be small and mean. Fix the model, instance type, request mix, and latency objective. Compare the current process layout with an MPS/Triton configuration, or the equivalent concurrency mechanism in your serving stack. Measure cost per successful request, not just throughput. Watch tail latency and error rates, because a cheaper overloaded service is not cheaper if it creates retries upstream.

If the result is flat, you learned something useful: your bottleneck is elsewhere. If it moves, you have a cost lever that does not require a model migration, vendor migration, or quality trade. That is why the AWS result is worth paying attention to. A 75% cost cut in one ASR deployment is not a universal benchmark. It is a reminder that many inference bills are still paying for isolation they do not need and idle slices they have not measured.

Both projects are on GitHub and PyPI. Install them.