AWS's new G7 instances won its own benchmark with half the GPUs

AWS published a benchmark on 8 September comparing three generations of its GPU instances on 30 billion parameter mixture-of-experts models. Its newest family, G7, built on NVIDIA's Blackwell RTX PRO 4500, came out ahead on throughput, on latency and on cost per token.

A company benchmarking its own newest hardware and winning is not a story. What is underneath the result is, because two of the numbers cut against the way most teams size an inference machine.

The box that won had half the GPUs#

The first test runs Qwen3-Coder-30B through the SageMaker DJL Large Model Inference container on three instances of the same 12xlarge size.

Half the accelerators, two thirds of the memory. The two-GPU machine won anyway.

AWS gives the mechanism in its own words:

MoE architectures are memory-bandwidth bound during token generation (decoding) because each token activates only a small subset of experts.

That sentence is the transferable part of the whole post. A mixture-of-experts model routes each token through a small fraction of its weights, so the speed of generation tracks how fast memory can be read, not how much of it there is. Once the model fits, extra capacity buys nothing for decoding. Bandwidth keeps paying.

Which is why the amount of memory is only the first question when you are sizing for a model, and never the whole one. Memory speed, not memory size, is what a decode step waits on, and it is a large part of why a GPU suits this work better than a CPU in the first place.

There is a second mechanism, and AWS is specific about it: only G7 has native FP4 tensor core support. G5 and G6 can run the same 4-bit NVFP4 weights, but without hardware acceleration for the format.

The cheapest instance and the fastest instance are not the same machine#

The second test is a different question asked a different way. It runs NVIDIA's Nemotron-3-Nano-30B-A3B-NVFP4 through SageMaker's inference recommendation workflow with vLLM, and asks not which instance is quickest but which configuration to deploy.

For a chat-shaped workload of 512 input tokens and 256 output tokens:

Those are two different instances, and neither of them is "the best G7". AWS is explicit that throughput was the target it set and that cost was derived afterwards, which is the honest way round to describe it.

The lesson survives the vendor. Any question of the form "which one should I run" is unanswerable until somebody says whether the thing being minimised is latency, cost per token, or total tokens per hour. Those three have different winners here, on the same model, in the same instance family, on the same day.

The same model, 2.5 times the cost per token#

Now hold the model and the family still and change only the shape of the traffic.

Same model. Same instance family. The prompt got longer, and the cost of producing one token went up roughly two and a half times, because the machine now spends much more of its time reading the question and much less writing the answer.

Anyone who prices a retrieval product off a chat benchmark is budgeting the wrong number, and the error is not small. This is the ordinary shape of what inference actually costs, and it stays invisible for as long as you only ever price the model rather than the workload.

Against the older families, AWS reports its cheapest G7 configuration at:

The cost numbers are derived, and the input is not on the page#

AWS shows its working, which is more than most benchmarks do: cost per one million output tokens is the hourly instance rate multiplied by a million, divided by output tokens per second multiplied by 3,600.

The throughput half of that is measured and published. The price half is not. The post names no hourly rate for any instance it prices, and the SageMaker pricing page builds its rate table from a live feed rather than writing it into the page. So the two dollar figures above cannot be reproduced from the post on its own.

That is not a hidden number, and it is not a criticism. It is a number you have to go and fetch, and it is also the one most likely to be different for you, because rates move with region and with whatever commitment you have signed. The formula is the genuinely useful part of that section, precisely because it is the part you can re-run against the rate you actually pay.

What this establishes, and what it does not#

Two limits, both visible in the post itself.

What does transfer, whoever runs it next, is the mechanism: for mixture-of-experts decoding, bandwidth beats capacity, and the shape of your traffic moves your cost per token further than the choice of instance does.