AWS Machine Learning Blog benchmarks OpenAI models on Amazon Bedrock by outcome, not per-token price
AWS compares cost per correct answer across deployment configurations, using Bedrock models with reasoning disabled and OpenAI API baselines at default settings. Its results illustrate why per-token prices need workload-level evaluation.
What happened
AWS Machine Learning Blog describes an open-source harness that measures correctness, billed tokens, and agent turns alongside price. Its comparison uses GPT-5.6 models on Amazon Bedrock with reasoning disabled and GPT-5.4 mini/nano baselines on the OpenAI API at default settings. AWS frames these as practical deployment configurations, not a controlled comparison of intrinsic model capability. Under those settings and its pricing assumptions, AWS reports AIME cost per correct answer of $0.0021 for gpt-5.6-luna versus $0.0139 for gpt-5.4-mini. Separately, its GDPval professional-deliverable evaluation used an 8,192-token cap that truncated some GPT-5.6 outputs. The source acknowledges small samples and that enabling reasoning changes both quality and spending.
Why it matters
Cost per successful outcome can reveal differences that per-token prices miss. AWS reports a lower AIME cost per correct answer for its luna configuration, but the providers and reasoning settings differ. The useful engineering lesson is to compare explicit deployment configurations against a workload and quality threshold, rather than treat this result as a universal model ranking.
Engineering impact
Record provider, reasoning settings, output limits, billed tokens, and task success when benchmarking candidate models. AWS’s GDPval result shows why truncation needs to be tracked separately from answer quality. For a controlled model comparison, align material settings; for a deployment comparison, make differing settings explicit and test the configuration actually intended for use.
Leadership impact
For leaders, the useful constraint is procurement and rollout policy: model defaults should be chosen against workload-specific success metrics and accepted quality thresholds, then rechecked when pricing or usage patterns change. AWS Machine Learning Blog’s reported price reduction and outcome-based ranking suggest budget forecasts should be built from observed success rates, not list prices alone.
Editorial judgment
This changes the selection question from cheapest token to cheapest successful outcome, which is a more useful internal benchmark for teams running real workloads. The result is still company-reported and cap-limited, so the operational takeaway is to rerun the same style of evaluation on local tasks before changing model defaults.
Watch next
Recheck the result on representative tasks and larger samples. For the AIME comparison, measure how reasoning settings affect both accuracy and cost. Separately, test whether a larger GDPval output allowance reduces truncation. Regional pricing and model changes also warrant a new measurement; the source’s reported configurations do not establish a permanent winner.
Limitations & uncertainty
The evidence is company-reported, benchmark-specific, and tied to stated pricing assumptions and an 8,192-token cap, so it does not independently establish universal model performance or serving economics.