AWS describes push-based KV transfer for vLLM disaggregated inference
AWS says vLLM’s prefill and decode stages can be split across engine instances, with push-based KV transfer aiming to reduce TTFT. The post also lays out a push-versus-pull routing tradeoff for deployment design.
What happened
AWS describes vLLM disaggregated inference as separating prefill and decode onto different engine instances, which makes KV cache transfer a critical step before generation can start. The company says its push connector lets the decode instance register destination blocks early, while a background thread moves the transfer off the forward-pass critical path. AWS contrasts this with pull mode, which delays decode-instance selection until after prefill and better supports load-aware routing and QoS management. For evaluation, AWS says it used Qwen3-32B on a 1P1D deployment with Amazon EC2 P5en instances and EFA, and disabled prefix caching so each request had to retrieve the full KV cache. The performance discussion is AWS’s own reported setup and conclusion, not independent reproduction.
Why it matters
The article shows that in disaggregated vLLM, KV transfer is on the critical path for TTFT, so connector choice is not a minor implementation detail. The practical decision is whether a deployment values earlier decode selection enough to trade away some routing flexibility, because the two modes optimize different operational goals.
Engineering impact
For infrastructure teams running disaggregated vLLM, the source implies push mode is the better fit when lowering TTFT matters and early decode-node selection is acceptable. The evaluation context was Qwen3-32B on 1P1D with P5en and EFA, so the result should be treated as workload- and topology-specific, not universal.
Leadership impact
For capacity and SLO planning, this is a routing-policy decision as much as a serving-architecture decision: push mode narrows late-stage routing options, while pull mode preserves load-aware placement and QoS control. Leaders should weigh whether the organization’s priority is latency reduction or scheduling flexibility in the serving layer.
Editorial judgment
This changes an infrastructure choice: teams optimizing TTFT in disaggregated vLLM can favor push-based KV transfer when early decode-node selection is acceptable, but should keep pull mode for routing flexibility. The article is useful because the tradeoff is operational, not just architectural.
Watch next
The most useful follow-up is whether AWS’s future pull-mode threading work reduces the gap without sacrificing late routing choice. Until then, the open question is which workloads benefit enough from earlier destination setup to justify push mode, especially when bidirectional KV transfer or post-prefill routing decisions are required.
Limitations & uncertainty
The evidence is a single AWS publication with an author-reported benchmark and recommendation. It establishes the described tradeoff and evaluation setup, but not independent reproduction, broader workload generalization, or the performance of other hardware, models, or routing regimes.