Skip to content
LLM Inference
S18Distributed Serving·186 lines

Disaggregated Prefill/Decode

Prefill is compute-bound and decode is bandwidth-bound. Run them on one GPU and neither hits its SLO, so separate them and ship the KV cache across.

  • P/D split
  • KV transfer
  • NIXL/RDMA
  • SLO targeting
  • instance ratio tuning

Motivation

The problem

Chunked prefill (S11) made prefill and decode coexist politely on one GPU. It did not make them compatible. They still want opposite things:

  • Prefill wants big batches and raw FLOPs. It runs once per request, in a burst, and holds little cache.
  • Decode wants memory bandwidth and a large KV pool. It runs thousands of times per request and must never be interrupted.

Colocated, every knob is a compromise. Raise the token budget to speed up prefill and inter-token latency degrades. Raise concurrency to improve decode throughput and the KV pool crowds out prefill batching. The two SLOs, time-to-first-token and inter-token latency, are coupled through a single set of parameters, so you cannot fix one without moving the other.

Core idea

The solution

Stop sharing. Run prefill on one pool of machines and decode on another. When prefill finishes, ship the KV cache across the network to a decode instance and hand off.

Now each pool is tuned for one job, each SLO has its own scaling knob, and neither workload can interfere with the other. If TTFT is violating its SLO, add prefill nodes; if inter-token latency is, add decode nodes.

the handoffpython
async def serve(request):
    p = router.pick_prefill_instance(request)     # ideally cache-aware
    d = router.pick_decode_instance(request)

    # Tell the decode instance to allocate blocks BEFORE prefill starts,
    # so the transfer can begin the moment the first layer is done.
    handle = await d.reserve_blocks(request.prompt_len)
    kv_ref = await p.prefill(request, push_to=handle)

    async for token in d.decode(request, kv_ref):
        yield token
    # The KV never round-trips through the router. It moves GPU to GPU.

The transfer is where the engineering goes

An 8B model with GQA produces about 128 MB of KV per 1,000 prompt tokens. A 32k-token prompt is 4 GB that has to reach another machine before the first token can be emitted. Over 100 Gb Ethernet that is roughly 300 ms of added TTFT. A single 400 Gb/s RDMA NIC is 50 GB/s, so the transfer still takes about 80 ms; striping it across a node's eight NICs (~400 GB/s aggregate) gets it down to about 10 ms. Disaggregation is therefore an RDMA feature. Without one, the transfer costs more than the interference it removes.

Mechanics

How it works

Diagramprefill pool, KV transfer, decode pool
TWO PHASES, OPPOSITE APPETITESprefillcompute-bound · wants FLOPs · one big burst per requestdecodebandwidth-bound · wants HBM and KV capacity · thousands of tiny stepsRun both on the same GPU and each interferes with the other: prefill spikes inter-token latency,and decode's resident KV cache limits how much prefill you can batch.DISAGGREGATED SERVINGrouterpicks a pairPREFILL POOLGPUGPUGPUtuned for compute:large batches, big chunks,small KV poolKVRDMA / NIXLDECODE POOLGPUGPUGPUGPUtuned for bandwidth: huge KV pool,maximum concurrency, small steps— nothing ever interrupts a decodestream outWHAT IT COSTSThe whole KV cache must cross the network: 128 MB per 1k tokens for an 8Bmodel. Only viable over RDMA, and it adds directly to TTFT.Layer-by-layer streaming hides most of it behind the remaining prefill layers.WHAT IT BUYSTTFT and inter-token latency become independently tunable: you scale thepool that is violating its SLO instead of the whole fleet.Worth it above a few nodes; pure overhead below that.
The router picks a pair, the cache moves GPU-to-GPU over RDMA, and each pool is configured for exactly one workload.

Hiding the transfer

The transfer does not have to happen all at once at the end. Layer i's KV is final as soon as layer i has run, so it can be sent while layer i+1 computes. With layer-by-layer streaming, all but the last layer's transfer is hidden behind prefill compute.

NIXL and vLLM's KV connector interface exist for exactly this: a transfer engine that overlaps with computation rather than following it.

Choosing the ratio, and why it is hard

The prefill:decode ratio depends on the workload shape. Long prompts with short answers (classification, extraction, RAG) are prefill-heavy. Short prompts with long answers (agents, reasoning chains) are decode-heavy. A 1:3 split that is right at 9am can be wrong at 3pm.

Production systems therefore make the ratio dynamic, reassigning instances between roles as traffic shifts. That machinery, together with the RDMA fabric, is why disaggregation only pays above a certain fleet size. Below a few nodes it is pure overhead, and chunked prefill on one pool is the better answer.

Explore

Try it

Simulatordisaggregated vs colocated, against SLOs
deployment
TTFT
1210 ms
inter-token latency
9.0 ms
both SLOs met
yes
KV transferred
512 MB · 10 ms
request path

prefill pool · 3 GPU

→

KV transfer · 10 ms

→

decode pool · 5 GPU

Prefill runs on machines tuned for compute, decode on machines tuned for bandwidth and KV capacity, and the cache moves between them over the interconnect. Each SLO is now controlled by its own pool.

split ratio sweep · green meets both SLOs
1P
2P
3P
4P
5P
6P
7P
  • both SLOs met
  • violates an SLO

Bar height is TTFT. Too few prefill GPUs and the queue explodes; too few decode GPUs and inter-token latency does. The green band is narrow, it moves whenever the workload mix changes, and keeping the deployment inside it is the actual operational cost of disaggregation.

Set a prompt-heavy workload in colocated mode: inter-token latency blows past its SLO because prefill keeps displacing decode steps. Switch to disaggregated and tune the split until both bars go green. The feasible band is narrow. Then drop the KV link bandwidth and watch TTFT fail instead, because the transfer has become the bottleneck. That is the trade this architecture makes.

Build it

Implementation

code/s18_disaggregation.py (excerpt)python
class LayerwiseKVSender:
    """Overlap the KV transfer with the prefill that is still running."""

    def __init__(self, transport, n_layers):
        self.transport, self.n_layers = transport, n_layers
        self.inflight = []

    def on_layer_done(self, layer: int, k, v, dest_blocks):
        # Layer i's KV is final the moment layer i finishes. Send it now,
        # while layers i+1..L are still computing.
        self.inflight.append(self.transport.send_async(layer, k, v, dest_blocks))

    def finish(self):
        for fut in self.inflight:
            fut.wait()          # only the LAST layer is actually exposed

Reserve destination blocks before prefill starts

If the decode instance allocates its blocks only after prefill completes, the transfer cannot begin early. Worse, the decode pool might be out of memory, and you have then wasted an entire prefill. Reserve first, prefill second. The reservation also gives the router real backpressure: if no decode instance can reserve, do not admit the request at all.
Run it locally
Simulates colocated and disaggregated deployments on a common GPU-seconds cost model with an explicit KV-transfer term, reports TTFT and ITL against configurable SLOs, and sweeps the prefill:decode ratio to find the feasible region.
$ python code/s18_disaggregation.py
Expect: Colocated violating the inter-token latency SLO under prompt-heavy load while a 7P:1D disaggregated split meets both, a narrow feasible band in the ratio sweep, and a bandwidth sweep where the transfer alone decides pass/fail at 32k prompts.

Only NumPy is required — setup instructions.

Production notes

In production

  • DistServe and Splitwise — the papers that established the idea and its goodput argument.
  • vLLM — a KV connector interface with NIXL and LMCache backends; disaggregated serving is supported through the production stack rather than the single-process engine.
  • NVIDIA Dynamo — a full disaggregated serving framework with dynamic role reassignment between prefill and decode workers.
  • Mooncake — takes it further: a KV-cache-centric architecture with a shared pooled cache that any decode instance can read, merging disaggregation with global prefix caching.

Exercises

  1. 1
    Compute the crossover bandwidth: at what interconnect speed does the KV transfer cost more TTFT than the prefill interference it removes? The answer depends on prompt length, so plot the curve.
  2. 2
    Implement layer-by-layer streaming and measure how much of the transfer is hidden behind prefill compute.
  3. 3
    Add a dynamic controller that reassigns instances between pools based on measured SLO violations, and show it tracking a workload that shifts from prefill-heavy to decode-heavy mid-run.

Continue

What's next

The engine is fast and it scales. Users still cannot talk to it. S19 builds the serving layer: an OpenAI-compatible streaming API, chat templates, cancellation, and the metrics you actually tune against.

Check yourself

Questions

Answer before you look. A wrong answer tells you more than a right one, because the explanation points at the part of the chapter to reread.

Check yourselfQuestion 1 of 6

Why does chunked prefill not remove the need for disaggregation?

Score 0/6