Skip to content
LLM Inference
S19Distributed Serving·363 lines

The Serving Layer

Users see TTFT and inter-token latency, not FLOPs. The API layer streams SSE deltas, handles cancellation, and exports the metrics you tune against.

  • SSE streaming
  • chat templates
  • cancellation
  • TTFT/ITL/TPOT
  • load balancing

Motivation

The problem

Eighteen chapters of engine, and no way to call it. The serving layer is the part users judge you on, and a surprising number of production incidents start here. HTTP is not the hard part. The trouble is that the engine's properties leak through it in awkward ways.

Responses take tens of seconds, so they must stream. Clients disconnect mid-generation, and the engine has to notice or it leaks KV blocks. Chat models need messages rendered into a prompt with the exact control tokens they were trained on. And the metrics you export here are the only visibility you have into everything upstream.

Core idea

The solution

An OpenAI-compatible endpoint over server-sent events. Compatibility is worth more than elegance here: every client library, evaluation harness and proxy already speaks it.

the streaming endpointpython
@app.post("/v1/chat/completions")
async def chat(req: ChatRequest, raw: Request):
    prompt = apply_chat_template(req.messages, tokenizer)

    async def stream():
        try:
            async for token_text in engine.generate(prompt, req.sampling):
                if await raw.is_disconnected():
                    await engine.abort(req.id)      # free the KV blocks
                    return
                chunk = {"choices": [{"delta": {"content": token_text}}]}
                yield f"data: {json.dumps(chunk)}\n\n"

            yield "data: [DONE]\n\n"
        finally:
            # Runs on client disconnect, server shutdown and exceptions
            # alike. This is the line that stops the memory leak.
            await engine.abort(req.id)

    return StreamingResponse(stream(), media_type="text/event-stream")

The finally block is not boilerplate

If a client closes the tab mid-generation and nothing aborts the request, the engine keeps decoding into a socket nobody is reading and keeps its KV blocks allocated until it hits the token limit. Given enough impatient users this is a memory leak that presents as a capacity problem, and adding GPUs does not fix it.

Mechanics

How it works

Diagramrequest in, tokens out
REQUEST PATHHTTP POST/v1/chat/completionsvalidateparams, limitschat templatemessages → prompttokenizetext → idsengine queueS10 schedulerRESPONSE PATH — ONE PASS THROUGH PER TOKENengine step1 tokendetokenizeincremental, UTF-8 safestop checkeos, stop strings, budgetSSE deltadata: {...}\n\nrepeat until stopTHE THREE NUMBERS TO EXPORTTTFTqueue + prefill + first tokenITL / TPOTgap between tokensthroughputtokens/s across the fleetTrack percentiles, not means. A P50 of 200 ms with a P99 of 8 s is a system most users think is broken.Queue time belongs in TTFT. Measuring from "engine start" hides the exact failure you need to see.CANCELLATIONclient disconnectsabort → free KV blocksMiss this and you leakmemory on every reload.
The response path runs once per token. Everything in it must be cheap, and the detokenizer must be incremental.

Chat templates are model-specific and unforgiving

A chat model was fine-tuned on one exact serialisation of a conversation: specific role markers, specific whitespace, a specific generation prompt at the end. Llama-3, Qwen and Mistral all differ.

Get it wrong and the model still responds, slightly worse, forever. Templates therefore ship with the model, as a Jinja string in the tokenizer config rather than something hardcoded in the server. Render that template; do not concatenate the strings yourself.

The metrics that matter

  • TTFT — measured from when the request arrives, not from when the engine starts it. Queue time is the part that goes wrong under load, and excluding it hides exactly the failure you need to see.
  • ITL / TPOT — the gaps between successive tokens: ITL is the distribution of individual gaps, while TPOT is a request's average, (E2E − TTFT)/(tokens − 1). Report the distribution: a mean of 20 ms with a P99 of 900 ms is a system that visibly stutters, and the mean will never show it.
  • Throughput — aggregate output tokens per second. Your capacity number, and the one that trades against the other two.
  • Queue depth, preemption rate, cache hit rate — the leading indicators. Preemption rate climbing is the earliest sign of the thrashing from S10.

Explore

Try it

Simulatorstreaming endpoint
TTFT (measured)
—
ITL (measured)
—
output tok/s
—
elapsed
0.00 s
what the user sees

press the button

The user judges the API by two numbers only: how long the cursor sat still before the first word, and whether the words after it arrive steadily. Total tokens per second across the fleet is your metric, not theirs.

what goes down the wire

(no events yet)

Server-sent events: one data: line per token, terminated by data: [DONE]. Each delta carries only the new content, not the accumulated text; the client concatenates.

latency budget
TTFT
decode (22 tokens)

Drag concurrency to 128: throughput per GPU is far better, and both TTFT and inter-token latency get visibly worse. Turn off the prefix-cache hit with a 16k prompt and TTFT dominates everything. These three controls are the whole tuning surface a serving operator has.

The stream is real; the timings come from a latency model that responds to the controls. Run it at concurrency 1 and at 128, and watch the two numbers users care about degrade in exchange for throughput you cannot see. Then disable the prefix-cache hit with a 16k prompt. TTFT swamps everything, and no argument for S07 is clearer than that.

Build it

Implementation

Stop strings are the detail that catches people, because they are defined over text while generation happens in tokens.

code/s19_server.py (excerpt)python
class StopChecker:
    """Stop strings are text; generation is tokens. Buffer accordingly."""

    def __init__(self, stops: list[str]):
        self.stops = stops
        self.window = ""
        self.keep = max((len(s) for s in stops), default=0)

    def push(self, text: str) -> tuple[str, bool]:
        self.window += text
        for s in self.stops:
            if (i := self.window.find(s)) != -1:
                # Emit only what precedes the stop string, then finish.
                return self.window[:i], True

        if self.keep <= 1:                    # nothing can straddle a boundary
            safe, self.window = self.window, ""
            return safe, False

        # Hold back the last keep-1 chars: a stop string may span chunks.
        split = max(0, len(self.window) - (self.keep - 1))
        safe, self.window = self.window[:split], self.window[split:]
        return safe, False

Do not emit text you might have to retract

A stop string can straddle two tokens. Emit greedily, discover the stop afterwards, and the client has already rendered characters that should never have been sent; SSE has no undo. Hold back the last max(len(stop)) − 1 characters until you know they are safe.
Run it locally
Runs a real HTTP server exposing an OpenAI-compatible streaming endpoint on the standard library alone, with chat templating, stop strings that span token boundaries, cancellation and a metrics endpoint. Also runs a load test reporting TTFT and ITL percentiles.
$ python code/s19_server.py
Expect: A server on localhost:8000 that works with curl and any OpenAI client library, a demonstration that cancellation frees the KV blocks, and a concurrency sweep showing throughput rising while both user-visible latencies get worse.

Only NumPy is required — setup instructions.

Production notes

In production

  • vLLM — FastAPI + Uvicorn exposing chat, completions and embeddings, with the engine in a separate process so HTTP handling never blocks the GPU loop.
  • baseRT — one CLI that serves several models behind a single OpenAI-compatible HTTP API, which is the shape most local deployments want.
  • llama.cpp server — the same API surface in a single C++ binary; the reference for "how small can this be".
  • Routing layers — above the server, a load balancer that is prefix-cache-aware rather than round-robin, so the S07 cache is not destroyed by the deployment topology.

Exercises

  1. 1
    Implement the n>1 parameter using the copy-on-write block sharing from S06, and verify four samples cost roughly one prompt of KV rather than four.
  2. 2
    Add per-key rate limiting and a queue-depth-based admission controller that returns HTTP 429 rather than accepting a request it cannot serve within the SLO.
  3. 3
    Write a load generator that reports TTFT and ITL percentiles, then use it to find the concurrency at which your P99 inter-token latency crosses 50 ms. That number is your real capacity.

Continue

What's next

Every component now exists. S20 assembles them into one engine, benchmarks it feature by feature, and shows which of the nineteen techniques actually earned its place.

Check yourself

Questions

Answer before you look. A wrong answer tells you more than a right one, because the explanation points at the part of the chapter to reread.

Check yourselfQuestion 1 of 6

A client closes the browser tab mid-generation and nothing aborts the request. What is the consequence?

Score 0/6