← All field notes

AI infrastructure · Model serving · Production systems

Serving the token: where latency, memory, and inference economics meet

A field guide to prefill and decode, continuous batching, KV-cache pressure, quantization, speculative decoding, admission control, observability, and cost per verified outcome.

Training creates a capability. Serving decides whether that capability can answer within an SLO, survive a traffic spike, fit inside finite memory, preserve acceptable quality, and produce enough business value to justify every generated token.

The serving problem is therefore not “put the model behind an endpoint.” It is an operating system for scarce accelerators, variable sequences, shared state, and incomplete demand forecasts.

A token is not successful because it was generated. It is successful when the complete task meets its quality, latency, safety, and economic contract.

One request, two compute regimes

Autoregressive inference has two distinct phases.

Prefill processes the prompt and builds the key-value cache for all prompt tokens. It exposes substantial parallelism and is often compute-intensive. Long prompts, retrieved documents, tool transcripts, and few-shot examples make this phase larger.

Decode generates new tokens one step at a time. Each step reuses the KV cache, performs a relatively small amount of work per sequence, and waits for the previous token. It is often constrained by memory movement and scheduler efficiency rather than raw arithmetic alone.

These phases create different user-facing metrics:

  • Time to first token is heavily influenced by queueing and prefill.
  • Inter-token latency describes the cadence of decode.
  • End-to-end latency includes queueing, prefill, decode, tool work, and application overhead.
  • Throughput measures useful tokens or completed requests over time.
  • Tail latency reveals what happens to the slowest important requests.

Optimizing only one metric can damage the others. A scheduler can increase aggregate throughput by assembling larger batches while making an interactive user wait. A tiny batch can improve one request's latency while leaving accelerator capacity idle.

Architecture / inference runtime

The first token and the next token are different jobs

FIG 01 - MOTION
Request-to-token serving architecture A request passes through routing and admission, then separate prefill and decode phases share model weights and KV cache before streaming tokens through verification and telemetry. ONE TOKEN / ITERATION REQUESTTask + contextroute + tenant + SLO ROUTEServing gatewaymodel + adapter + policy ADMITSchedulerqueue + fairness + memory PREFILLProcess promptparallel + compute bound DECODEGenerate tokenserial + memory bound WORKING SETPaged KV cacheblocks + occupancy VERIFYOutput contractformat + policy + quality STREAMToken responseTTFT + cadence + result MEASUREServing telemetrylatency + memory + cost PREFILLbuild the working setDECODEreuse it one token at a time
The request path separates routing, admission, prefill, decode, cache management, verification, and the telemetry needed to operate them.

Schedule iterations, not whole requests

Traditional request-level batching waits for every sequence in a batch to finish before admitting new work. That is a poor fit for generative workloads because prompts and outputs vary widely in length.

Orca ↗ introduced iteration-level scheduling and selective batching for Transformer inference. Instead of treating a complete request as one indivisible job, the server schedules individual generation iterations. Finished sequences leave, new sequences join, and the active batch changes over time.

This is the basis of continuous batching. The scheduler repeatedly chooses which prefill chunks and decode steps should run next under latency, fairness, and memory constraints.

The policy has to account for workload classes. Interactive chat, synchronous copilots, offline extraction, and bulk evaluation should not blindly compete in one queue. Admission control and service tiers make the trade-off explicit:

  • Reserve latency targets for interactive traffic.
  • Cap long prompts or send them to a compatible pool.
  • Limit concurrent sequences when KV-cache headroom is low.
  • Shed, defer, or downgrade work before an overloaded server collapses.
  • Prevent one tenant or batch job from monopolizing decode slots.

The KV cache is working memory

Each generated token attends to keys and values from earlier tokens. Caching those tensors avoids recomputing the entire prefix at every decode step, but the cache grows with active sequence length, layer count, hidden dimensions, and concurrency.

The allocation pattern is hostile to simple contiguous memory. Request lengths are unknown, sequences finish at different times, and reserved blocks can become fragmented or stranded.

PagedAttention and vLLM ↗ apply an operating-system-style paging idea to KV-cache management. A sequence's logical cache can map to non-contiguous physical blocks, reducing fragmentation and enabling blocks to be shared in cases such as parallel sampling.

Data flow / fan-in bottleneck

The queue is larger than the cache

FIG 02 - MOTION
Fan-in queue under finite KV-cache capacity Interactive, agent, and batch requests fan into an admission controller that assigns paged KV-cache blocks to an active decode set while backpressure defers excess work. FAN IN ADMIT BACKPRESSURE INTERACTIVEShort SLOsmall queue budget AGENTBursty sequencemany model turns BATCHLong deadlinehigh throughput CONTROLAdmission schedulerestimate + tier + fairness FINITE MEMORYPaged KV blocksallocated + shared + freed GPUDecodeactive set DEFERQueue, downgrade, or shed CONTROL VARIABLEmemory headroom, not request count alone
Admission control protects a finite working set by estimating cache demand and applying fairness, service tiers, and backpressure before allocation fails.

Paging improves utilization; it does not make capacity infinite. The serving controller still needs:

  • Per-request cache estimates before admission.
  • Headroom for bursts and allocator overhead.
  • Eviction or recomputation policy for reusable prefixes.
  • Metrics for allocated, reserved, and actually used cache.
  • Queue backpressure tied to memory, not just request count.

Prefix caching can help when many requests share an identical trusted prefix. It also creates correctness and isolation questions. The cache key must incorporate every value that changes model behavior—model revision, adapter, tokenizer, system policy, template, and relevant tenant boundary. A fast cache hit on the wrong context is a correctness incident.

Move less data through the accelerator

FlashAttention ↗ treats attention as an IO problem. Its exact attention algorithm tiles computation to reduce reads and writes between high-bandwidth memory and on-chip SRAM. The important systems lesson is broader than one kernel: accelerator performance depends on data movement, not only the number of floating-point operations.

Kernel choice, tensor parallelism, pipeline parallelism, network topology, and batch shape interact. A deployment should be benchmarked with its actual prompt-length and output-length distribution rather than a single synthetic token rate.

Quantization changes both capacity and risk

Lower-precision weights reduce memory use and bandwidth demand. That can fit a larger model on the same hardware, increase replicas, or leave more memory for KV cache.

SmoothQuant ↗ enables weight-and-activation quantization by migrating quantization difficulty from activations into weights through an equivalent transformation. AWQ ↗ uses activation evidence to protect a small fraction of salient weights during low-bit weight-only quantization.

The operational question is not whether a method achieves an average benchmark score. It is whether the chosen representation preserves the behaviors that matter for this route:

  • Structured-output validity.
  • Tool-selection accuracy.
  • Retrieval grounding.
  • Long-context fidelity.
  • Safety and refusal behavior.
  • Domain-specific terminology and calculations.

Quantization is therefore a release candidate with its own evaluation record. Memory saved is not value created if the application retries more often or sends more work to a human.

Draft cheaply, verify exactly

Speculative Decoding ↗ and Accelerating Large Language Model Decoding with Speculative Sampling ↗ use a smaller draft model to propose multiple tokens that the larger target model evaluates in parallel. With the paper's acceptance procedure, the output distribution remains that of the target model.

The speedup depends on acceptance rate, the relative cost of draft and target models, batch conditions, and serving overhead. A poor draft model creates rejected work. A strong but expensive draft model erases the benefit.

Speculation should be routed where the observed workload supports it. Acceptance rate, latency saved, energy used, and quality verification belong on the same dashboard.

Route by requirement, not model prestige

Not every request needs the largest model or longest context. A serving gateway can classify requests by capability, risk, latency budget, and cost ceiling:

  • Use a small model for deterministic extraction it has passed.
  • Use a domain adapter when the behavior—not current knowledge—is specialized.
  • Use retrieval when the evidence must be fresh or attributable.
  • Escalate to a stronger model when uncertainty or verifier failure warrants it.
  • Send high-risk actions through a policy and approval path regardless of model quality.

Routing requires calibrated evidence. If a fallback silently changes tools, policies, or context windows, it is not merely a model substitution; it is a different product behavior.

Operate a latency-quality-cost loop

Serving telemetry needs to connect infrastructure signals to user and business outcomes.

Loop / serving operations

Tune the system, not one metric

FIG 03 - MOTION
Latency, quality, and cost operating loop Workload demand drives routing and admission, runtime scheduling produces model and task telemetry, analysis evaluates latency quality and cost together, and a governed release decision updates serving policy. DEMANDWorkload mixprompt + output + deadline CONTROLRoute and admitmodel + tier + cache budget SERVESchedule inferencebatch + prefill + decode MEASUREEngine + task telemetrytail latency + verified result ANALYZEQuality-cost frontierSLO + accuracy + unit cost RELEASEGoverned policy changebenchmark + canary + rollback Cost per verified taskthe shared objective across model and infrastructure OPERATING RULEnever optimize token throughput without task outcomes
Demand, serving policy, runtime behavior, and business outcomes form one governed loop; each release must improve the whole contract.

A useful operating view includes:

  • Queue time, prefill time, time to first token, inter-token latency, and end-to-end latency.
  • Prompt and output token distributions by route and tenant.
  • Batch occupancy, decode utilization, and scheduler preemption.
  • KV-cache occupancy, fragmentation, allocation failures, and prefix-cache hit rate.
  • Quantization or model-route selection.
  • Speculative acceptance rate.
  • Tool-call and application latency outside the model server.
  • Verified task completion, retry, escalation, and abandonment.
  • Cost per verified task, not only cost per token.

Token-level metrics explain the engine. Task-level metrics explain whether the engine matters.

What this solves now

Customer support

Interactive triage needs a low time to first token, but long retrieved histories can make prefill dominant. A tiered route can summarize approved history, preserve source links, reserve interactive capacity, and escalate complex cases without making every request pay the largest-model cost.

Document intelligence

Contract and claims processing often has long inputs and bounded outputs. Chunked prefill, workload-specific batching, quantized models validated for extraction, and asynchronous service tiers can increase throughput without pretending the workload is chat.

Engineering copilots

Code tasks mix large repository context, short tool decisions, and bursts of generated patches. Prefix reuse can help stable policy and repository context, while strict cache keys prevent code or instructions crossing revision and tenant boundaries.

Commerce and operations

Demand forecasting explanations, account summaries, and operational recommendations often have deadlines rather than conversational latency needs. The platform can defer them to high-throughput queues and reserve expensive models for verifier failures or high-impact decisions.

Agent systems

An agent may call the model many times before completing one task. Cost and latency compound across planning, tools, reflection, and verification. Serving policy should meter the complete run and route each step according to its capability requirement rather than binding the entire agent to one model.

A production checklist

Before a model route carries production traffic, the team should be able to answer:

  • What are the prefill and decode distributions for this workload?
  • Which SLO matters: first token, token cadence, completion, or a task deadline?
  • How does the scheduler isolate interactive and batch traffic?
  • What limits admission when KV cache is the scarce resource?
  • Which exact context fields participate in prefix-cache identity?
  • Which behaviors were re-evaluated after quantization?
  • Where does speculative decoding improve measured acceptance and cost?
  • Can routing change models without changing the policy or tool contract?
  • Which telemetry connects accelerator utilization to verified outcomes?
  • What is the cost per successful task at the tail, not only at the median?

The production system is not serving a model. It is allocating memory, time, and confidence under demand. The token is only the visible edge of that decision.

Research referenced

Continue readingReturn to field notes →