The Next Layer of Inference Efficiency: Cross-Instance KV Cache and Multi-Stage Serving
Two developments in the vLLM ecosystem — LMCache's cross-instance KV cache sharing and vLLM-Omni's multi-stage serving — point at where inference efficiency problems are heading next, and why a one-time configuration decision won't keep up.
Most teams think about inference efficiency as a single-model, single-replica problem: the right GPU tier, the right max-num-seqs, the right batch size. Two recent developments in the vLLM ecosystem suggest that frame is already too narrow.
---
LMCache: KV cache that outlives a single replica
vLLM's own prefix caching (covered in our KV Cache Pressure post) reuses cached attention state within a single running engine. LMCache extends the same idea across the fleet: it backs the KV cache with a shared store — CPU memory, local disk, or a remote/distributed cache — so a prefix computed by one replica can be reused by a completely different replica serving a later request.
That matters anywhere prefixes repeat across replicas rather than within one: shared system prompts, multi-turn sessions that get load-balanced across pods, RAG pipelines reusing the same retrieved context. Without cross-instance sharing, every replica pays the full prefill cost the first time it sees a prefix, no matter how many times another replica has already seen it.
LMCache also plugs into vLLM's support for prefill/decode disaggregation — running the compute-bound prefill phase and the memory-bandwidth-bound decode phase on separate instances, connected by a KV transfer layer. That lets each phase run on hardware shaped for what it actually needs, instead of forcing one instance to be good at both.
The net effect: lower time-to-first-token from avoided recomputation, more consistent per-token latency from freed-up compute and phase isolation, and the option to size prefill and decode capacity independently.
The catch is the same one that applies to any cache: hit rate depends on how well the cache matches current traffic, and traffic isn't static. A cache that's earning its memory and complexity overhead today can stop earning it the moment the workload mix shifts toward more unique, one-off requests. Sizing and enabling it is a decision made once; whether it's still the right decision is a question that has to be asked continuously.
---
vLLM-Omni: when "the model" is actually a pipeline of models
vLLM-Omni — vLLM project's own framework for omni-modality serving, which reached its first stable release this year — tackles a different problem: serving models that take and produce more than text. Image, video, audio, and non-autoregressive architectures like Diffusion Transformers all need different execution patterns than a standard autoregressive LLM.
Its core idea is a stage graph: an any-to-any model gets decomposed into a graph of stages — an LLM core for reasoning, a Diffusion Transformer for image or video generation, an audio decoder, an encoder for multimodal input — each of which can run as its own process, on its own hardware, sequenced by an orchestrator and connected by a transfer layer purpose-built to move intermediate state between stages. Qwen3-Omni is a concrete example: it's broken into Thinker, Talker, and Code2wav stages, each a distinct model.
This formalizes something that's already happening informally wherever multimodal and agentic pipelines get built: different stages of one logical request have wildly different resource profiles. A diffusion stage is compute-heavy in a different way than an LLM core; an audio decoder might be lightweight compared to both. Treating the whole pipeline as one workload on one box stops making sense once stages disaggregate — and the right placement for stage one is rarely the right placement for stage three.
---
The pattern underneath both
LMCache and vLLM-Omni are solving different problems, but they push on the same assumption from two directions: that a request maps cleanly onto one model running on one replica. LMCache breaks that by letting a single phase of one request (prefill) be served by different hardware than another phase (decode) of the same request, with cache state shared across a fleet rather than scoped to a process. vLLM-Omni breaks it further by splitting a single logical request across an arbitrary number of heterogeneous model stages.
Both also introduce something worth tracking that doesn't exist in the simple single-model world: a cache hit rate that's a fleet-wide property, not a per-process one, and a resource profile that varies stage-by-stage within a single request rather than being constant for the life of a deployment.
---
What this means for operators
If you're adopting either of these, two things are worth doing before you reach for tuning knobs:
- Instrument before you optimize. Know your actual hit rate, eviction rate, and per-stage GPU profile before changing configuration. Guessing at sizing for a cache or a multi-stage pipeline is no more reliable than guessing at GPU tier sizing — and the failure modes (silent latency regressions, unexplained cost increases) look similar.
- Re-check the decision, not just the configuration. A cache or pipeline topology that was correctly sized at launch can become wrong without anyone changing a setting — traffic composition shifts, session patterns change, models get swapped. The decision to enable, size, or restructure either of these isn't a one-time call; it's a question that needs re-asking as the workload evolves.
That second point is the throughline of almost everything we've written in this series. Static configuration was never really the goal — it was a stand-in for "correctly sized for the traffic I have right now." As inference architectures get more sophisticated — cache shared across a fleet, requests split across heterogeneous model stages — the gap between "configured once" and "correct right now" only gets wider.
See how Paralleliq tracks fleet-wide efficiency as inference architectures evolve →