Beyond Prometheus: What Observability Actually Means for Model Inference

Published: September 2026

Prometheus is the tool almost every infrastructure team reaches for first when they start monitoring GPU fleets. For model inference specifically, it's the beginning of the story — not the end of it.

Prometheus Gets You In The Room

Prometheus is good at what it does: scrape a metrics endpoint, store a time series, alert on a threshold. Every serving engine worth using — vLLM, TensorRT-LLM, TGI — exposes a Prometheus-format /metrics endpoint, and that's usually step one for any team standing up observability on a new GPU fleet.

But "Prometheus" and "observability for model inference" are not the same thing. Prometheus is a collection and storage mechanism. It has no opinion about what a healthy inference workload looks like, no concept of a KV cache, and — this one surprises people — no reliable way to tell you a percentile of a rate over time, because a rate is inherently a measurement across many samples, not something a single scrape can produce. Knowing what to point Prometheus at, and what to build on top of it, is where the real observability work happens.

The Metrics That Are Specific to Inference

Generic web-service monitoring doesn't map cleanly onto autoregressive generation, and treating "request latency" as one number hides the two very different things actually happening inside it:

• TTFT (time to first token) — dominated by the prefill pass, which is compute-bound: the model processes your entire input in one parallelized shot before it emits anything.
• TPOT (time per output token), also called inter-token latency — dominated by decode, which is memory-bandwidth-bound: one token at a time, repeatedly reading the full KV cache.

A tier upgrade that fixes a TTFT problem can do nothing for a TPOT problem, and vice versa — they're bottlenecked by different hardware resources entirely. If your dashboard only shows "p95 latency," you're averaging over two failure modes that need opposite fixes.

A few more that don't have an equivalent outside inference serving:

• KV cache hit rate, eviction rate, and fragmentation — unique to transformer serving. There's no "cache" concept in a typical stateless web request.
• Goodput — a newer, increasingly-cited reframe of throughput: not raw tokens/sec, but tokens/sec that actually met your latency SLO. A system can post excellent raw throughput while most of it is technically an SLO violation; goodput is the number that won't let you miss that.
• Batch composition over time — modern serving engines batch dynamically, mid-flight, as requests arrive and complete. A static "batch size" gauge tells you almost nothing; watching how batch composition shifts under load is where the real signal lives.

One Layer Below: What's Actually Happening On The GPU

Prometheus-format metrics from the serving engine tell you what the model is doing. They don't tell you what the hardware is doing underneath it — and those two pictures can disagree. A GPU can look busy at the application layer while sitting idle at the tensor-core level, or vice versa.

That's what tools like NVIDIA DCGM (Data Center GPU Manager) and the NVML library underneath it are for: tensor core activity, memory bandwidth utilization, ECC error counts, NVLink throughput, power and thermal state. This is the layer that answers "is this GPU actually doing compute-bound work, or is it just occupied?" — a distinction that matters enormously when you're trying to tell a compute-bound prefill bottleneck apart from a memory-bandwidth-bound decode one.

The Problem Prometheus Alone Doesn't Solve: History and Scale

Vanilla Prometheus has limited retention and no built-in way to query across clusters. That's fine for "what's the value right now" — it falls apart the moment you need "what did this look like over the last 30 days across my whole fleet," which is exactly the kind of question capacity planning and burst-traffic analysis require.

This is what Mimir, Thanos, and VictoriaMetrics exist to fix: long-term storage and cross-cluster querying layered on top of Prometheus's data model, without changing how anything gets scraped. If you're serious about historical analysis — traffic burstiness, week-over-week utilization trends, anything that needs a percentile computed across many samples over time rather than a single point-in-time read — this is the layer that makes it possible.

When A Request Has More Than One Hop

Disaggregated serving — a router splitting prefill and decode across separate pools — means a single request's latency is now spread across multiple services instead of living inside one process. Prometheus counters on each service in isolation won't tell you where time is actually going across the hop; they'll just tell you each service, individually, looks fine.

That's what distributed tracing is for. OpenTelemetry has become the standard way to emit traces, metrics, and logs together, with Jaeger or Grafana Tempo as common places to actually go look at them. Once inference stops being one process end to end, tracing is the difference between "throughput looks bad" and "throughput looks bad because the decode pool is starved waiting on prefill" — a distinction no amount of per-service dashboards will hand you on their own.

A Different Layer Entirely: Is The Model Any Good

Everything above is infrastructure observability — is the fleet healthy, is the hardware being used well, is latency where it should be. There's a separate, equally real category of observability that has nothing to do with GPUs at all: is the model's output any good, and what did it cost to produce.

Platforms like Langfuse (recently acquired by ClickHouse) track prompts, completions, token counts, and quality evaluations per request — application-layer observability, not infrastructure-layer. It's worth knowing the two are different disciplines with different tools, even though they both get called "AI observability." One tells you if your GPUs are healthy. The other tells you if your model is.

Why This Is Getting Harder, Not Easier

Enterprise IT teams are being asked to do more than ever, with less room to fail. They're being asked to deploy AI at a pace the business demands, defend against a threat surface that AI itself is expanding, and modernize infrastructure without disrupting what the business depends on. And do it all without adding headcount or risk. This session is for the teams living that reality. We'll cover what's working, what the fastest-moving organizations are doing differently, and what's now possible that wasn't six months ago.

Building and maintaining full coverage across everything above — serving-engine metrics, GPU-level telemetry, long-term storage, distributed tracing, and quality evaluation — is a real, multi-tool undertaking. Most teams don't have the headcount to become experts in five separate observability stacks on top of everything else they're already responsible for.

That's the gap Paralleliq is built to close. Instead of asking your team to stitch together Prometheus, DCGM, and a long-term storage layer by hand and then learn to read all of it, Paralleliq reads the signals that actually matter for GPU fleet health — tier misplacement, dark capacity, KV cache pressure, throughput suppression — and turns them into specific, dollar-quantified findings your team can act on, with nothing executed without human approval. You get the outcome good observability is supposed to produce, without needing to become an observability team first.

Paralleliq is the model-aware GPU fleet optimization layer for AI infrastructure. Start with [piqc](https://github.com/paralleliq/piqc) — the source-available GPU waste scanner — or [talk to us](/contact) about what full fleet visibility looks like for your infrastructure.