GPU Fleet Observability: What to Monitor and Why
A single GPU dashboard is not fleet observability. At scale, the metrics that matter are aggregated, correlated, and surfaced as actionable signals — not raw telemetry. Here's what to build.
Why Single-GPU Metrics Aren't Enough
Most GPU monitoring starts with nvidia-smi or a per-node Grafana dashboard. For a single GPU or a small cluster, that's sufficient. For a fleet — multiple clusters, mixed GPU tiers, dozens of models, multiple cloud providers — per-node metrics create more noise than signal.
The questions that matter at fleet scale are different:
- Which models are underutilizing their GPU tier across the whole fleet?
- Which clusters have systematic waste patterns vs. one-off anomalies?
- What is the fleet-wide trend in KV cache pressure over the past 7 days?
- Which provider is delivering the worst GPU-to-cost ratio this week?
Answering these requires aggregated, correlated observability — not more dashboards.
---
The Four Layers of GPU Fleet Observability
Layer 1 — Hardware Metrics (per GPU)
The foundation. Collected via NVIDIA DCGM and exposed through Prometheus.
| Metric | Why It Matters |
|---|---|
| SM Utilization | Actual compute usage vs. capacity |
| Memory Bandwidth Utilization | Whether the GPU is memory-bandwidth-bound |
| VRAM Used / Free | Headroom before OOM or KV cache pressure |
| GPU Temperature | Thermal throttling risk |
| Power Draw | Cost correlation and thermal headroom |
| PCIe Throughput | Data transfer bottlenecks |
Layer 2 — Inference Server Metrics (per model)
Collected from vLLM, TGI, SGLang, or Triton metrics endpoints.
| Metric | Why It Matters |
|---|---|
| Request throughput (req/s) | Capacity vs. demand |
| Time to first token (TTFT) | Prefill efficiency |
| Inter-token latency | Decode efficiency |
| KV cache hit rate | Prefix caching effectiveness |
| Queue depth | Whether the server is keeping up |
| Batch size distribution | Continuous batching effectiveness |
Layer 3 — Workload Metrics (per deployment)
Collected from your orchestration layer (Kubernetes, Ray, custom scheduler).
| Metric | Why It Matters |
|---|---|
| Pod restart count | OOM or crash frequency |
| Replica count vs. traffic | Autoscaling efficiency |
| Request error rate | Model or infrastructure health |
| Cold start frequency | Scale-to-zero configuration effectiveness |
Layer 4 — Fleet-Level Aggregations
This is what most teams are missing. Aggregating layers 1–3 across the whole fleet to answer fleet-scale questions.
| Aggregation | Signal |
|---|---|
| Fleet-wide GPU utilization distribution | What % of GPUs are under 50% SM util? |
| Tier mismatch rate | How many models are on the wrong GPU tier? |
| Provider cost efficiency | Cost per useful GPU-hour by provider |
| KV cache pressure by model | Which models are cache-constrained? |
---
Instrumentation Stack
A practical fleet observability stack:
NVIDIA DCGM Exporter (per node)
→ Prometheus (metrics aggregation)
→ Grafana (dashboards)
→ Alertmanager (threshold alerts)
vLLM / TGI metrics endpoint (per model)
→ Prometheus
Kubernetes metrics (per pod/deployment)
→ kube-state-metrics → Prometheus
Fleet aggregation layer
→ Recording rules in Prometheus
→ Custom fleet dashboard in GrafanaThe key is recording rules — pre-computed aggregations that answer fleet-scale questions without running expensive ad-hoc queries against raw telemetry.
---
Alert Design Principles
Most GPU alert setups generate too many alerts on transient spikes and miss the slow-burn patterns that actually cost money.
Alert on trends, not spikes:
# Bad: alerts on momentary spike
alert: HighGPUMemory
expr: gpu_memory_used_bytes > 0.9 * gpu_memory_total_bytes
# Better: alerts on sustained pressure
alert: SustainedGPUMemoryPressure
expr: avg_over_time(gpu_memory_used_ratio[15m]) > 0.88Alert on fleet patterns, not individual nodes:
alert: FleetWideUnderutilization
expr: avg(gpu_sm_utilization) by (cluster) < 0.45
for: 30mAlert on cost signals, not just technical ones:
alert: ExpensiveTierUnderutilized
expr: gpu_sm_utilization{tier="h100"} < 0.35
for: 1h
annotations:
summary: "H100 running below 35% SM util for 1 hour — possible tier mismatch"---
The Visibility Gap at Scale
The most dangerous fleet observability failure mode isn't missing metrics — it's having metrics but no one looking at the right level. Per-node dashboards exist but fleet-level patterns go undetected for weeks.
The discipline of fleet observability is about designing the system so that the signals that matter — tier mismatches, systematic waste, KV cache pressure trends — surface automatically as actionable findings, not buried in dashboards that require human interpretation.
See how Paralleliq aggregates fleet-level GPU observability into actionable findings →
---
Next in the GPU Ops Field Guide: [Serverless GPU Cold Start Latency: Causes and Solutions →](/blog/gpu-ops-serverless-cold-start)