GPU Ops Field Guide

Your GPUs Are an Airline Fleet: Serving vs. Batch Inference

By Sam Hosseini·November 13, 2026·9 min read
Your GPUs Are an Airline Fleet: Serving vs. Batch Inference

Airlines live or die by load factor, flight frequency and aircraft size. Your GPU fleet runs on the same economics. Here is how those three decisions work for live LLM serving and for batch inference, why the answers differ, and where the savings are, with numbers.

Ask an airline CFO what drives profit and you'll hear three things: load factor (how many seats are filled), frequency (how many flights you fly), and gauge (how big the aircraft are). Get those right for each route and the airline makes money. Get them wrong and it flies expensive metal half empty.

Your GPU fleet runs on exactly the same economics. Every GPU is an aircraft you pay for whether its seats are full or not. And the same three decisions, how full, how many, how big, decide what each unit of AI output costs you.

The comparison works for both ways teams run large language models: live serving, where users send requests and wait for answers, and batch inference, where a fixed amount of work has to be done by a deadline. The decisions are the same in both. The right answers aren't.

This builds on two earlier posts: Model FLOPs Utilization: What Your GPUs Are Really Doing and Why a Cheaper GPU Can Make Your Tokens More Expensive. You don't need to have read them.


The aircraft

Diagram of a GPU drawn as an aircraft: the seats are what the GPU can do, passengers on board are requests in flight, fuel and equipment are the model's weights, the engines are memory bandwidth, ground turnaround is the CPU and data pipeline, and load factor is MFU
The airlineThe GPU fleetWhat it means
SeatsWhat the GPU can doFixed by the aircraft you choose
Fuel and onboard equipmentThe model's weightsUses up payload whatever you carry; a bigger model leaves less room
Passengers on boardRequests in flight (the batch size)Limited by the payload left over
Engines and cruise speedMemory bandwidthHow fast each trip goes
Load factorMFUThe share of what you paid for that's actually used
Ground turnaroundCPU and data pipelineCan limit how many trips you fly, whatever the aircraft
Gate agentThe load balancer (router)Puts each passenger on a plane with a free seat
Cost per flightGPU price per hourYou pay it full or empty

The airline's core formula carries straight over:

cost per passenger = cost per seat / load factor
cost per token     = cost of a token on a perfectly used GPU / MFU

A seat that costs $100 to fly costs $333 per passenger at 30% load factor. That's the whole problem in one line.

Two aircraft for the rest of this post:

Large jet (think H100)Turboprop (think A100 40GB)
Seats per flight300100
Cost per flight$30,000$20,000
Cost per seat, flown full$100$200
SpeedFasterSlower

The turboprop is cheaper per flight but twice as expensive per seat: two-thirds of the cost for a third of the seats. Every decision below turns on that. (All figures are illustrative and rounded.)


Two kinds of airline

Live serving: an air shuttleBatch inference: a cargo charter
The workPassengers turn up all day without booking and expect to leave within the hourA fixed load has to arrive by a date
Who decides how full each flight isThe passengers: you fly with whoever turns upYou: you decide how much goes on each flight
Does a slower aircraft matter?Yes. Passengers notice (your latency target)Only against the delivery date, and through cost
Too many aircraft meansPlanes leaving with empty seatsAircraft that don't get the load there any sooner

Live serving: an air shuttle

Live serving works like an air shuttle: a frequent, high-volume service between two nearby cities, run more like a bus line than a typical flight. Departures go every hour or so, the flight is short, boarding is quick, and nobody books ahead. Passengers just turn up and expect to leave soon, the way requests arrive at an AI model and expect an answer in seconds.

The best-known shuttles added one more promise: a guaranteed seat. If the plane at the gate filled, another one was rolled out alongside it for the overflow. So at a busy hour, several aircraft could leave together, and the gate agent put each passenger on whichever plane had a seat. That's exactly how a GPU fleet serves a model:

  • The aircraft leaving together are replicas, copies of the model running side by side.
  • The gate agent is the load balancer, or router: it sends each request to a replica with room, so a busy hour's extra demand is shared across all of them instead of piling onto one.
  • Rolling out another plane is scaling up, and sending fewer at quiet hours is scaling down.
  • At least one plane every hour, however few passengers, is the replica floor: the minimum number of replicas kept running.
  • The most planes the airline can put on one departure (aircraft, gates and crews available) is the replica ceiling: the maximum, often set by budget or GPU quota.
  • A passenger who can't get on waits for the next departure, which is a request waiting in a queue.

Unlike passengers, requests don't mind which replica answers them, so for GPUs the spreading is instant.

Now the numbers. On a normal hour, 720 passengers turn up for the shuttle. At the busiest hours, 1.5 times that: 1,080. The airline sends 8 large jets every departure, a level set for a launch rush and never revisited.

Step 1: fewer aircraft per departure (horizontal).

BeforeAfter
Aircraft per departure8 large jets4 large jets
Passengers per aircraft, normal hour90180
Load factor30%60%
Cost per departure$240,000$120,000
Cost per passenger$333$167

Because the gate agent spreads passengers across every plane, the question is whether all the aircraft together can carry the busiest hour. With 4 jets: 1,080 / 4 = 270 passengers each, still inside 300 seats, so nobody is left at the gate. Three wouldn't work: even spread evenly, 1,080 / 3 = 360 per plane, more than a jet holds.

The aircraft didn't change; they just fly fuller. In GPU terms this is removing replicas the traffic doesn't need, and it's the safest saving there is: nothing gets slower, and it's quick to reverse if demand returns. (A shuttle that adjusted through the day would send 3 jets at normal hours and roll out the fourth only at the busiest, which is autoscaling. We keep 4 here to keep the arithmetic simple.)

Step 2: smaller aircraft? (vertical)

Turboprops seat 100. To carry the busiest hour the shuttle would need 11 of them per departure (1,080 / 11 = 98 each, inside 100 seats), for $220,000: $306 a passenger, worse than the $167 the large jets now manage. On a busy shuttle the large jet is the cheaper way to fly, because it can be filled. Switching would be the classic mistake: cheaper per flight, dearer per passenger.

When smaller aircraft do pay. Take a thin shuttle: 60 passengers on a normal hour, 90 at the busiest, but the airline still promises a plane every hour. That promise is the replica floor.

1 large jet each hour1 turboprop each hour
Passengers per flight, normal hour6060
Load factor20%60%
Cost per departure$30,000$20,000
Cost per passenger$500$333

The busiest hour's 90 passengers fit in a turboprop. Here downgauging pays: the minimum schedule keeps the large jet mostly empty, so a smaller aircraft that flies fuller is cheaper. Two conditions, exactly as with GPUs:

  • The load has to fit. The model has to fit in the smaller GPU's memory, with room left for requests.
  • The slower aircraft has to keep the schedule. Every passenger takes a little longer, so the latency target has to still hold. That's why a person should approve a change like this.

The order matters. Decide aircraft size before the number of aircraft and you judge it on planes that were mostly empty for the wrong reason. Cut the number first, then look at how full the remaining planes are.

And when the shuttle is too full. The same reasoning works in reverse when passengers are left waiting at the gate:

  1. Add planes. Roll out more aircraft per departure. The autoscaler does this on its own, up to the ceiling, and nobody has to decide anything.
  2. At the ceiling and still waiting. Now it's a decision: raise the ceiling, which means more GPUs and more budget, or use bigger planes, a larger GPU per replica so each one carries more.
Too empty (waste)Too full (waiting)
FirstFewer planes, down to the floorMore planes, up to the ceiling
ThenSmaller planes, once at the floorBigger planes or a higher ceiling, once at the ceiling

So the same picture tells you when to spend more, not only when to cut.


Batch inference: a cargo charter

Now a different job: 1,200 tons of freight, due within 5 days. The charter runs 8 large freighters, each moving only 30 tons a day, because pallets reach the aircraft a few at a time. That's 240 tons a day, delivered in 5 days, for $1.2 million: $1,000 a ton.

For these freighters, a large one costs $30,000 a day and can move 300 tons a day when loaded full; a small one costs $20,000 a day and moves 100. Again: $100 versus $200 a ton, flown full.

One fact makes cargo different from passengers. For a fixed load, an aircraft's cost per ton is its cost per day divided by the tons it moves per day, however many aircraft you run. Eight freighters for one day cost the same as four for two days. So the number of aircraft only matters through the deadline, and through anything the aircraft share.

Step 1: load each flight properly (throughput).

This is the step a shuttle doesn't have. On a charter you decide how much goes on each flight, so if pallets are trickling on, fix that first. In GPU terms: make the batch job send enough requests at once. How full a properly loaded flight gets depends on what limits it.

Step 2: cut aircraft that don't speed the job up (horizontal).

Extra aircraft cost money only when they share a bottleneck. Say all 8 freighters load at one cargo terminal that handles 240 tons a day in total, like many GPUs reading from one slow input feed. The fleet gets 240 tons a day however many aircraft you run:

FleetTons per dayDaysCostPer ton
8 large freighters, one terminal2405$1,200,000$1,000
3 large freighters, same terminal2405$450,000$375

Same delivery rate, same 5 days, five fewer aircraft. If instead each aircraft has its own handling, fewer aircraft just means more days at the same cost.

Step 3: smaller aircraft? (vertical)

It depends on what limits a properly loaded flight. Now say each freighter has its own handling, so the shared terminal from step 2 no longer applies.

Case A: something other than the aircraft is the limit. Each freighter's own ground crew can only load 100 tons a day, like a slow CPU or data pipeline feeding each GPU. No aircraft moves more than that, large or small:

FleetTons per dayDaysCostPer ton
3 large freighters at 100 tons/day3004$360,000$300
3 small freighters at 100 tons/day3004$240,000$200

The smaller aircraft wins. The large one could never be filled beyond a third, so you were paying for payload and speed you couldn't use.

Case B: the aircraft is the limit. Loaded properly, each flies full:

FleetTons per dayDaysCostPer ton
1 large freighter, full3004$120,000$100
3 small freighters, full3004$240,000$200

Now the large aircraft wins easily, and smaller ones would double the cost. If the deadline were 3 days, you'd fly 2 large freighters for 2 days: still $120,000. Aircraft that each fly full cost the same however many you run.

Does the slower aircraft matter? Only against the delivery date. In Case A the small freighters finish in 4 days, inside the 5 allowed, so being slower costs nothing.


The two playbooks

Live serving (air shuttle)Batch inference (cargo charter)
FirstFewer aircraft per departure: match them to demandLoad properly: fill each flight
SecondDowngauge: only if the load fits and the schedule holdsCut aircraft: those sharing a bottleneck and adding no speed
ThirdDowngauge: only if something other than the aircraft limits the load, and the date holds
Slower aircraftAlways matters: passengers are waitingMatters only against the deadline
Classic mistakeDowngauging a busy shuttleDowngauging before loading properly

In both, aircraft size is decided last, because whether a smaller aircraft saves money depends on how full the large one would fly once everything before it is fixed. Decide it too early and the "cheaper" aircraft can double your cost.


Back to GPUs

The airlineLive serving on GPUsBatch inference on GPUs
Too many aircraft per departureMore replicas than traffic needsMore parallel pods than the job can keep busy
Aircraft too largeA larger GPU than the model needsA larger GPU than a properly fed job can use
Low load factorLow traffic per replicaToo few requests in flight per GPU
Passengers waitingThe latency targetNo one: the job's deadline instead
Slow ground turnaroundRareA slow CPU or data pipeline starving the GPU
Gate agent putting passengers on whichever plane has a seatThe load balancer spreading requests across replicasThe job's work queue spreading items across pods
At least one plane every hourThe autoscaler's replica floorThe job's parallelism setting

The takeaway

The airlines that win are rarely the ones with the biggest fleets. They're the ones that match aircraft to demand route by route: planes that fly full, extra capacity rolled out only when passengers are actually waiting, and new aircraft bought only once the ones they have are already busy. That's what lets them offer lower prices without leaving passengers stranded.

GPU fleets work the same way. If you sell inference, a fleet that runs full is what lets you price competitively and still keep your margin, while keeping response times where your customers expect them. If you run models for your own business, it's what lets you put off the next GPU order until the GPUs you already pay for are genuinely busy.


Where Paralleliq fits

An airline has a network-planning team whose whole job is getting load factor, frequency and gauge right on every route, and revisiting them as demand changes. Most GPU fleets have no equivalent. Each team sizes its own deployments once and moves on.

Paralleliq does that job for a GPU fleet. It reads the telemetry and configuration your cluster already produces and recommends the specific change, in the order above. For live serving, it recommends cutting replicas first, and a smaller GPU only once a deployment is at its replica floor, the model fits, and the current GPU is underused. Both need a week of data and the latency target you set. A smaller GPU carries less, so it also tells the operator that the replica count and autoscaler limits have to be resized with it. For batch jobs it flags jobs sending too few requests at once and jobs on a bigger GPU than they need; cutting parallel pods that add no speed is in development. Every recommendation shows its evidence and estimated saving, nothing is applied until an operator approves it, and every decision is recorded.

Want to see your own fleet's load factor? piqc is our free, open-source scanner: point it at a Kubernetes cluster and it shows what's deployed, on what hardware, and where capacity is being left on the table, read-only and with no account needed. Or request a demo to see how Paralleliq turns that into approved, audited changes.

More articles

Find out what your GPUs can already do.

Request Demo