Your GPUs Are an Airline Fleet: Serving vs. Batch Inference

Airlines live or die by load factor, flight frequency and aircraft size. Your GPU fleet runs on the same economics. Here is how those three decisions work for live LLM serving and for batch inference, why the answers differ, and where the savings are, with numbers.
Ask an airline CFO what drives profit and you'll hear three things: load factor (how many seats are filled), frequency (how many flights you fly), and gauge (how big the aircraft are). Get those right for each route and the airline makes money. Get them wrong and it flies expensive metal half empty.
Your GPU fleet runs on exactly the same economics. Every GPU is an aircraft you pay for whether its seats are full or not. And the same three decisions, how full, how many, how big, decide what each unit of AI output costs you.
The comparison works for both ways teams run large language models: live serving, where users send requests and wait for answers, and batch inference, where a fixed amount of work has to be done by a deadline. The decisions are the same in both. The right answers aren't.
This builds on two earlier posts: Model FLOPs Utilization: What Your GPUs Are Really Doing and Why a Cheaper GPU Can Make Your Tokens More Expensive. You don't need to have read them.
The aircraft
| The airline | The GPU fleet | What it means |
|---|---|---|
| Seats | What the GPU can do | Fixed by the aircraft you choose |
| Fuel and onboard equipment | The model's weights | Uses up payload whatever you carry; a bigger model leaves less room |
| Passengers on board | Requests in flight (the batch size) | Limited by the payload left over |
| Engines and cruise speed | Memory bandwidth | How fast each trip goes |
| Load factor | MFU | The share of what you paid for that's actually used |
| Ground turnaround | CPU and data pipeline | Can limit how many trips you fly, whatever the aircraft |
| Gate agent | The load balancer (router) | Puts each passenger on a plane with a free seat |
| Cost per flight | GPU price per hour | You pay it full or empty |
The airline's core formula carries straight over:
cost per passenger = cost per seat / load factor
cost per token = cost of a token on a perfectly used GPU / MFUA seat that costs $100 to fly costs $333 per passenger at 30% load factor. That's the whole problem in one line.
Two aircraft for the rest of this post:
| Large jet (think H100) | Turboprop (think A100 40GB) | |
|---|---|---|
| Seats per flight | 300 | 100 |
| Cost per flight | $30,000 | $20,000 |
| Cost per seat, flown full | $100 | $200 |
| Speed | Faster | Slower |
The turboprop is cheaper per flight but twice as expensive per seat: two-thirds of the cost for a third of the seats. Every decision below turns on that. (All figures are illustrative and rounded.)
Two kinds of airline
| Live serving: an air shuttle | Batch inference: a cargo charter | |
|---|---|---|
| The work | Passengers turn up all day without booking and expect to leave within the hour | A fixed load has to arrive by a date |
| Who decides how full each flight is | The passengers: you fly with whoever turns up | You: you decide how much goes on each flight |
| Does a slower aircraft matter? | Yes. Passengers notice (your latency target) | Only against the delivery date, and through cost |
| Too many aircraft means | Planes leaving with empty seats | Aircraft that don't get the load there any sooner |
Live serving: an air shuttle
Live serving works like an air shuttle: a frequent, high-volume service between two nearby cities, run more like a bus line than a typical flight. Departures go every hour or so, the flight is short, boarding is quick, and nobody books ahead. Passengers just turn up and expect to leave soon, the way requests arrive at an AI model and expect an answer in seconds.
The best-known shuttles added one more promise: a guaranteed seat. If the plane at the gate filled, another one was rolled out alongside it for the overflow. So at a busy hour, several aircraft could leave together, and the gate agent put each passenger on whichever plane had a seat. That's exactly how a GPU fleet serves a model:
- The aircraft leaving together are replicas, copies of the model running side by side.
- The gate agent is the load balancer, or router: it sends each request to a replica with room, so a busy hour's extra demand is shared across all of them instead of piling onto one.
- Rolling out another plane is scaling up, and sending fewer at quiet hours is scaling down.
- At least one plane every hour, however few passengers, is the replica floor: the minimum number of replicas kept running.
- The most planes the airline can put on one departure (aircraft, gates and crews available) is the replica ceiling: the maximum, often set by budget or GPU quota.
- A passenger who can't get on waits for the next departure, which is a request waiting in a queue.
Unlike passengers, requests don't mind which replica answers them, so for GPUs the spreading is instant.
Now the numbers. On a normal hour, 720 passengers turn up for the shuttle. At the busiest hours, 1.5 times that: 1,080. The airline sends 8 large jets every departure, a level set for a launch rush and never revisited.
Step 1: fewer aircraft per departure (horizontal).
| Before | After | |
|---|---|---|
| Aircraft per departure | 8 large jets | 4 large jets |
| Passengers per aircraft, normal hour | 90 | 180 |
| Load factor | 30% | 60% |
| Cost per departure | $240,000 | $120,000 |
| Cost per passenger | $333 | $167 |
Because the gate agent spreads passengers across every plane, the question is whether all the aircraft together can carry the busiest hour. With 4 jets: 1,080 / 4 = 270 passengers each, still inside 300 seats, so nobody is left at the gate. Three wouldn't work: even spread evenly, 1,080 / 3 = 360 per plane, more than a jet holds.
The aircraft didn't change; they just fly fuller. In GPU terms this is removing replicas the traffic doesn't need, and it's the safest saving there is: nothing gets slower, and it's quick to reverse if demand returns. (A shuttle that adjusted through the day would send 3 jets at normal hours and roll out the fourth only at the busiest, which is autoscaling. We keep 4 here to keep the arithmetic simple.)
Step 2: smaller aircraft? (vertical)
Turboprops seat 100. To carry the busiest hour the shuttle would need 11 of them per departure (1,080 / 11 = 98 each, inside 100 seats), for $220,000: $306 a passenger, worse than the $167 the large jets now manage. On a busy shuttle the large jet is the cheaper way to fly, because it can be filled. Switching would be the classic mistake: cheaper per flight, dearer per passenger.
When smaller aircraft do pay. Take a thin shuttle: 60 passengers on a normal hour, 90 at the busiest, but the airline still promises a plane every hour. That promise is the replica floor.
| 1 large jet each hour | 1 turboprop each hour | |
|---|---|---|
| Passengers per flight, normal hour | 60 | 60 |
| Load factor | 20% | 60% |
| Cost per departure | $30,000 | $20,000 |
| Cost per passenger | $500 | $333 |
The busiest hour's 90 passengers fit in a turboprop. Here downgauging pays: the minimum schedule keeps the large jet mostly empty, so a smaller aircraft that flies fuller is cheaper. Two conditions, exactly as with GPUs:
- The load has to fit. The model has to fit in the smaller GPU's memory, with room left for requests.
- The slower aircraft has to keep the schedule. Every passenger takes a little longer, so the latency target has to still hold. That's why a person should approve a change like this.
The order matters. Decide aircraft size before the number of aircraft and you judge it on planes that were mostly empty for the wrong reason. Cut the number first, then look at how full the remaining planes are.
And when the shuttle is too full. The same reasoning works in reverse when passengers are left waiting at the gate:
- Add planes. Roll out more aircraft per departure. The autoscaler does this on its own, up to the ceiling, and nobody has to decide anything.
- At the ceiling and still waiting. Now it's a decision: raise the ceiling, which means more GPUs and more budget, or use bigger planes, a larger GPU per replica so each one carries more.
| Too empty (waste) | Too full (waiting) | |
|---|---|---|
| First | Fewer planes, down to the floor | More planes, up to the ceiling |
| Then | Smaller planes, once at the floor | Bigger planes or a higher ceiling, once at the ceiling |
So the same picture tells you when to spend more, not only when to cut.
Batch inference: a cargo charter
Now a different job: 1,200 tons of freight, due within 5 days. The charter runs 8 large freighters, each moving only 30 tons a day, because pallets reach the aircraft a few at a time. That's 240 tons a day, delivered in 5 days, for $1.2 million: $1,000 a ton.
For these freighters, a large one costs $30,000 a day and can move 300 tons a day when loaded full; a small one costs $20,000 a day and moves 100. Again: $100 versus $200 a ton, flown full.
One fact makes cargo different from passengers. For a fixed load, an aircraft's cost per ton is its cost per day divided by the tons it moves per day, however many aircraft you run. Eight freighters for one day cost the same as four for two days. So the number of aircraft only matters through the deadline, and through anything the aircraft share.
Step 1: load each flight properly (throughput).
This is the step a shuttle doesn't have. On a charter you decide how much goes on each flight, so if pallets are trickling on, fix that first. In GPU terms: make the batch job send enough requests at once. How full a properly loaded flight gets depends on what limits it.
Step 2: cut aircraft that don't speed the job up (horizontal).
Extra aircraft cost money only when they share a bottleneck. Say all 8 freighters load at one cargo terminal that handles 240 tons a day in total, like many GPUs reading from one slow input feed. The fleet gets 240 tons a day however many aircraft you run:
| Fleet | Tons per day | Days | Cost | Per ton |
|---|---|---|---|---|
| 8 large freighters, one terminal | 240 | 5 | $1,200,000 | $1,000 |
| 3 large freighters, same terminal | 240 | 5 | $450,000 | $375 |
Same delivery rate, same 5 days, five fewer aircraft. If instead each aircraft has its own handling, fewer aircraft just means more days at the same cost.
Step 3: smaller aircraft? (vertical)
It depends on what limits a properly loaded flight. Now say each freighter has its own handling, so the shared terminal from step 2 no longer applies.
Case A: something other than the aircraft is the limit. Each freighter's own ground crew can only load 100 tons a day, like a slow CPU or data pipeline feeding each GPU. No aircraft moves more than that, large or small:
| Fleet | Tons per day | Days | Cost | Per ton |
|---|---|---|---|---|
| 3 large freighters at 100 tons/day | 300 | 4 | $360,000 | $300 |
| 3 small freighters at 100 tons/day | 300 | 4 | $240,000 | $200 |
The smaller aircraft wins. The large one could never be filled beyond a third, so you were paying for payload and speed you couldn't use.
Case B: the aircraft is the limit. Loaded properly, each flies full:
| Fleet | Tons per day | Days | Cost | Per ton |
|---|---|---|---|---|
| 1 large freighter, full | 300 | 4 | $120,000 | $100 |
| 3 small freighters, full | 300 | 4 | $240,000 | $200 |
Now the large aircraft wins easily, and smaller ones would double the cost. If the deadline were 3 days, you'd fly 2 large freighters for 2 days: still $120,000. Aircraft that each fly full cost the same however many you run.
Does the slower aircraft matter? Only against the delivery date. In Case A the small freighters finish in 4 days, inside the 5 allowed, so being slower costs nothing.
The two playbooks
| Live serving (air shuttle) | Batch inference (cargo charter) | |
|---|---|---|
| First | Fewer aircraft per departure: match them to demand | Load properly: fill each flight |
| Second | Downgauge: only if the load fits and the schedule holds | Cut aircraft: those sharing a bottleneck and adding no speed |
| Third | Downgauge: only if something other than the aircraft limits the load, and the date holds | |
| Slower aircraft | Always matters: passengers are waiting | Matters only against the deadline |
| Classic mistake | Downgauging a busy shuttle | Downgauging before loading properly |
In both, aircraft size is decided last, because whether a smaller aircraft saves money depends on how full the large one would fly once everything before it is fixed. Decide it too early and the "cheaper" aircraft can double your cost.
Back to GPUs
| The airline | Live serving on GPUs | Batch inference on GPUs |
|---|---|---|
| Too many aircraft per departure | More replicas than traffic needs | More parallel pods than the job can keep busy |
| Aircraft too large | A larger GPU than the model needs | A larger GPU than a properly fed job can use |
| Low load factor | Low traffic per replica | Too few requests in flight per GPU |
| Passengers waiting | The latency target | No one: the job's deadline instead |
| Slow ground turnaround | Rare | A slow CPU or data pipeline starving the GPU |
| Gate agent putting passengers on whichever plane has a seat | The load balancer spreading requests across replicas | The job's work queue spreading items across pods |
| At least one plane every hour | The autoscaler's replica floor | The job's parallelism setting |
The takeaway
The airlines that win are rarely the ones with the biggest fleets. They're the ones that match aircraft to demand route by route: planes that fly full, extra capacity rolled out only when passengers are actually waiting, and new aircraft bought only once the ones they have are already busy. That's what lets them offer lower prices without leaving passengers stranded.
GPU fleets work the same way. If you sell inference, a fleet that runs full is what lets you price competitively and still keep your margin, while keeping response times where your customers expect them. If you run models for your own business, it's what lets you put off the next GPU order until the GPUs you already pay for are genuinely busy.
Where Paralleliq fits
An airline has a network-planning team whose whole job is getting load factor, frequency and gauge right on every route, and revisiting them as demand changes. Most GPU fleets have no equivalent. Each team sizes its own deployments once and moves on.
Paralleliq does that job for a GPU fleet. It reads the telemetry and configuration your cluster already produces and recommends the specific change, in the order above. For live serving, it recommends cutting replicas first, and a smaller GPU only once a deployment is at its replica floor, the model fits, and the current GPU is underused. Both need a week of data and the latency target you set. A smaller GPU carries less, so it also tells the operator that the replica count and autoscaler limits have to be resized with it. For batch jobs it flags jobs sending too few requests at once and jobs on a bigger GPU than they need; cutting parallel pods that add no speed is in development. Every recommendation shows its evidence and estimated saving, nothing is applied until an operator approves it, and every decision is recorded.
Want to see your own fleet's load factor? piqc is our free, open-source scanner: point it at a Kubernetes cluster and it shows what's deployed, on what hardware, and where capacity is being left on the table, read-only and with no account needed. Or request a demo to see how Paralleliq turns that into approved, audited changes.