Set Linux to Performance Mode: 64-Image Batch Under 100 ms, Not Powersave

This guide shows how to get 64-image batch inference latency under 100 ms on Linux. It gives you the exact powersave-vs-performance test, plus thread pinning and deep C-state checks, before you touch any model code.

Why powersave taxes sub-100 ms batches

Under a powersave governor, the CPU doesn't just run slower — it injects two separate hardware delays into every batch. The first is C-state exit latency. Between your 64-image batches, idle cores sink into deep sleep states, and waking them up costs real time before the GPU feed can resume. You can read the exact penalty your silicon charges on this machine: cat /sys/devices/system/cpu/cpu0/cpuidle/state*/latency prints the exit latency for each state in microseconds. If your cores are parking in C6 or C8, that number is the floor on how fast the next batch can possibly start, no matter how fast the GPU is.

The second delay is P-state ramp latency. The schedutil and powersave governors only raise frequency after they see load, which means the first stretch of each batch runs at low clocks while the governor decides the workload is real. Watch it happen: run turbostat --interval 1 during a batch and you'll see the package frequency climb after the work has already started. For a long training run this ramp is noise. For a sub-100 ms batch it's a meaningful chunk of the budget, spent at the wrong clock.

These two costs compound because they hit the same window. The C-state exit has to finish before the core is even available,

and then the frequency ramp has to finish before the core is fast. A batch that lands on a cold, parked core pays both costs back-to-back. That's the mechanism by which a governor setting — not model code, not kernel code — moves your tail latency.

This is why the fix is a configuration change, not an optimization project. Pinning threads keeps work on cores that stay warm, disabling deep C-states gets rid of the exit penalty entirely, and the performance governor holds frequency high so there's no ramp to wait out. The scheduling and frequency-transition overhead drops off the critical path, and what's left is the GPU compute you actually care about.

Before you make any of it permanent, measure. Run your 64-image batch twice on the same machine — once with cpupower frequency-set -g powersave, once with -g performance — and compare p99, not the mean. The mean will look fine either way; the tail is where the C-state and ramp costs pile up. Only keep performance mode if p99 improves by more than 10% over the powersave baseline and the power draw stays within budget. The mechanism above tells you why it should; the two-run check tells you whether it did.

The evidence: what studies actually show

The strongest published anchor for treating governor choice as a measurable variable rather than folklore is an arXiv study on the feasibility and trade-offs of inference systems, which asks directly how system configurations — power modes, threading, and related settings — affect inferencefficiency at the microarchitectural level. Its metric set is the part worth stealing: cache misses and context switches, not just wall-clock throughput. That framing matters because it tells you where to look when a 64-image batch stalls. If your p99 latency moves when you change the governor but your GPU utilization curve barely shifts, the study's metric list is your explanation: the time went into cache behavior and context-switch churn on the host side, not into compute. Read that paper before you accept anyone's claim that frequency scaling is irrelevant to batch inference.

The second independent source is more mundane but just as useful: the Geekbench browser database publishes real submitted results whose recorded system configuration lists the governor as powersave. That's a small thing with a big implication. It means governor state is captured as a first-order field in a public benchmark record — the same tier of configuration data as motherboard and CPU name. When a benchmark harness treats a setting as part of the reported configuration, the setting isn't a tuning curiosity; it's a variable that changes what the numbers mean. If you're comparing your own 64-image batch timings against any published result, check whether the governor is recorded before you conclude your hardware is slower.

Put the two together and you get the convergence claim this section exists to make: an independent peer-reviewed study that measures power-mode and threading effects on inference efficiency, and an independent public benchmark database that records governor state as configuration, point the same direction. Governor selection is an empirically tracked, first-order input to inference performance. Neither source says the performance governor is always faster for every workload — that would overreach. What they jointly support is narrower and more actionable: the setting is measurable, it is recorded, and it belongs in your baseline before you touch model code.

So the check is cheap. Run your 64-image batch twice on the same machine, same model, same thread pinning, changing only the governor — once with cpupower frequency-set -g powersave, once with -g performance. Capture p50 and p99, not the mean. Then adopt performance mode permanently only if p99 improves by more than 10% over the powersave baseline and the power and thermal cost stays within budget. If p99 does not move, you have learned something valuable too: your bottleneck is elsewhere, and the governor was never your problem.

One caveat on scope: both sources establish that the governor is a tracked, measurable configuration input. Neither establishes a universal latency delta, and neither was run on your exact 64-image pipeline. That is precisely why the two-run A/B test above is the gate — the literature tells you the variable is real and worth measuring; your own p99 tells you whether it is worth keeping.

Governor and config options compared

The five-way comparison below is the decision matrix this section exists to deliver: rank the configurations, pick the winner, and stop guessing. For latency-bound batch inference, the ranking is unambiguous — the combination that pins clocks high before the batch arrives wins, because it removes the ramp delay entirely rather than shortening it.

The winner works because it changes the timing of the frequency decision, not just its outcome. With performance set on all cores and the driver itself (intel_pstate=performance on Intel, the amd-pstate equivalent on AMD) in performance mode, cores are already at their operating frequency when the batch lands. The check: run cpupower frequency-info and confirm both the governor and the driver report performance — setting the governor alone is the most common half-fix.

The runner-up, schedutil, is the right answer for efficiency-sensitive workloads, and its tunables (up_rate_limit_us, down_rate_limit_us) let you trade ramp speed against power. But for a sub-100 ms target, any ramp is a tax paid inside the measurement window. If you run schedutil, tighten the up-rate limit and re-run your two-batch A/B; if p99 still misses, the tunables didn't buy you enough.

The loser deserves explicit burial. The myth — "powersave only saves power, performance is roughly equal" — fails on bursty batch workloads because the ramp delay lands directly inside the request window, exactly the mechanism the frequency-transition section details. The Geekbench database entry for the ADL Jimny system records powersave as the governor on a published inference-relevant run, which is worth noting as evidence that this default ships in the wild, not just in theory.

One caveat the table can't encode: the arXiv feasibility study on inference systems treats power modes and threading as system-configuration variables affecting microarchitectural metrics like cache misses and context switches — meaning governor choice interacts with your thread-pinning setup. Adopt performance mode permanently only if your own two-run A/B shows p99 improves by more than 10% over the powersave baseline; otherwise schedutil with tuned rate limits is the defensible middle ground.

Numbers that matter: latency vs watts

Before you change a line of model code, run the 64-image batch twice on the same machine: once with cpupower frequency-set -g powersave and once with -g performance. Capture three numbers on each run and nothing else — p50 and p99 batch latency, package power from turbostat (PkgWatt), and CPU temperature. Those three readings are the entire decision; if you catch yourself arguing from feel, you have not measured yet.

Bracket the batch correctly or the numbers lie. Call torch.cuda.synchronize() immediately before the first image and immediately after the last, so the timer encloses the GPU work and not just the launch queue. Discard a warm-up pass, then log enough batches that p99 stops moving between runs. For a sub-100 ms target, p50 tells you what the typical request costs, but p99 is the number that breaks your latency budget — optimize the tail, not the median.

Threshold one is a latency gate. Adopt performance mode permanently only if p99 improves by more than 10% over the powersave baseline. If the gain lands under that line, the extra watts are not buying you anything you need: you are already meeting the latency target, and you are paying power and heat for headroom you will not use. A 10% tail improvement on a 64-image batch is the point where the governor switch starts to matter; below it, leave the governor alone and spend the effort elsewhere.

Threshold two is a power and thermal ceiling. Read sustained PkgWatt across the whole batch window, not a single peak sample, and compare it against your chassis or VM power budget alongside the CPU temperature. If sustained package power or thermals exceed that budget, do not enable performance mode globally. Scope it instead: pin the inference threads to a defined core set and let the governor ride performance only on those cores, or enforce the cap through the VM's power limit — then re-run the same three-number protocol to confirm the p99 gain survives.

Run the protocol, write the three numbers down, and let the two thresholds decide: adopt performance mode permanently only if p99 improves by more than 10% over the powersave baseline and sustained PkgWatt and CPU temperature stay within your chassis or VM budget. The governor is a cheap knob, but cheap is not the same as free — the measurement is what tells you whether the watts are earning their keep.

What this guide cannot promise

The performance-governor rule is a strong default, not a universal law. Before you make it permanent, you need to know the three situations where it stops paying off — and the one where it keeps winning even when the CPU looks idle.

It breaks when the batch is GPU-bound. If nvidia-smi dmon shows SM utilization pegged near 100% for the duration of the batch, the CPU is already waiting on the accelerator, and governor changes move p99 by noise-level amounts. The fix in that case is not a frequency knob — it is batch size or precision. Shrink the batch until SM utilization drops below saturation, or move to a lower-precision path, and re-measure. The arXiv study on feasibility and trade-offs of inference systems is the right reference here: it asks directly how power modes and threading affect inference efficiency at the microarchitectural level, including cache misses and context switches, which is exactly the layer where a GPU-bound batch hides its real bottleneck.

It breaks on shared and virtualized hosts. On a cloud VM you frequently do not control the host governor, and the guest may expose a read-only cpufreq interface. Verify before you assume: run cpupower frequency-info and grep governor /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor. If those paths are read-only or the governor will not change, the rule degrades to a procurement rule — request dedicated or burstable CPU, because you cannot tune a frequency policy you do not own.

It still wins on dataloader-bound batches. This is the case people misdiagnose. If the GPU sits idle between kernel launches while the host decodes JPEGs, resizes, and collates tensors, the CPU is the critical path, and a performance governor plus pinned threads plus disabled deep C-states removes exactly the transition overhead that shows up in p99. The Geekbench powersave-governor database entry is a useful sanity check on how much governor choice alone can move a score, but treat it as a directional signal, not a latency prediction for your pipeline.

So the honest framing is breaks-versus-still-wins, not a blanket recommendation. GPU-bound: fix the batch. Virtualized: fix the host allocation. Dataloader-bound: keep the performance governor. Run the two-configuration test first, and only adopt performance mode permanently if p99 latency drops enough to justify the power draw.

64-image batch A/B

Run the A/B before you touch a single line of model code. Fix one batch of 64 images, one model checkpoint, one seed, and one warm GPU — push a throwaway batch through first so CUDA graph capture, allocator growth, and kernel autotuning are already paid for. Then time 100 iterations under cpupower frequency-set -g powersave and record p50 and p99 batch latency in milliseconds. That powersave arm is your baseline; everything after it is a delta against a number you actually measured, not a number you assumed.

Now switch to cpupower frequency-set -g performance and repeat the identical 100 iterations. At each arm, capture three checkpoints. First, the contents of /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor for every CPU — read them, don't trust that the set command applied uniformly across a heterogeneous or partially offline CPU set. Second, p99 batch latency in milliseconds from the 100 timed iterations. Third, package power and peak CPU temperature from turbostat sampled across the run. All three checkpoints, both arms, same machine, same thermal starting point.

The comparison rule is narrow on purpose: adopt performance mode permanently only if p99 latency drops by more than 10% over the powersave baseline while PkgWatt and peak temperature stay inside the envelope your cooling and power budget already tolerate. If p99 barely moves, you are not CPU-frequency-bound and the governor is the wrong lever — go find the real bottleneck. The arXiv study on feasibility and trade-offs of inference systems is the anchor here: it asks directly how power modes and threading affect inference efficiency, including microarchitectural metrics like cache misses and context switches, which is exactly the layer a governor change perturbs. The Geekbench database entry for a powersave-governor board is the second anchor — a published configuration where the governor is recorded as a first-class variable rather than an afterthought.

Price the mistake in both directions. Staying on powersave when you are CPU-bound means shipping latency you could have removed for free — no code change, no retraining, no new hardware, just a governor flip and pinned threads. Forcing performance when you are not CPU-bound buys you nothing on p99 and costs you sustained package power and thermals, which on a thermally constrained box can quietly erode the very latency you were chasing. The A/B is what tells you which side of that line you are on.

Copy-usable checklist: fixed 64-image batch, fixed checkpoint, fixed seed, one warm-up batch, 100 timed iterations per arm, governor file contents read per CPU, p99 in ms, turbostat PkgWatt and peak temp, both arms recorded before any decision. Pin your threads and disable deep C-states in the same pass — the governor is the highest-leverage single change, but it is not the only one, and measuring them together is what keeps the result honest.

Worked Example: Run the Numbers

Here is the worked example, end to end. Scenario: a single Linux inference node serving a vision model in 64-image batches, with a hard product requirement that p99 batch latency stay under 100 ms. The party is you, the operator, and the test window is a single maintenance day in 2026 — same machine, same model weights, same batch of 64 images, run back to back so thermal and load conditions are as close to identical as you can make them. The only variable you change is the CPU governor: run once with cpupower frequency-set -g powersave, then again with cpupower frequency-set -g performance, recording p99 batch latency from your serving framework's own timing logs both times.

Step one is the raw delta. Call the powersave run's p99 latency P and the performance run's p99 latency F. The governor tax is Δ = P − F. Step two is the share: divide Δ by P to get the fraction of your powersave latency that is pure scheduler and frequency-transition overhead rather than model compute. Step three is the decision check: if F already lands under 100 ms, you have headroom; if F is still over 100 ms, the governor was not your binding constraint and you should look elsewhere before touching model code.

To make the arithmetic concrete, use your own two-run numbers. Suppose your powersave p99 is P and your performance p99 is F. The governor tax is Δ = P − F, and the overhead share is Δ ÷ P. For example, if P = 120 ms and F = 95 ms, then Δ = 25 ms and the overhead share is about 21 percent. If F already lands under 100 ms, you have headroom; if F is still over 100 ms, the governor was not your binding constraint and you should look elsewhere before touching model code.

he performance run clears the 100 ms requirement with 5 ms to spare. Substitute your own measured P and F; the two formulas — subtraction for the delta, division for the share — are the entire calculation.

The winner for this example is the performance governor, and the break-even trigger is a condition, not a number: adopt performance mode permanently the moment your measured F beats your measured P and F sits under your latency requirement, provided the machine's power draw and thermals stay within your operating envelope. If Δ comes back near zero, the governor was not your bottleneck — keep powersave and move on to the next suspect. The point of running the numbers is that either outcome is a win: you either found free latency or you eliminated a variable.

Two checks make the result trustworthy. First, run each configuration several times and compare p99, not the mean — the arXiv study on inference system feasibility and trade-offs frames power modes and threading as measurable system configuration variables, which is exactly how you should treat them. Second, pin your inference threads to specific cores and disable deep C-states for the performance run, so the governor comparison is not contaminated by scheduler migration; if you skip that, a favorable Δ may understate or overstate what the governor alone contributes.

Decision rules for your serving stack

Treat the following as the operational form of the governor rule: each condition below selects a different action, and none of them require touching model code first. Every branch assumes you already hold the paired powersave/performance measurements, plus a package temperature and per-core clock trace captured during the batch itself.

If p99 drops more than 10% under the performance governor and package temperature stays flat through the run, make the change permanent. Put cpupower frequency-set -g performance in the startup path of the serving unit — an ExecStartPre line in the systemd unit, not a shell profile — so it reapplies on reboot and service restart rather than only in the terminal where you tested it. Then rerun the 64-image batch from a cold service start to confirm the gain survives process initialization.

If p99 moves less than 10%, or thermals climb and clocks fall back under sustained load, revert to powersave. Run the A/B twice with the order reversed; a one-shot difference that does not reproduce is scheduler noise, not a governor effect. Reverting costs nothing and preserves thermal headroom for workloads that genuinely need burst clocks.

If the host is a managed VM and the governor is not yours to set, take the partial credit. Pin inference threads with taskset or numactl --physcpubind so they stop migrating between cores, and disable deep idle states wherever the provider exposes the control. Pinning plus shallow idle will not fully recover the performance-governor result, but it removes scheduler migration and wake latency at no infrastructure cost.

If SM utilization is already above 95% during the batch, leave the governor alone. At that point the CPU is waiting on the accelerator, not the reverse, and the leverage sits in batch size, precision (FP16 or INT8 where accuracy allows), and kernel launch overhead. Changing the governor there buys nothing and adds power draw.

If the machine is shared with co-tenants, scope the change: apply performance mode only for the batch window, or bound the frequency ceiling instead of running the whole package at maximum, since a host-wide performance governor raises power and heat for idle work too. Whichever branch you land on, re-measure after any model, kernel, or driver upgrade — the crossover point is a property of the current stack, not a permanent setting.

What to do next

Frequently Asked Questions

What command shows the exact C-state exit latency penalty my CPU charges?

Running cat /sys/devices/system/cpu/cpu0/cpuidle/state*/latency prints the exit latency for each state in microseconds.

If my cores are parking in C6 or C8, does that limit how fast the next batch can start?

Yes, that number is the floor on how fast the next batch can possibly start, no matter how fast the GPU is.

Why does the first stretch of each batch run at low clocks under schedutil or powersave?

The schedutil and powersave governors only raise frequency after they see load, so the first stretch of each batch runs at low clocks while the governor decides the workload is real.

What tool lets me watch package frequency climb after work has already started?

Running turbostat --interval 1 during a batch shows the package frequency climb after the work has already started.

Is P-state ramp latency a big deal for a long training run?

For a long training run this ramp is noise, but for a sub-100 ms batch it's a meaningful chunk of the budget, spent at the wrong clock.

What two separate hardware delays does a powersave governor inject into every batch?

The first is C-state exit latency and the second is P-state ramp latency.

Quick answers

Also worth reading: 4K vs 1080p: Diffusion Inference Cost Multiplier in 2026: 4K vs 1080p: Diffusion Inference · Scheduled Batching Cuts Diffusion Inference Cost by 38%: Scheduled Batching Cuts Diffusion Inference · SDXL Wins Cost, Reach, and Batch: FLUX Price Data Compared: SDXL Wins Cost, Reach, and