Image generation costs: 80GB H100 Half Precision (FP16) vs Full Precision (FP32)

38% savings for half precision versus full precision for image generation, reported by diffusioninference.com, reframes FP32 as a premium option rather than a default for large volume production right now under tight budget pressure.

For teams running diffusion workloads on the H100, the mechanism is straightforward. Converting the UNet to FP16 cuts memory usage by 50% and lets modern GPUs move faster, which lowers billed time for identical prompts and steps. That makes FP16 inference production-identical for everyday catalogs, previews, and iterations, while reserving full precision end to end for rare archival hero images.

The scale economics are already proven in public cloud runs. Salad Technologies GPU cloud pricing starts at $0.02 per hour, with a large bulk run totaling $1,872. At that scale, sticking with FP32 safety across everything wastes budget every day, whereas a disciplined FP16 default keeps quality steady and frees spend for the few images that truly demand maximum fidelity.

Inside H100 Tensor Cores

On an 80GB H100 SXM, the decision is memory movement, not math. Stable Diffusion XL carries a 2.6B-parameter UNet, which at 4 bytes per parameter in FP32 occupies roughly twice the weight footprint as at 2 bytes per parameter in FP16. That halving is what lets production configs step from batch-size 4 to batch-size 8 without spilling activations or sharding, because weights plus attention caches plus latents for eight 1024px jobs finally fit comfortably with headroom for the VAE held separately in FP32.

Peak dense throughput explains why teams overestimate the win. The H100 SXM path for FP16 Tensor Core work is an order of magnitude above the FP32 CUDA-core path, yet a 50-step denoising loop with classifier-free guidance does not realize that ratio. Each step launches two UNet forwards, reloads weights from HBM, and waits on attention memory traffic and scheduler overhead. In practice that memory-bandwidth limit compresses the theoretical gap to roughly a 1.61x realized speedup across the full loop, which is exactly why batching matters more than clock speed here.

The bottleneck to optimize is HBM traffic. Think of 50 UNet forwards per image as 50 reloads of the same large weight set plus streaming of keys, values, and latents. Halving precision roughly halves that data movement per 1,000 1024px images, from the multi-terabyte range to about half that volume. According to diffusioninference.com, 2026, that reduction is what drives the 38% saving in cost per 1,000 images, because billed GPU-seconds fall even when quality evaluators cannot distinguish the outputs once VAE decode stays in FP32.

The implementation skill is scoping CUDA Automatic Mixed Precision correctly. Keep OpenCLIP ViT-G/14 text encoders plus the UNet inside autocast in FP16, where matrix multiplies map cleanly to Tensor Cores and dynamic range is well behaved. Hold scheduler math and VAE decode outside autocast in FP32. The scheduler accumulates small beta, alpha, and sigma updates over 30 to 50 steps where FP16 rounding compounds, and the VAE expands a compressed latent to full RGB where banding and washed highlights become visible. That split is the canonical rule: FP16 everywhere except FP32 VAE decode, reserving full end-to-end FP32 only for text-heavy hero images where legible typography justifies the cost.

Utilization is where the invoice changes. In FP32 batch inference on H100, SM occupancy typically sits in the low-60% range because warps stall waiting on HBM. In FP16 with batch-size 8, occupancy moves into the low-90% range because Tensor Cores stay fed and larger tiles keep more concurrent blocks resident. At 30 steps that shift cuts billed GPU-seconds per image from the mid-3-second range to the low-2-second range, which compounds across a nightly batch of thousands of images without changing prompts, seeds, or sampler settings.

Do this next: set autocast to include the UNet and both text encoders, force the scheduler step and VAE decode to FP32, then raise batch size until HBM headroom stabilizes at 8 on 80GB. If small text still breaks, do not revert the whole batch to FP32 — isolate that prompt as a hero rerun. That preserves the batch saving while fixing the one edge case where full precision earns its keep.

A100 Receipts

Lambda Labs' April 2024 SDXL benchmark on an A100 40GB is the receipt that settles batch planning: 2.1 sec/image in FP16 versus 3.4 sec/image in FP32 at 1024px and 30 steps, a 38.2% time saving. According to Lambda Labs, that delta holds with VAE decode kept in FP32, which is why perceptible quality does not move while throughput does. For production scheduling, treat FP32 end-to-end as the exception for text-heavy hero images only, and run everything else in FP16 with FP32 VAE decode on Tensor Core GPUs.

As a vision researcher who profiles batch inference, I read that speedup as memory movement, not just math. According to the Hugging Face Diffusers team measurement, the SDXL pipeline peaks at 14.7GB VRAM in FP16 versus 23.9GB in FP32 on A100, which allows doubled batch without OOM. That headroom is what lets you keep the Tensor Cores fed instead of stalling on transfers, and it is the reason the time saving compounds into cost saving at scale. According to gmicloud.ai on April 13, 2026, the UNet forward pass accounts for 70-80% of total generation time, making it the primary target for latency reduction, while CLIP text encoding is performed once per generation regardless of step count.

The myth to kill is that cheaper hourly rental beats precision choice. According to HackerNoon on August 31, 2023, Salad Technologies GPU cloud pricing starts at $0.02 per hour, yet a low hourly rate on the wrong precision still loses because you pay for seconds per image. Translate current rental into per-image cost and FP16 wins explicitly. According to RunPod Secure Cloud March 2026 pricing for A100 80GB at $1.64/hr, cost converts to $0.95 per 1,000 images in FP16 versus $1.53 in FP32 at measured throughputs. That is not a synthetic benchmark; it is division of hourly price by observed images per hour.

Serverless comparisons tell the same story after overhead. According to Together AI serverless diffusion pricing analysis, self-hosted FP16 costs $0.00124/image versus $0.00200 hosted FP32-equivalent, a 38% gap after queue overhead. The queue matters because autoscaling gaps and cold starts tax hosted FP32 more heavily, while a warm FP16 batch worker sustains higher images per GPU-hour. According to gmicloud.ai on April 13, 2026, selective INT8 quantization offers 2-3x additional acceleration for compatible layers within diffusion models, but for 1024px production batches that technique is an optional second step, not a substitute for the FP16 with FP32 VAE decode baseline.

Reproducibility is why I trust the A100 number for budgeting. According to the Databricks MosaicML 2025 diffusion serving report, mean cost reduction was 38% across three A100 runs with range 37.1% to 39.0% for 1024px batches. Use the low end for conservative bids and the mean for internal planning, then enforce the canonical rule in your job spec: FP16 everywhere with FP32 VAE decode, reserving full end-to-end FP32 only for text-heavy hero images. The next action is to pin batch size to the 14.7GB FP16 peak, not the 23.9GB FP32 peak, and re-price your 1,000-image lot at the FP16 throughput.

FP16 vs FP32 on L40S

On the L40S, the decision is not about precision; it is about memory bandwidth and batch density. The 48GB VRAM constraint forces a trade-off that FP16 resolves by halving the weight footprint, allowing higher concurrency. According to gmicloud.ai (2026-04-13), FP32 to FP16 conversion yields a 1.5-2x speed improvement on modern GPUs while reducing memory usage by 50%. This 50% reduction in memory consumption is the mechanical driver for the throughput gains observed in production pipelines.

The cost differential is stark: $0.71 per 1,000 images in FP16 versus $1.14 in FP32 at TensorDock’s $0.82/hr rate. This 38% savings compounds directly with the 1.9s vs 3.1s latency gap. However, the real leverage lies in VRAM headroom. At 1024px resolution, FP16 peaks at 11.8GB, permitting a batch size of 8 before OOM errors. FP32 consumes 22.4GB, capping batches at 3. This allows FP16 to process nearly triple the volume per second, amortizing the fixed hourly GPU cost over more units.

Evaluation metrics from the MJHQ-30K dataset confirm no perceptible quality loss. FID scores are 12.3 for FP16 versus 12.1 for FP32, and CLIP ViT-B/32 scores are 0.312 versus 0.314. These deltas fall below human-noticeable thresholds. The only significant risk is failure rate: FP16 exhibits a 0.8% black-image rate compared to 0.3% for FP32. Applying an FP32 VAE decode fix mitigates most artifacts, but retries still add approximately $0.006 per 1,000 images. Even with this overhead, FP16 remains cheaper.

For any batch exceeding 200 images daily on an L40S, FP16 is the explicit winner due to throughput and cost efficiency. FP32 is reserved only for single archival masters where the $0.0004 extra cost per image is irrelevant to the bottom line.

What the Data Doesn't Tell You

FP16 is not a universal constant; it is a conditional optimization that collapses under specific mathematical and hardware constraints. While the 38% cost reduction holds for standard batch inference, the precision ceiling of FP16 introduces hard failure modes that are invisible in aggregate metrics but catastrophic at the edge. The primary mechanism of failure is the FP16 dynamic range limit of 65,504. In Stable Diffusion v1.5, when guidance scales exceed 12 or prompts invoke high-energy neon HDR aesthetics, intermediate activations overflow this ceiling. This does not result in subtle artifacts; it triggers NaN propagation through the UNet, yielding full-black frames that require immediate regeneration. For production pipelines processing thousands of images, these silent failures represent a hidden latency tax that erodes the efficiency gains of half-precision math.

Beyond binary failure, FP16 introduces quantization noise that degrades perceptual quality in low-frequency gradients. When the VAE decoder operates in FP16 rather than FP32, dark-gradient banding becomes statistically significant. On night-sky gradients, this precision loss adds a +0.04 penalty to LPIPS scores compared to smooth FP32 decoding, manifesting as visible 8-bit posterization. Furthermore, legibility collapse occurs for sub-12px text and fine facial details. In end-to-end FP16 runs, EasyOCR accuracy drops by 11 points, and CelebA-HQ facial cosine similarity decreases by 0.07. These deltas are negligible for abstract art but disqualify FP16 for text-heavy hero images where semantic fidelity is paramount.

Sampler variance further complicates the FP16 advantage. Euler Ancestral sampling in FP16 exhibits 2.3x higher seed-to-seed variance than DPM-Solver++, with FID standard deviations fluctuating by +/-1.8 across 5k-image runs. This instability makes deterministic reproducibility difficult without extensive seed locking. Finally, hardware architecture dictates the cost curve. On GTX 1080 Ti systems lacking Tensor Cores, FP16 emulation runs 14% slower with a 6% retry rate due to software fallback overhead. In this specific legacy environment, FP32 remains 9% cheaper per usable image. According to Medium: Running Stable Diffusion at Scale Without Going Broke, the RTX 4090’s 24GB VRAM handles SDXL comfortably at $0.28/hr rental rates, but only if the workload avoids the FP16 overflow traps identified above. Always verify prompt complexity against the 65,504 ceiling before committing to half-precision batches.

10,000 Images on A10G

Adding S3 Standard storage for the 10,000 PNGs at 8.2GB total incurs $0.19 in storage fees plus $0.42 in egress charges. These costs are identical for both precisions and do not alter the financial winner. However, the true test lies in the QA loop. ImageReward scores sit at 0.82 for FP16 versus 0.83 for FP32. While the difference appears marginal, it triggers a 1.2% reshoot rate for the FP16 batch, requiring 120 additional images. At the FP16 compute rate, this reshoot costs $0.49, leaving a net billed saving of $24.61 for the entire batch. The mechanism here is clear: the slight quality variance is economically negligible compared to the raw compute savings.

This evidence converges on the thesis that FP16 diffusion generation with FP32 VAE decode is the optimal production strategy for A10G instances. The A10G's architecture favors the speed of half-precision UNet passes, while keeping the VAE decode in FP32 preserves the final pixel fidelity required for production assets. The data confirms that the "quality loss" argument against FP16 is mathematically outweighed by the 38% cost reduction, provided the VAE remains in full precision.

Default to FP16 UNet with FP32 VAE decode on Tensor Core GPUs, and pull only the text-heavy hero subset back to full FP32. That split is the entire cost control in 2026 production batch inference, because the UNet dominates compute while the VAE decode determines whether fine edges survive.

Start with volume and hardware. If you render more than 200 images per day at full production frame size and nvidia-smi reports compute capability 8.0 or higher, lock the pipeline to FP16 UNet with FP32 VAE decode. The mechanism is Tensor Core throughput on matrix multiplies in attention and ResNet blocks, while the VAE stays in higher precision to preserve gradient range in dark areas. Below that volume threshold or on older silicon, the orchestration overhead outweighs the gain, so keep your existing path.

How to Choose Well

Isolate the exception before you optimize the bulk. If the brief explicitly requires legible logos, sub-12px text, or photoreal hero faces for print at 2K, route that under-5% subset to full end-to-end FP32 and keep everything else in FP16. According to Hacker News item id 37504349, an expanded InstaFlow network with 1.7B parameters improves FID to 22.4 on MS COCO 2017-5k, which is useful context here: global FID can look excellent while tiny glyphs still break. Do not judge logo legibility with FID; judge it by pixel inspection of that small subset.

Watch guidance and palette as a NaN trigger. If guidance scale exceeds 9.0 or the prompt calls for HDR neon plus dark gradients, force FP32 VAE decode with TensorRT attention slicing. High guidance amplifies latent magnitudes, and saturated primaries against near-black push the decoder into a range where reduced precision overflows. The fix is not to drop the whole job to FP32, just to hold the decoder in FP32 and slice attention so peak activation memory stays bounded.

Use memory pressure as a routing signal, not a failure. If FP32 OOMs at batch 4 or leaves under 4GB VRAM headroom, switch to FP16 batch 8 but gate the full run on a 100-image ImageReward spot-check within 0.02. In practice that means render 100 varied prompts, score them, compare to your FP32 reference, and only launch the bulk queue if the gap stays inside that tolerance. If fallback is CPU-only or pre-2018 GPU without Tensor Cores, do not use FP16 at all; stay FP32 or move to INT8 via xFormers quantized VAE path, because without Tensor Cores you get the numerical risk with roughly none of the speedup and behavior varies by driver.

Watch guidance and palette as a NaN trigger. If guidance scale exceeds 9.0 or the prompt calls for HDR neon plus dark gradients, force FP32 VAE decode with TensorRT attention slicing. High guidance amplifies latent magnitudes, and saturated primaries against near-black push the decoder into a range where reduced precision overflows. The fix is not to drop the whole job to FP32, just to hold the decoder in FP32 and slice attention so peak activation memory stays bounded.

Use memory pressure as a routing signal, not a failure. If FP32 OOMs at batch 4 or leaves under 4GB VRAM headroom, switch to FP16 batch 8 but gate the full run on a 100-image ImageReward spot-check within 0.02. In practice that means render 100 varied prompts, score them, compare to your FP32 reference, and only launch the bulk queue if the gap stays inside that tolerance. If fallback is CPU-only or pre-2018 GPU without Tensor Cores, do not use FP16 at all; stay FP32 or move to INT8 via xFormers quantized VAE path, because without Tensor Cores you get the numerical risk with roughly none of the speedup and behavior varies by driver.

What to do next

Frequently Asked Questions

What is the specific VRAM usage difference for Stable Diffusion XL on an A100 between FP16 and FP32?

According to the Hugging Face Diffusers team measurement, the SDXL pipeline peaks at 14.7GB VRAM in FP16 versus 23.9GB in FP32 on A100.

How does keeping the VAE decode in FP32 affect the cost savings achieved by using FP16 for the rest of the pipeline?

The 38% time saving reported by Lambda Labs holds with VAE decode kept in FP32, which is why perceptible quality does not move while throughput does.

What is the realized speedup ratio for a full 50-step denoising loop compared to the theoretical Tensor Core advantage?

In practice that memory-bandwidth limit compresses the theoretical gap to roughly a 1.61x realized speedup across the full loop.

Which components should be excluded from Automatic Mixed Precision to prevent rounding errors or visual artifacts?

Hold scheduler math and VAE decode outside autocast in FP32 because FP16 rounding compounds over steps and banding becomes visible during VAE expansion.

What is the estimated per-image cost difference between self-hosted FP16 and hosted FP32-equivalent services according to Together AI?

Self-hosted FP16 costs $0.00124/image versus $0.00200 hosted FP32-equivalent, a 38% gap after queue overhead.

At what batch size does an 80GB H100 SXM typically stabilize headroom when running mixed precision inference?

Raise batch size until HBM headroom stabilizes at 8 on 80GB because weights plus attention caches plus latents for eight 1024px jobs finally fit comfortably.

Quick answers

Also worth reading: CMMD vs FID: 5-to-1 Decision Verdict, Cost Is FID's Only Win: CMMD vs FID: 5-to-1 Decision · Why Static FID And CLIP Fail 2026 Diffusion CI/CD Pipelines: Why Static FID And CLIP · Image generation speed: 2026 A100 Stable Diffusion XL Batch 16 beats 32: Image generation speed: 2026 A100