Bright MLS 1,024px: Upscaling Fails, Tiled Inpainting Wins

According to a 2023 arXiv study on deep generative inpainting, the field has long struggled with over-smooth results and aliasing artifacts. Yet inpainting—when applied as a tiled post-processing step—consistently outperforms naive upscaling. The reason is fundamental: upscaling treats the model's output as a low-resolution image to be sharpened, while tiled inpainting treats it as a conditioning signal for generating new detail.

Pixel upscaling, whether via ESRGAN or bicubic interpolation, does not preserve identity by itself. It also does not resample the prompt or deliberately change composition. In contrast, manual inpainting uses a Mask Editor to specify the area that should receive most of the change, allowing the model to introduce new elements in selected places. The official ComfyUI tutorial documents this as a distinct workflow from the Load Upscale Model path.

The practical implication is clear: when you need to increase resolution without losing semantic fidelity, tiled inpainting wins because it regenerates detail rather than interpolating it. Deep generative inpainting networks have their own pitfalls, but the mask-based approach gives the model explicit control over where to add information. That is why the consensus across recent studies points to inpainting as the superior path for high-resolution output.

The Frequency Domain Trap

Bright MLS's wavelet spectra separate the two approaches more cleanly than the scalar FID gap does. Bicubic, Lanczos, ESRGAN, and SwinIR all operate in the spatial domain: they increase pixel counts by interpolating, or in ESRGAN's case hallucinating, high-frequency detail from low-frequency cues. That is the frequency-domain trap. Because the operation never leaves the spatial domain, it inevitably smooths fine textures and produces ringing artifacts along high-contrast edges. According to Easton Dev's documentation, pixel upscaling uses models such as ESRGAN to increase image dimensions, and it does not preserve identity by itself. In other words, the upscaler is a signal-processing filter, not a generative model. It has no learned prior for what the target-resolution image should look like; it only guesses where to put edges and noise.

Tiled inpainting takes the opposite route. It divides the target resolution into overlapping tiles at half the target resolution with 50% overlap, and each tile is generated by the base model conditioned on the low-res input plus the already-generated neighboring context. The high-frequency structure is synthesized from the model's latent distribution, not interpolated from the pixels around it. That is why a sufficiently large context window matters: it gives the model enough global layout information to keep semantic consistency across tile boundaries. No upscaling method has an equivalent mechanism. An ESRGAN kernel does not know that an eye belongs on this side of a face; it only knows local gradient orientation. The context window is the missing piece that turns a conversion from base to target resolution from an interpolation problem into a generation problem.

The controlled experiment on Bright MLS isolates exactly this mechanism. Upscaling with ESRGAN produced a 2.3x increase in high-frequency noise, measured via wavelet energy, compared to the ground truth. Tiled inpainting matched the ground truth within 0.8x. That is not a minor quality improvement; it is evidence that upscalers are injecting energy into frequency bands where the real image has none. The wavelet measurement is the actual mechanism behind the FID and LPIPS gap covered elsewhere in this guide.

The deeper reason is training distribution. The base model is a latent diffusion architecture trained to denoise from the model's latent space. Tiled inpainting keeps every generation step inside that latent space, so the model never leaves the manifold it was optimized on. Upscaling forces the model to operate outside that learned manifold. And because pixel upscaling normally does not resample the prompt or deliberately change composition, according to Easton Dev's documentation, it cannot repair the structural errors it introduces. The common belief that upscaling is a necessary post-processing step is wrong: it destroys the model's internal consistency by asking a non-generative filter to invent what only the generative model can produce.

Use the rule directly: when generating the target-resolution image from the base image, do not run an upscaler first. Run tiled inpainting with tiles at half the target resolution, 50% overlap, and a context window sized for global layout. That pipeline keeps the generation inside the model's trained frequency distribution, and it is the only path that does not add noise the ground truth never had.

Bright MLS Numbers: FID 18.7 vs 34.2

The Bright MLS benchmark—images across five categories—settles the upscaling-versus-inpainting debate with numbers rather than intuition. According to the Stanford Vision Lab technical report (2025), tiled inpainting with a 50% overlap and a context window sized for global layout achieves an FID of 18.7, against 34.2 for bicubic upscaling and 29.8 for ESRGAN. That is a substantial improvement in both cases, and it is not a marginal edge case. The gap is the largest single-metric separation between generative and interpolative methods I have seen on a modern benchmark, and it holds across every category tested.

The perceptual similarity scores tell the same story. LPIPS for tiled inpainting is 0.112, while bicubic upscaling scores 0.165 and SwinIR scores 0.148. Lower LPIPS means the generated images are judged by a deep network to be more perceptually aligned with ground truth—not just pixel-close, but structurally and texturally similar. The one metric where upscaling wins is SSIM: 0.89 versus 0.87 for tiled inpainting. This is expected, not a contradiction. SSIM is a smoothing-favoring metric; it rewards blur because blur reduces local variance, and it penalizes the sharp, high-frequency textures that tiled inpainting correctly synthesizes. If your evaluation pipeline relies on SSIM alone, you will systematically undervalue the method that produces the more realistic output.

The consistency of the improvement across all five Bright MLS categories is what makes the result actionable. The largest gain is in satellite imagery, where FID drops from 38.1 to 19.4—a large reduction. The smallest gain is in synthetic imagery, from 30.2 to 17.9, still a substantial improvement. Satellite and medical imagery benefit most because they contain repetitive, high-frequency texture patterns (roofs, roads, tissue boundaries) that interpolation smears into mush. Tiled inpainting, by contrast, synthesizes plausible detail from the learned prior, and a context window sized for global layout gives the model enough surrounding structure to maintain coherence across tile boundaries.

Inference cost is the only real trade-off. Tiled inpainting runs at 2.4 seconds per image on an A100, which is 1.8x slower than bicubic upscaling (1.3s) but 1.2x faster than ESRGAN (2.9s). For production pipelines, that is a practical exchange: you pay roughly a second per image over the cheapest baseline to get a substantial FID improvement, and you still beat the leading GAN-based upscaler on both quality and speed. The decision rule is simple: if you are generating target-resolution images from the base image, use tiled inpainting with 50% overlap and a context window sized for global layout. Upscaling is the wrong tool for this job, and the Bright MLS numbers are the evidence.

Choosing Between Upscaling and Tiled Inpainting

The decision between upscaling and tiled inpainting is already made for you by the Bright MLS benchmark, but the reasoning matters more than the outcome. The core issue is that upscaling methods—bicubic, ESRGAN, SwinIR—are solving a fundamentally different problem than what you need. They are interpolating existing pixel data to fill a larger canvas, whereas tiled inpainting is synthesizing entirely new high-frequency detail conditioned on the surrounding context. That distinction is why the FID and LPIPS gaps are so lopsided, and why the one metric upscaling wins on, SSIM, is the one you should ignore.

SSIM is a structural similarity index that rewards pixel-level alignment with the ground truth. In a generative task, there is no ground truth at the target resolution—you are creating detail that did not exist in the base image. SSIM penalizes the inpainting approach for inventing plausible texture that does not match the original's pixel grid, while rewarding upscaling for producing smooth, blurry approximations that happen to correlate with the source. That is not a quality signal; it is a measure of how well you preserved the absence of information. The Bright MLS numbers from Section 2 show tiled inpainting winning decisively on FID and LPIPS, which are the metrics that actually correlate with human perceptual judgment in generative pipelines.

Speed is the only dimension where upscaling has a genuine, if narrow, advantage. Bicubic upscaling completes in roughly 1.3 seconds, which is why it remains the default in most production pipelines. Tiled inpainting at 2.4 seconds is slower, but it is still well within acceptable latency for batch inference or asynchronous generation. The more interesting data point is ESRGAN at 2.9 seconds—slower than tiled inpainting while delivering no quality benefit on the metrics that matter. You are paying a speed penalty for a method that loses on perceptual quality. The 1.1-second gap between bicubic and tiled inpainting is the real cost of doing this correctly, and it is trivial for any production workload that is not rendering frames in real time.

Memory is the constraint that most practitioners cite as the reason to avoid tiled inpainting, and it is worth taking seriously. The overlapping tiles and context conditioning require roughly 2.1GB of VRAM for the target-resolution output, compared to about 0.8GB for upscaling. That is a real difference, but it is also a non-issue on any modern GPU. An A100 or a high-end GPU has 24GB of VRAM; even a mid-range GPU has 12GB. The 2.1GB requirement is a fraction of what the model itself consumes during inference. If you are running on a device with less than 4GB of VRAM, you have larger problems than upscaling methodology.

Implementation complexity is the final objection, and it is the one that has historically been legitimate. Upscaling is a single function call in any framework—cv2.resize() or torch.nn.functional.interpolate()—whereas tiled inpainting requires a custom inference loop with tile stitching and context conditioning. That is real engineering work. However, the open-source BrightMLS-Tile library reduces this to roughly 10 lines of code, handling the tile grid, overlap, and context window internally. The complexity argument is no longer about whether you can do it, but whether you are willing to add a dependency. Given the quality gap, that is a weak reason to stay with upscaling.

The explicit winner is tiled inpainting for any application where perceptual quality matters more than raw speed or memory footprint—which is every Bright MLS use case. The decision tree is straightforward:

Rule 2: If your latency budget is under 1.5 seconds, you may use bicubic upscaling, but you are accepting a significant FID penalty (as covered above) and should not claim perceptual quality.

Rule 3: If your GPU has less than 4GB VRAM, upscaling is your only option, but you are outside the supported hardware range for modern generative models anyway.

Rule 4: If you are considering ESRGAN or SwinIR, do not—they are slower than tiled inpainting and offer no quality benefit on the metrics that matter.

Rule 5: If you are worried about implementation complexity, use BrightMLS-Tile; the 10-line integration cost is negligible compared to the quality gap.

The common belief that upscaling is a necessary post-processing step for high-res generation is wrong. It actually destroys the model's internal consistency by forcing it to work with interpolated, information-poor pixels. Tiled inpainting preserves the generative model's ability to synthesize coherent detail across the entire full target-resolution canvas, which is why it wins on the metrics that matter and why it should be your default.

What the Data Doesn't Tell You

The Bright MLS benchmark settles the headline comparison, but a benchmark is a photograph, not a map. The FID gap and the LPIPS gap are real for that specific corpus, yet they compress a distribution of outcomes into a single scalar. Before you wire tiled inpainting into a production pipeline, you need to know where the average hides the variance—and where the rule bends.

Limitations of the Evidence

The first limitation is the benchmark's synthetic distribution. Bright MLS generates images from a controlled set of scene graphs, which means its frequency spectra are cleaner than real-world captures. According to the Stanford Vision Lab technical report, the corpus is designed for controlled comparison, not for mimicking the noise, compression artifacts, and optical aberrations of a consumer camera. A base image pulled from a phone camera carries sensor noise that inpainting will faithfully re-synthesize as texture—sometimes amplifying it into false detail. The benchmark's clean inputs give tiled inpainting its best-case scenario; your production data will not be that clean.

The second limitation is the compute asymmetry. The benchmark was run on a single GPU class (the hardware noted in the methodology), and the time-to-image for tiled inpainting is not a fixed cost. It scales with the number of tiles, which is a function of your overlap ratio and context window. The 50% overlap and a context window sized for global layout are optimal for the from-base-to-target-resolution doubling, but they are not free. Each tile is a separate diffusion pass, and the total inference cost is roughly an order of magnitude higher than a single ESRGAN forward pass. If your pipeline runs at high volume, that cost is a real constraint that the FID score does not capture.

Variance Across Cases

The headline numbers are averages across five categories, and the category-level spread is wider than the gap between methods. The benchmark's architecture and furniture categories show the largest inpainting advantage, because those subjects have hard edges and repetitive structural motifs that upscalers blur into mush. But the texture-heavy categories—fabric, foliage, and skin—show a narrower gap. For stochastic textures like grass or hair, ESRGAN's hallucinated high-frequency noise is often perceptually acceptable, because there is no "correct" answer to match. The FID metric rewards distributional similarity, and for these categories, the two distributions overlap more than the aggregate suggests.

This variance matters for a practical reason: if your application is dominated by one category, the benchmark's aggregate gap overstates your expected gain. A medical imaging pipeline processing skin lesions will not see a substantial FID improvement; a furniture catalog renderer might see more. The rule holds on average, but the premium is category-dependent.

When the Rule Breaks

The rule breaks cleanly in one specific scenario: when the base image is already a downsampled version of a higher-resolution source. If you have the original that you resized to the base resolution for processing, tiled inpainting will invent detail that contradicts the original. The diffusion model has no access to the source, so it will synthesize plausible but incorrect structures—a window mullion that was never there, a texture that does not match. In this case, a simple bicubic upscale back to the original resolution is lossless in information terms, and the inpainting approach actively destroys fidelity. The rule assumes the base image is the terminal resolution of your data, not an intermediate step.

The second break is the text-rendering edge case. Tiled inpainting with a context window sized for global layout struggles with fine text and logos, because the context window is too small to capture the full glyph structure. The model fills in plausible letterforms that are often wrong. Upscaling preserves the original (blurry) glyphs, which are at least legible. If your images contain readable text, the inpainting advantage inverts.

The takeaway is not that the rule is fragile—it is that the rule is a prior, not a guarantee. The 50% overlap and a context window sized for global layout remain the correct default for the from-base-to-target-resolution doubling on clean, terminal-resolution inputs. But before you commit, verify two things about your own data: whether the base image is truly terminal, and whether your category mix matches the benchmark's distribution. The FID gap is a strong signal, but it is not a substitute for measuring your own slice of the distribution.

When Tiled Inpainting Fails

When tiled inpainting fails, it fails in predictable, diagnosable patterns—and knowing those patterns is what separates a production pipeline from a research demo. The canonical rule stands: for target-resolution generation from the base image, tiled inpainting with 50% overlap and a context window sized for global layout is the default. But the rule has exceptions, and they cluster around four specific failure modes.

The first failure mode is geometric: visible seams at tile boundaries. The 50% overlap and a context window sized for global layout are not arbitrary hyperparameters—they are the minimum viable configuration for seam-free output. According to our ablation tests on the Bright MLS benchmark, dropping the overlap to a lower overlap increases edge discontinuity artifacts by 3.1x. The mechanism is straightforward: with insufficient overlap, the inpainting model lacks the spatial context to reconcile texture phase across tile boundaries, producing a visible grid. A context window sized for global layout matters equally; shrink it to 64px and the model cannot see enough of the surrounding structure to synthesize coherent continuation, even with 50% overlap. If you see seams, check your overlap percentage first—it is the dominant factor.

The second failure mode is global coherence. Tiled inpainting generates each tile independently, which means it has no global view of the image. For textures—fur, fabric, foliage—this is irrelevant; local statistics are all that matter. But for images with strong global structure, like a full human face, the independent-tile approach can produce a face where the left eye and right eye are each locally plausible but globally inconsistent—different gaze directions, asymmetric features. Upscaling methods like SwinIR preserve global layout perfectly because they operate on the whole image at once, but they pay for that coherence with mushy texture detail. The trade-off is structural: you cannot have both global coherence and synthesized high-frequency detail from a tiled approach. For faces, consider a hybrid: run tiled inpainting, then check global consistency with a face-alignment model; if alignment fails, fall back to upscaling.

The third failure mode is input-domain mismatch. The Bright MLS benchmark is synthetic and clean—ideal conditions. Real-world images, particularly low-light photography, carry noise that the inpainting model was not trained to handle. According to our tests on noisy real-world inputs, tiled inpainting's FID advantage shrinks from the benchmark's full margin to a reduced margin, and upscaling with a denoising pre-processing step becomes competitive. The noise confuses the inpainting model, which tries to synthesize detail where it should be preserving sensor noise structure. If your input is noisy, denoise first, then decide: if the denoised image still has texture you want to enhance, inpaint; if it is already smooth, upscale.

The fourth failure mode is architectural. Our results are based on a latent diffusion base model, which has native conditional inpainting capability. If your base model is a GAN—StyleGAN, for example—tiled inpainting may not work at all, because GANs lack the conditional inpainting mechanism that diffusion models have. In that case, upscaling is not a compromise; it is the only option. Check your base model's architecture before committing to a pipeline.

Finally, the FID improvement is an average, not a guarantee. Variance across runs is ±3.2 FID points, and for roughly a small fraction of the Bright MLS test set, upscaling actually produces lower FID because the inpainting model fails to generate certain textures—typically fine, repetitive patterns like brickwork or woven fabric where the model's synthesis collapses into artifacts. For those cases, upscaling's interpolation is more faithful than the model's hallucination.