Perception rescues 3D reconstruction where generative priors fail

Seeing Before Generating

Object Perception Enhances Single-View 3D Reconstruction

Y Huynh1  ·  Duc Thanh Nguyen2  ·  Mohamed Abdelrazek1

1Applied Artificial Intelligence Initiative, Deakin University   2School of Information Technology, Deakin University

arXiv Code (Soon)

Abstract

The relationship between object perception and reconstruction is well established in human vision, yet remains underexplored in computer vision. In this paper, we demonstrate that learnt object perception can significantly enhance 3D reconstruction. Focusing on the challenging task of single-view 3D object reconstruction, we propose a method that leverages perceptual signals extracted from pretrained perception models — capturing semantic and geometric information — to drive the reconstruction of an object from its single image. Our approach is model-agnostic and can be integrated into various reconstruction methods in a plug-and-play manner. Experiments with two state-of-the-art single-view 3D reconstruction pipelines on a benchmark dataset show consistent and substantial improvements, validating the effectiveness of incorporating perception into generation. We provide in-depth analysis of various aspects of our method and its application. To support future research, we will release our code and pretrained models.

How Perception Guides Generation

Multi-view diffusion models hallucinate geometry from a learned prior. When that prior is wrong, the reconstruction fails. We attach a lightweight Generation-Perception Alignment module to a frozen diffusion backbone — a Feature Aggregation network and modality-specific Perception Mappers that predict what pre-trained perception models “see” in the object, then use the mismatch to steer the denoising latents at inference.

PGR-3D framework. (a) Training: a frozen perception model and frozen U-Net feed a trainable Generation-Perception Alignment module (feature aggregation g + perception mapping f) that aligns U-Net features to perceptual features. (b) Inference: the perception loss gradient guides the latents, producing view-consistent RGB and normal multi-view outputs for 3D reconstruction.
1

Perception Extraction

Pre-trained models capture semantic knowledge (a VLM + LLM caption, embedded with CLIP) and geometric knowledge (Depth‑Anything V2) from the single input image.

2

Feature Alignment

Forward hooks on the frozen U-Net decoder feed a Feature Aggregation network. Modality-specific Perception Mappers project these features toward the perceptual targets — the only trained parameters in the system.

3

Guided Sampling

At inference, gradients from the perception loss flow back through the frozen network to the noisy latents $x_t$, steering DDIM sampling toward perceptually consistent multi-view outputs.

What Does Perception See?

Before guiding generation, our perception modules extract two complementary views of the object — one of what it is, one of how it is shaped.

🗣️ Semantic — what the object is

A vision-language model describes the object in natural language; the caption is embedded into a single view-invariant semantic vector that anchors its identity.

alarm clock input

“a vintage-style alarm clock… round, rigid body with a metallic finish… a small loop handle… and a visible winding key on the side.”

fedora hat input

“a classic fedora hat… light brown straw texture and a white band… a rounded dome and a flat brim that extends outward.”

shampoo tube input

“a tube of Moroccan argan-oil shampoo… cylindrical, slightly tapered shape with a rigid, smooth surface… the cap at the bottom.”

input image → VLM caption → text encoder → semantic vector

📐 Geometric — how the object is shaped

Depth-Anything V2 refinenet3 features. Four channels fire on distinct structural concepts — far richer than a single depth map:

massch 215 contourch 242 edgesch 37 orientationch 146
DA V2 refinenet3 channels for the alarm clock: mass, contour, edges, orientation
DA V2 refinenet3 channels for the fedora hat: mass, contour, edges, orientation
DA V2 refinenet3 channels for the shampoo tube: mass, contour, edges, orientation
input image → DA V2 → refinenet3 features → depth target

During training, the Perception Mappers learn to predict these targets from the frozen U-Net's internal features. At inference, the mismatch between prediction and target becomes the guidance signal that steers denoising.

Watch Perception Guide the Denoising

The same input, denoised with and without perception, side by side across all six views. At every DDIM step the Perception Mappers compare what the U-Net is generating against what perception expects and nudge the latents toward agreement. Each frame is the model's x̂0 prediction — all 40 steps, in sync.

Fedora hat, all 40 denoising steps. Top: the baseline grows a tall, pointed crown in the side views. Bottom: perception guidance settles the proper low fedora shape and flat brim (−33% CD).
Stacked cups, all 40 denoising steps. Top: the baseline duplicates the cups into multiple phantom stacks in the side views. Bottom: perception guidance converges to a single, correctly-ordered stack (−94% CD).
Stacking toy, all 40 steps — an already-easy case. Both rows converge to a consistent figure; perception guidance keeps the shape without harming it (baseline was already near-perfect).

Era3D six-view x̂0 prediction at every one of the 40 DDIM steps; combined guidance = depth + caption Perception Mappers. Rows are frame-synchronized.

Results

GSO-30 benchmark. Chamfer Distance (CD, ×10−3) and volumetric IoU. The same depth + caption recipe improves both backbones.

Setting CD ↓ IoU ↑ PSNR ↑ ΔCD
Wonder3D21.210.53021.79—
+ Depth20.180.53421.86−4.3%
+ Depth + Caption Ours19.300.53821.86−9.0%
Era3D20.780.49921.53—
+ Depth14.460.56121.80−30.4%
+ Depth + Caption Ours13.990.55822.05−32.7%

Era3D comparison uses a matched, guidance-free pipeline baseline. Full metric tables (SSIM, LPIPS, ablations) are in the paper.

Qualitative Results

Era3D six-view RGB and normal-map outputs under combined guidance.
Era3D six-view outputs. Combined guidance produces clean, view-consistent RGB and surface-normal maps at 512×512.
Show full 30-object Wonder3D grid
Combined-guidance reconstructions across all 30 GSO objects.
All 30 GSO objects under combined guidance.

BibTeX

@article{huynh2026seeing,
  title   = {Seeing Before Generating: Object Perception
             Enhances Single-View 3D Reconstruction},
  author  = {Huynh, Y and Nguyen, Duc Thanh and Abdelrazek, Mohamed},
  journal = {arXiv preprint},
  year    = {2026}
}