GARDEN: Gravity-Aligned Reconstruction of
Disentangled ENvironments from RGB Images

ECCV 2026

Jiahao Sun^1,2,* Dingkun Wei^1,2,* Zehong Shen^2,† Hongyu Zhou^1,2,* Yujun Shen^2,✉ Liang Li^1,✉

¹Zhejiang University ²Ant Group

^*Work done during internships at Ant Group ^†Project lead ✉Corresponding author

TL;DR GARDEN turns multi-view RGB into a gravity-aligned, simulation-ready scene.

Abstract

Converting multi-view RGB into simulation-ready 3D scenes is hard because reconstruction pipelines produce monolithic geometry—defined up to an arbitrary global rotation and entangling objects with the background—while CAD-retrieval fixes sacrifice scene-specific fidelity. We propose GARDEN, an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization: using gravity as a universal prior, it aligns the scene to a Gravity-View frame, recovers object-centric rigid meshes with accurate 6-DoF poses, and removes duplicate object geometry from the background. The resulting structured hybrid representation is directly simulation-ready while preserving visual realism, improving placement reliability, disentanglement, and efficiency over retrieval-based baselines.

Method

GARDEN casts reconstruction as gravity-driven physically-grounded factorization. From multi-view RGB, it builds a unified Gravity-View (GV) frame to resolve orientation ambiguity, then splits the scene into independent rigid objects (explicit 6-DoF meshes) and a clean background (point cloud or 3DGS)—a structured hybrid representation ready for both physics simulation and photorealistic rendering.

The pipeline has four stages:

Gravity-View alignment. A lightweight module reuses DepthAnything-3 camera tokens to regress the rotation into a gravity-aligned frame, resolving the global orientation ambiguity left by multi-view foundation models.
Target-driven object factorization. From a single 2D box (user- or VLM-provided), SAM-3 and SAM-3D recover a complete amodal mesh, and FoundationPose refines its 6-DoF pose—reduced to translation and yaw thanks to the fixed vertical axis.
Point-conditional background disentanglement. A Transformer-based point classifier removes the duplicated object geometry from the scene while preserving a high-fidelity background.
Unified physics simulation. Object and floor meshes drive rigid-body dynamics while the cleaned background provides a photorealistic backdrop in a shared, gravity-consistent frame.

Results

Following the LiteReality protocol, we evaluate GARDEN on object-centric perceptual quality and holistic scene quality, and additionally assess gravity-estimation accuracy. Unless noted, GARDEN's rendering results are measured after a 10-second MuJoCo simulation, a conservative stress test that penalizes any gravity-inconsistent placement.

Cross-dataset comparison against SAM-3D and DepthAnything-3. Results are shown from post-simulation states.

RMSE and LPIPS: lower is better. SSIM: higher is better. The best result in each column is in bold.

Object-Centric Material Recovery

Method	RMSE ↓	SSIM ↑	LPIPS ↓
MIR	0.2377	0.3981	0.6111
PhotoShape	0.3225	0.2371	0.6558
MIR + AO	0.2156	0.4203	0.5899
Sem&Vis	0.2835	0.3758	0.6362
LiteReality	0.2163	0.4353	0.5854
Ours (3DGS, post-sim)	0.1880	0.4181	0.5035
Ours (point cloud, post-sim)	0.1887	0.4240	0.4444
Ours (3DGS, static)	0.1796	0.4494	0.4182
Ours (point cloud, static)	0.1736	0.4656	0.3680

Holistic Graphics-Ready Scene

Method	RMSE ↓	SSIM ↑	LPIPS ↓
Phone2Proc	0.3604	0.5512	0.7338
ACDC	0.3653	0.5531	0.7364
ACDC + Sem&Vis	0.3226	0.5425	0.6717
ACDC + MIR	0.3046	0.5492	0.6648
LiteReality	0.2664	0.5818	0.6522
Ours (3DGS, post-sim)	0.1670	0.5573	0.4709
Ours (point cloud, post-sim)	0.1616	0.5735	0.4081
Ours (3DGS, static)	0.1593	0.5718	0.4379
Ours (point cloud, static)	0.1511	0.5916	0.3770

Gravity Estimation Accuracy angular error in degrees, lower is better

Method	Hypersim			TartanAir
Method	Mean ↓	P90 ↓	Fail@10 ↓	Mean ↓	P90 ↓	Fail@10 ↓
Plane-RANSAC	24.87	89.68	26.7	19.26	83.22	34.4
Normal clustering	30.98	89.77	33.3	21.33	88.70	37.5
COLMAP's Manhattan	2.62	6.38	0.0	8.37	15.91	12.5
GeoCalib	7.59	4.79	6.7	3.01	6.50	9.4
GeoCalib + RANSAC	1.71	3.79	0.0	2.69	5.91	9.4
Ours GV	1.40	1.90	0.0	1.56	3.48	0.0

Citation

@inproceedings{sun2026garden,
  title     = {GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images},
  author    = {Sun, Jiahao and Wei, Dingkun and Shen, Zehong and Zhou, Hongyu and Shen, Yujun and Li, Liang},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB Images