Diffusion-based Generation, Optimization, and Planning in 3D Scenes

Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu +3 more
2/14/2026

No official code found. Best implementation: scenediffuser/Scene-Diffuser (407 stars, MIT, last commit 2023-06-27).

Abstract

We introduce the SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior work, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion. Such a design alleviates the discrepancies among different modules and the posterior collapse of previous scene-conditioned generative models. We evaluate the SceneDiffuser on various 3D scene understanding tasks, including human pose and motion generation, dexterous grasp generation, path planning for 3D navigation, and motion planning for robot arms. The results show significant improvements compared with previous models, demonstrating the tremendous potential of the SceneDiffuser for the broad community of 3D scene understanding.

View on arXivView PDF

Code Implementations(1)

scenediffuser/Scene-DiffuserCommunity100%
MIT

Official implementation of CVPR23 paper "Diffusion-based Generation, Optimization, and Planning in 3D Scenes"

40729Shell, PythonFeb 2, 20233 years agoMIT
3d-scene-understandingdiffusiongenerative-model

Similar Papers

ViHOI: Human-Object Interaction Synthesis with Visual Priors

83% match2 repos172026

From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models

83% match2024

Compositional Scene Understanding through Inverse Generative Modeling

83% match2025

Compositional Scene Understanding through Inverse Generative Modeling

83% match2026

Physics-based Scene Layout Generation from Human Motion

83% match2024

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

83% match2026

Cite this paper

@article{huang2026diffusionbased,
  title  = {Diffusion-based Generation, Optimization, and Planning in 3D Scenes},
  author = {Siyuan Huang and Zan Wang and Puhao Li and Baoxiong Jia and Tengyu Liu and Yixin Zhu and Wei Liang and Song-Chun Zhu},
  year   = {2026},
  eprint = {2301.06015},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2301.06015},
  journal = {CVPR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.06015.svg)](https://codeofpaper.com/paper/2301.06015)

Discussion