Perceiving Systems Empirical Inference Conference Paper 2024

Ghost on the Shell: An Expressive Representation of General 3D Shapes

Thumb ticker sm zhenliu
Perceiving Systems
Thumb ticker sm photo yao
Perceiving Systems
  • Guest Scientist
Thumb ticker sm 84b79ecc0b10a9579306eaa7c8037a
Perceiving Systems
  • Guest Scientist
Thumb ticker sm weiyang photo
Empirical Inference
  • Postdoctoral Researcher
Thumb ticker sm headshot2021
Perceiving Systems
Director
Thumb ticker sm l1170153
Empirical Inference
  • Director
Thumb xxl image

The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parameterize open surfaces by defining a manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications: differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.

Author(s): Zhen Liu and Yao Feng and Yuliang Xiu and Weiyang Liu and Liam Paull and Michael J. Black and Bernhard Schölkopf
Book Title: Proceedings of the Twelfth International Conference on Learning Representations (ICLR)
Year: 2024
Month: May
Bibtex Type: Conference Paper (inproceedings)
Event Name: The Twelfth International Conference on Learning Representations (ICLR)
Event Place: Vienna, Austria
State: Published
Electronic Archiving: grant_archive
Links:

BibTex

@inproceedings{gshell,
  title = {Ghost on the Shell: An Expressive Representation of General {3D} Shapes},
  booktitle = {Proceedings of the Twelfth International Conference on Learning Representations (ICLR)},
  abstract = {The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parameterize open surfaces by defining a manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications: differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.},
  month = may,
  year = {2024},
  slug = {gshell},
  author = {Liu, Zhen and Feng, Yao and Xiu, Yuliang and Liu, Weiyang and Paull, Liam and Black, Michael J. and Sch{\"o}lkopf, Bernhard},
  month_numeric = {5}
}