Neural Capture and Synthesis Conference Paper 2021

Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction

Paper Video
Thumb ticker sm justus thies
Neural Capture and Synthesis, Perceiving Systems
Max Planck Research Group Leader
Thumb  1

We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects. Specifically, we implicitly model a deformation graph via a deep neural network. This neural deformation graph does not rely on any object-specific structure and, thus, can be applied to general non-rigid deformation tracking. Our method globally optimizes this neural graph on a given sequence of depth camera observations of a non-rigidly moving object. Based on explicit viewpoint consistency as well as inter-frame graph and surface consistency constraints, the underlying network is trained in a self-supervised fashion. We additionally optimize for the geometry of the object with an implicit deformable multi-MLP shape representation. Our approach does not assume sequential input data, thus enabling robust tracking of fast motions or even temporally disconnected recordings. Our experiments demonstrate that our Neural Deformation Graphs outperform state-of-the-art non-rigid reconstruction approaches both qualitatively and quantitatively, with 64% improved reconstruction and 62% improved deformation tracking performance.

Author(s): Bozic, Aljaz and Palafox, Pablo and Zollöfer, Michael and Thies, Justus and Dai, Angela and Nießner, Matthias
Links:
Book Title: IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)
Pages: 1450--1459
Year: 2021
Bibtex Type: Conference Paper (inproceedings)
DOI: 10.1109/CVPR46437.2021.00150
Event Name: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)
Event Place: Virtual
State: Published
URL: https://ieeexplore.ieee.org/document/9577883
Electronic Archiving: grant_archive

BibTex

@inproceedings{bozic2021neuraldeformationgraphs,
  title = {Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction},
  booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
  abstract = {We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects. Specifically, we implicitly model a deformation graph via a deep neural network. This neural deformation graph does not rely on any object-specific structure and, thus, can be applied to general non-rigid deformation tracking. Our method globally optimizes this neural graph on a given sequence of depth camera observations of a non-rigidly moving object. Based on explicit viewpoint consistency as well as inter-frame graph and surface consistency constraints, the underlying network is trained in a self-supervised fashion. We additionally optimize for the geometry of the object with an implicit deformable multi-MLP shape representation. Our approach does not assume sequential input data, thus enabling robust tracking of fast motions or even temporally disconnected recordings. Our experiments demonstrate that our Neural Deformation Graphs outperform state-of-the-art non-rigid reconstruction approaches both qualitatively and quantitatively, with 64% improved reconstruction and 62% improved deformation tracking performance.},
  pages = {1450--1459 },
  year = {2021},
  slug = {bozic2021neuraldeformationgraphs},
  author = {Bozic, Aljaz and Palafox, Pablo and Zoll{\"o}fer, Michael and Thies, Justus and Dai, Angela and Nie{\ss}ner, Matthias},
  url = {https://ieeexplore.ieee.org/document/9577883}
}