Neural Shape Modeling of 3D Clothed Humans

@phdthesis{Ma:Thesis:2023,
  title = {Neural Shape Modeling of {3D} Clothed Humans},
  abstract = {Parametric models for 3D human bodies play a crucial role in the synthesis
  and analysis of humans in visual computing. While current models
  effectively capture body pose and shape variations, a significant aspect has
  been overlooked – clothing. Existing 3D human models mostly produce
  a minimally-clothed body geometry, limiting their ability to represent the
  complexity of dressed people in real-world data sources.
  
  The challenge lies in the unique characteristics of garments, which make
  modeling clothed humans particularly difficult. Clothing exhibits diverse
  topologies, and as the body moves, it introduces wrinkles at various spatial
  scales. Moreover, pose-dependent clothing deformations are non-rigid and
  non-linear, exceeding the capabilities of classical body models constructed
  with fixed-topology surface meshes and linear approximations of pose-aware
  shape deformations.
  
  This thesis addresses these challenges by innovating in two key areas:
  the 3D shape representation and deformation modeling techniques. We
  demonstrate that, the seemingly old-fashioned shape representation, point
  clouds – when equipped with deep learning and neural fields – can be a
  powerful tool for modeling clothed characters.
  
  Specifically, the thesis begins by introducing a large-scale dataset of dynamic
  3D humans in various clothing, which serves as a foundation for
  training the models presented in this work. The first model we present is
  CAPE: a neural generative model for 3D clothed human meshes. Here, a
  clothed body is straightforwardly obtained by applying per-vetex offsets to
  a pre-defined, unclothed body template mesh. Sampling from the CAPE
  model generates plausibly-looking digital humans wearing common garments,
  but the fixed-topology mesh representation limits its applicability to
  more complex garment types.
  
  To address this limitation, we present a series of point-based clothed
  human models: SCALE, PoP and SkiRT. The SCALE model represents a
  clothed human using a collection of points organized into local patches.
  The patches can freely move and deform to represent garments of diverse
  topologies, unlocking the generalization to more challenging outfits such
  as dresses and jackets. Unlike traditional approaches based on physics
  simulations, SCALE learns pose-dependent cloth deformations from data
  with minimal manual intervention.
  
  To further improve the geometric quality, the PoP model eliminates the
  concept of patches and instead learns a continuous neural deformation
  field from the body surface. Densely querying this field results in a highresolution
  point cloud of a dressed human, showcasing intricate clothing
  wrinkles. PoP can generalize across multiple subjects and outfits, and can
  even bring a single, static scan into animation.
  
  Finally, we tackle a long-standing challenge in learning-based digital
  human modeling: loose garments, in particular skirts and dresses. Building
  upon PoP, the SkiRT pipeline further learns a shape “template” and neural
  field of linear-blend-skinning weights for clothed bodies, improving the
  models’ robustness for loose garments of varied topology.
  
  Our point-based human models are “interplicit”: the output point clouds
  capture surfaces explicitly at discrete points but implicitly in between. The
  explicit points are fast, topologically flexible, and are compatible with existing
  graphics tools, while the implicit neural deformation field contributes to
  high-quality geometry. This thesis primarily demonstrates these advantages
  in the context of clothed human shape modeling; future work can apply
  our representation and techniques to general 3D deformable shapes and
  neural rendering.},
  month = oct,
  year = {2023},
  author = {Ma, Qianli},
  doi = {https://doi.org/10.3929/ethz-b-000652034},
  month_numeric = {10}
}