Back

Perceiving Systems Members Publications Website

TeCH: Text-guided Reconstruction of Clothed Humans

TeCH [File Icon] takes an input image and reconstructs a lifelike 3D clothed human. It automatically generates text descriptions using visual question answering that guide the reconstruction using a personalized text-to-Image diffusion model. Multi-view supervision is established through Score Distillation Sampling.

Members

Perceiving Systems
  • Guest Scientist
Perceiving Systems
  • Guest Scientist
Neural Capture and Synthesis, Perceiving Systems
Max Planck Research Group Leader

Publications

Perceiving Systems Conference Paper TeCH: Text-guided Reconstruction of Lifelike Clothed Humans Huang, Y., Yi, H., Xiu, Y., Liao, T., Tang, J., Cai, D., Thies, J. In International Conference on 3D Vision (3DV 2024), 3DV, March 2024 (Published)
Despite recent research advancements in reconstructing clothed humans from a single image, accurately restoring the "unseen regions" with high-level details remains an unsolved challenge that lacks attention. Existing methods often generate overly smooth back-side surfaces with a blurry texture. But how to effectively capture all visual attributes of an individual from a single image, which are sufficient to reconstruct unseen areas (e.g., the back view)? Motivated by the power of foundation models, TeCH reconstructs the 3D human by leveraging 1) descriptive text prompts (e.g., garments, colors, hairstyles) which are automatically generated via a garment parsing model and Visual Question Answering (VQA), 2) a personalized fine-tuned Text-to-Image diffusion model (T2I) which learns the "indescribable" appearance. To represent high-resolution 3D clothed humans at an affordable cost, we propose a hybrid 3D representation based on DMTet, which consists of an explicit body shape grid and an implicit distance field. Guided by the descriptive prompts + personalized T2I diffusion model, the geometry and texture of the 3D humans are optimized through multi-view Score Distillation Sampling (SDS) and reconstruction losses based on the original observation. TeCH produces high-fidelity 3D clothed humans with consistent & delicate texture, and detailed full-body geometry. Quantitative and qualitative experiments demonstrate that TeCH outperforms the state-of-the-art methods in terms of reconstruction accuracy and rendering quality.
Code Home Video arXiv URL BibTeX