TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation
Paper Project Code Poster Video
We address the problem of regressing 3D human pose and shape from a single image, with a focus on 3D accuracy. The current best methods leverage large datasets of 3D pseudo-ground-truth (p-GT) and 2D keypoints, leading to robust performance. With such methods, we observe a paradoxical decline in 3D pose accuracy with increasing 2D accuracy. This is caused by biases in the p-GT and the use of an approximate camera projection model. We quantify the error induced by current camera models and show that fitting 2D keypoints and p-GT accurately causes incorrect 3D poses. Our analysis defines the invalid distances within which minimizing 2D and p-GT losses is detrimental. We use this to formulate a new loss Threshold-Adaptive Loss Scaling (TALS) that penalizes gross 2D and p-GT losses but not smaller ones. With such a loss, there are many 3D poses that could equally explain the 2D evidence. To reduce this ambiguity we need a prior over valid human poses but such priors can introduce unwanted bias. To address this, we exploit a tokenized representation of human pose and reformulate the problem as token prediction. This restricts the estimated poses to the space of valid poses, effectively providing a uniform prior. Extensive experiments on the EMDB and 3DPW datasets show that our reformulated keypoint loss and tokenization allows us to train on in-the-wild data while improving 3D accuracy over the state-of-the-art.
| Author(s): | Sai Kumar Dwivedi and Yu Sun and Priyanka Patel and Yao Feng and Michael J. Black |
| Links: | |
| Book Title: | IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) |
| Pages: | 1323-1333 |
| Year: | 2024 |
| Month: | June |
| BibTeX Type: | Conference Paper (inproceedings) |
| DOI: | 10.1109/CVPR52733.2024.00132 |
| Event Name: | CVPR 2024 |
| Event Place: | Seattle, USA |
| State: | Published |
| URL: | https://tokenhmr.is.tue.mpg.de/ |
| Electronic Archiving: | grant_archive |
BibTeX
@inproceedings{dwivedi_2024_tokenhmr,
title = {{TokenHMR}: Advancing Human Mesh Recovery with a Tokenized Pose Representation},
booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
abstract = {We address the problem of regressing 3D human pose and shape from a single image, with a focus on 3D accuracy. The current best methods leverage large datasets of 3D pseudo-ground-truth (p-GT) and 2D keypoints, leading to robust performance. With such methods, we observe a paradoxical decline in 3D pose accuracy with increasing 2D accuracy. This is caused by biases in the p-GT and the use of an approximate camera projection model. We quantify the error induced by current camera models and show that fitting 2D keypoints and p-GT accurately causes incorrect 3D poses. Our analysis defines the invalid distances within which minimizing 2D and p-GT losses is detrimental. We use this to formulate a new loss Threshold-Adaptive Loss Scaling (TALS) that penalizes gross 2D and p-GT losses but not smaller ones. With such a loss, there are many 3D poses that could equally explain the 2D evidence. To reduce this ambiguity we need a prior over valid human poses but such priors can introduce unwanted bias. To address this, we exploit a tokenized representation of human pose and reformulate the problem as token prediction. This restricts the estimated poses to the space of valid poses, effectively providing a uniform prior. Extensive experiments on the EMDB and 3DPW datasets show that our reformulated keypoint loss and tokenization allows us to train on in-the-wild data while improving 3D accuracy over the state-of-the-art.},
pages = {1323-1333},
month = jun,
year = {2024},
author = {Dwivedi, Sai Kumar and Sun, Yu and Patel, Priyanka and Feng, Yao and Black, Michael J.},
doi = {10.1109/CVPR52733.2024.00132},
url = {https://tokenhmr.is.tue.mpg.de/},
month_numeric = {6}
}