Institute Homepage
Institute Homepage Sign In

Back

Research Overview

Inferring and exploiting contact

Generative Proxemics: A Prior for 3D Social Interaction from Images

BITE -- Dog Shape and Pose from an Image

HOLD -- inferring 3D hand and object shape from video

MOVER -- Reconstructing 3D Scenes and People using Interaction

Datasets for understanding humans and animals

The Poses for Equine Research Dataset (PFERD)

BEAT2 Dataset for Holistic Co-Speech Gesture Generation

ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation

The BioAMASS Dataset

OpenCapBench dataset

Human health and the 3D body

Body Shape Models in Treating Anorexia Nervosa

Customized Bone Plants for Humerus Shaft Fractures

Reconstructing Signing Avatars From Video Using Linguistic Priors

The AI animator

HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles

Gaussian Garments

PuzzleAvatar: Assembling 3D Avatars from Personal Albums

FLARE: Fast Learning of Animatable and Relightable Mesh Avatars

Language, Vision, and World Models

AWOL: Analysis WithOut synthesis using Language

Re-Thinking Inverse Graphics with Large Language Models

TeCH: Text-guided Reconstruction of Clothed Humans

Human pose, shape, and motion capture

WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion

3D Human Pose Estimation via Intuitive Physics

Accurate 3D Body Shape Regression using Metric and Semantic Attributes

BEV

Generating human motion

Generating Human Interaction Motions in Scenes with Text Control

TEMOS: Generating Diverse Human Motions from Text

EMAGE: Full-body Gestures from Audio

TEACH: Temporal Action Compositions for 3D Humans

Robot Perception Group

AirCap: 3D Motion Capture

AirCap: Perception-Based Control

AirCapRL: Aerial Motion Capture Using Deep RL

Data Team

Lab Tours and Public Outreach

Collecting Data - From the Idea to the Publication

Capture Technologies Setup

Completed Projects

Human Pose, Shape and Action

3D Pose from Images

2D Pose from Images

Beyond Motion Capture

Action and Behavior

Body Perception

Body Applications

Pose and Motion Priors

Clothing Models (2011-2015)

Reflectance Filtering

Learning on Manifolds

Markerless Animal Motion Capture

Multi-Camera Capture

2D Pose from Optical Flow

Body Perception

Neural Prosthetics and Decoding

Part-based Body Models

Intrinsic Depth

Lie Bodies

Layers, Time and Segmentation

Understanding Action Recognition (JHMDB)

Intrinsic Video

Intrinsic Images

Action Recognition with Tracking

Neural Control of Grasping

Flowing Puppets

Faces

Deformable Structures

Model-based Anthropometry

Modeling 3D Human Breathing

Optical flow in the LGN

FlowCap

Smooth Loops from Unconstrained Video

PCA Flow

Efficient and Scalable Inference

Motion Blur in Layers

Facade Segmentation

Smooth Metric Learning

Robust PCA

3D Recognition

Object Detection

Perceiving Systems

Human Pose

Kinectposeshapecap
Since humans are often the subject of photographs, detecting them and analyzing their pose is critical for image understanding. The photographic study of human pose and motion dates from the late 1800's with the work of Muybridge and Marey. Our research continues this tradition but with advanced graphics models of the body, new algorithms for pose and shape estimation, machine learning methods, and quantitative analysis of human motion and pose on ground-truth datasets.
 
In the last five years we have made significant progress on automatically estimating 2D human pose from images by leveraging training datasets and machine learning methods.  We also leverage our expertise in optical flow estimation to extend 2D pose estimation over time, resulting in increased accuracy.
 
Beyond 2D pose, we are pushing the notion of "motion capture" in new directions.  The goal is always to leverage what we know about bodies to get more from less - more accuracy and more shape detail from a small number of simple sensors.  From 3D mocap markers, we recover detailed shape, pose, and soft tissue motion.  Using a single RGB-D sensor and a parametric model of the human body, we are able to estimate human body shapes and poses from complex sequences of unconstrained motion.  

Our work on human pose includes:
  • 2D pose estimation from images.
  • 2D pose from video and optical flow.
  • 3D pose form multi-camera data.
  • 3D pose from images and monocular video.
  • 3D pose from RGB-D sequences.
  • 3D pose from IUMs and other non-vision sensors.
  • Marker-based and markerless motion capture.
  • Learning pose priors.
  • Tracking people interacting with objects.
  • Datasets for quantitative evaluation.
  • Human activity recognition.

Our current work is pushing the state of the art in monocular pose and motion capture to automatically go from 2D images or monocular video to 3D pose and shape of the human body.  We are also expanding our research from tracking humans to tracking animals of many kinds.