Human Pose, Shape and Action
3D Pose from Images
2D Pose from Images
Beyond Motion Capture
Action and Behavior
Body Perception
Body Applications
Pose and Motion Priors
Clothing Models (2011-2015)
Reflectance Filtering
Learning on Manifolds
Markerless Animal Motion Capture
Multi-Camera Capture
2D Pose from Optical Flow
Body Perception
Neural Prosthetics and Decoding
Part-based Body Models
Intrinsic Depth
Lie Bodies
Layers, Time and Segmentation
Understanding Action Recognition (JHMDB)
Intrinsic Video
Intrinsic Images
Action Recognition with Tracking
Neural Control of Grasping
Flowing Puppets
Faces
Deformable Structures
Model-based Anthropometry
Modeling 3D Human Breathing
Optical flow in the LGN
FlowCap
Smooth Loops from Unconstrained Video
PCA Flow
Efficient and Scalable Inference
Motion Blur in Layers
Facade Segmentation
Smooth Metric Learning
Robust PCA
3D Recognition
Object Detection
Neural Rendering
] (top) generates realistic face images that are controlled by FLAME parameters. SPICE [] (middle) learns to repose an image of a person without any paired training data by exploiting information about 3D bodies. SMPLpix [] (bottom) generates realistic images of people from a sparse set of colored SMPL vertices.Conventional graphics pipelines take a 3D model like SMPL, apply texture and material properties, light it, and render it as an image. Without expensive artist involvement, this results in unrealistic images that fall into the "uncanny valley". To address this, we develop neural rendering methods that keep the 3D body model but replace the rendering pipeline with neural networks. This approach keeps the flexibility of parametric models while producing realistic looking images without artist intervention.
GIF [] generates realistic images of faces, by conditioning StyleGAN2 on the FLAME face model []. Given FLAME parameters for shape, pose, expressions, plus parameters for appearance, lighting, and an additional style vector, GIF outputs photo-realistic face images.
To generate images of people with realistic hair and clothing, we train SMPLpix [] to transform a sparse set of 3D mesh vertices and their RGB values into photorealistic images. The 3D mesh vertices are controllable with the pose and shape parameters of SMPL.
SPICE [] takes a different approach and synthesizes an image of a person in a novel pose given a source image of the person and a target pose. In contrast to typical approaches that require paired training data, SPICE uses only unpaired data. This is enabled by a novel cycle-GAN training method that exploits information about the 3D SMPL body.
The combination of parametric 3D models with neural rendering enables realistic human rendering with intuitive animation controls.
Members
Publications