Human Pose, Shape and Action
3D Pose from Images
2D Pose from Images
Beyond Motion Capture
Action and Behavior
Body Perception
Body Applications
Pose and Motion Priors
Clothing Models (2011-2015)
Reflectance Filtering
Learning on Manifolds
Markerless Animal Motion Capture
Multi-Camera Capture
2D Pose from Optical Flow
Body Perception
Neural Prosthetics and Decoding
Part-based Body Models
Intrinsic Depth
Lie Bodies
Layers, Time and Segmentation
Understanding Action Recognition (JHMDB)
Intrinsic Video
Intrinsic Images
Action Recognition with Tracking
Neural Control of Grasping
Flowing Puppets
Faces
Deformable Structures
Model-based Anthropometry
Modeling 3D Human Breathing
Optical flow in the LGN
FlowCap
Smooth Loops from Unconstrained Video
PCA Flow
Efficient and Scalable Inference
Motion Blur in Layers
Facade Segmentation
Smooth Metric Learning
Robust PCA
3D Recognition
Object Detection
Intrinsic Images
The problem of decomposing an image into its different intrinsic layers, such as shading, reflectance, and shape components, is one of the fundamental problems of computer vision. We believe that many computer vision tasks will benefit from having access to such separations. As an example, object detection should benefit from factoring out shadow and lighting effects, or image segmentation of materials should be improved by working on reflectance properties alone. In this project we aim to both decompose images into their intrinsic layers, and also show that this results to empirical improvements on different tasks.
Our approach is to model the physical process of light interacting with materials. We model material properties, shape, and lighting conditions as separate sources that result in the observed pixel color and brightness. In "Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance" we extend the Retinex theory by including models of material reflectances. Our conditional random field model encourages solutions in which the image is explained by a sparse set of reflectance types. This corresponds to the intuition that there are only few materials in an image. In the figure we show some example outputs of this model
Intrinsic image decomposition problems arise in many different contexts and we study several different variants that arise when some scene modalities may be known. For example when using multiple depth cameras for full body scanning, we are provided with a good 3D estimate of the scene, or given a video we know that light varies smoothly. Our approach is to model the physical process using techniques from computer graphics together with probabilistic inference to reason about the unknown sources that constituted the scene.
Members
Publications