Human Pose, Shape and Action
3D Pose from Images
2D Pose from Images
Beyond Motion Capture
Action and Behavior
Body Perception
Body Applications
Pose and Motion Priors
Clothing Models (2011-2015)
Reflectance Filtering
Learning on Manifolds
Markerless Animal Motion Capture
Multi-Camera Capture
2D Pose from Optical Flow
Body Perception
Neural Prosthetics and Decoding
Part-based Body Models
Intrinsic Depth
Lie Bodies
Layers, Time and Segmentation
Understanding Action Recognition (JHMDB)
Intrinsic Video
Intrinsic Images
Action Recognition with Tracking
Neural Control of Grasping
Flowing Puppets
Faces
Deformable Structures
Model-based Anthropometry
Modeling 3D Human Breathing
Optical flow in the LGN
FlowCap
Smooth Loops from Unconstrained Video
PCA Flow
Efficient and Scalable Inference
Motion Blur in Layers
Facade Segmentation
Smooth Metric Learning
Robust PCA
3D Recognition
Object Detection
Markerless Animal Motion Capture
Experiments in motor neurophysiology often involve animals performing repeated actions. Stereotyped and practiced actions facilitate data analysis by allowing the experimenter to average neural firing activity across multiple trials. Does such activity, however, reflect what happens during natural, unconstrained, and spontaneous movement? Do models of neural activity developed in such settings translate to intracortical brain-machine interfaces (iBMIs) where humans need to control devices in a dynamically changing context?
To answer these questions we must record natural behavior and the corresponding neural activity [
]. For the latter, our collaborators at Stanford have developed an implantable device that enables the wireless recording of spiking activity from many neurons at once (see Figure). For the former, traditional marker-based motion capture techniques are impractical for full-body animal tracking. Consequently we exploit our work on human tracking to enable markerless articulated animal tracking.
Our system transmits neural activity from an intracortical array using a head-mounted device and records behavior using multi-camera markerless motion capture. We demonstrate this with the first recordings from motor cortex of rhesus monkeys walking quadrupedally on a treadmill. We find that multi-unit threshold-crossings encode the phase of walking and that the average firing rate covaries with the speed of individual steps.
Freely-moving animal models may allow neuroscientists to examine a wider range of behaviors and can provide a flexible experimental paradigm for examining the neural mechanisms that underlie movement generation across behaviors and environments. For iBMIs, freely-moving animal models have the potential to aid prosthetic design by enabling the study of how neural encoding changes with posture, environment, and other real-world context changes. Understanding behavior in more naturalistic settings is essential for overall progress of basic motor neuroscience and for the successful translation of BMIs to people with paralysis.
Members
Publications