AirCap: 3D Motion Capture

The goal of AirCap is markerless, unconstrained, human and animal motion capture (mocap) outdoors. To that end, we have developed a flying mocap system using a team of aerial vehicles (MAVs) with only on-board, monocular RGB cameras. In AirCap, mocap involves two phases: i) online data acquisition, and ii) offline pose and shape estimation.

During online data acquisition, the micro air vehicles (MAVs) detect and track the 3D position of a subject []. To do so, they perform on-board detection using a deep neural network (DNN). DNNs often fail to detect small people, which are typical in scenarios with aerial robots. By cooperatively tracking the person our system actively selects the relevant region of interest (ROI) in the images from each MAV. Then cropped high-resolution regions around the person are passed to the DNNs.

Then, human pose and shape are estimated offline using the RGB images and the MAV's self-localization (the camera extrinsics). Recent 3D human pose and shape regression methods produce noisy estimate of human pose. Our approach [] exploits multiple noisy 2D body joint detectors and noisy camera pose information. We then optimize for body shape, body pose, and camera extrinsics by fitting the SMPL body model to the 2D observations. This approach uses a strong body model to take low-level uncertainty into account and results in the first fully autonomous flying mocap system.

Offline mocap, does not enable active positioning of the MAVs to maximize the mocap accuracy. To address this, we introduce a deep reinforcement learning (RL) based multi-robot formation controller for MAVs. We formulate this problem as a sequential decision making task and solve it using an RL method [].

To enable fully on-board, online, mocap, we are developing a novel, distributed, multi-view fusion network for 3D pose and shape estimation of humans using uncalibrated moving cameras.

Reseources and Source Code

Source Code and Resources

News : Nodes and packages specific to our submission to ICRA 2019 added on our Github project page.
Github project page with all sources: AirCap Github Profile

List of required hardware

Flying platform capable of carrying 2 kg of payload or more
OpenPilot Revolution flight controllers
HD Cameras
NVIDIA Jetson TX1 embedded GPU
On board computer (PC) - Intel Core I7 CPU - Ubuntu 16.04

List of required 3rd party Open Source Packages

ROS-kinetic

Project Source Code

Members

Perceiving Systems

Aamir Ahmad

Tenure-track Professor, University of Stuttgart, Research Group Leader (mpi-is)

Perceiving Systems

Eric Price

Research Scientist

Perceiving Systems

Nitin Saini

Guest Scientist

Perceiving Systems

Guilherme Lawless

Perceiving Systems

Roman Ludwig

Student Assistant

Perceiving Systems

Igor Martinovic

Affiliated Researcher

Perceiving Systems

Michael Black

Emeritus / Acting Director

Perceiving Systems

Aamir Ahmad

Tenure-track Professor, University of Stuttgart, Research Group Leader (mpi-is)

Perceiving Systems

Yilin Ji

Perceiving Systems

Elia Bonetto

Guest Scientist

Publications

Perceiving Systems Conference Paper AirCap – Aerial Outdoor Motion Capture Ahmad, A., Price, E., Tallamraju, R., Saini, N., Lawless, G., Ludwig, R., Martinovic, I., Bülthoff, H. H., Black, M. J. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), Workshop on Aerial Swarms, November 2019

Abstract ›

This paper presents an overview of the Grassroots project Aerial Outdoor Motion Capture (AirCap) running at the Max Planck Institute for Intelligent Systems. AirCap's goal is to achieve markerless, unconstrained, human motion capture (mocap) in unknown and unstructured outdoor environments. To that end, we have developed an autonomous flying motion capture system using a team of aerial vehicles (MAVs) with only on-board, monocular RGB cameras. We have conducted several real robot experiments involving up to 3 aerial vehicles autonomously tracking and following a person in several challenging scenarios using our approach of active cooperative perception developed in AirCap. Using the images captured by these robots during the experiments, we have demonstrated a successful offline body pose and shape estimation with sufficiently high accuracy. Overall, we have demonstrated the first fully autonomous flying motion capture system involving multiple robots for outdoor scenarios.

Talk slides BibTeX

Perceiving Systems Conference Paper Markerless Outdoor Human Motion Capture Using Multiple Autonomous Micro Aerial Vehicles Saini, N., Price, E., Tallamraju, R., Enficiaud, R., Ludwig, R., Martinović, I., Ahmad, A., Black, M. Proceedings 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 823-832, IEEE, International Conference on Computer Vision (ICCV), October 2019 (Published)

Abstract ›

Capturing human motion in natural scenarios means moving motion capture out of the lab and into the wild. Typical approaches rely on fixed, calibrated, cameras and reflective markers on the body, significantly limiting the motions that can be captured. To make motion capture truly unconstrained, we describe the first fully autonomous outdoor capture system based on flying vehicles. We use multiple micro-aerial-vehicles(MAVs), each equipped with a monocular RGB camera, an IMU, and a GPS receiver module. These detect the person, optimize their position, and localize themselves approximately. We then develop a markerless motion capture method that is suitable for this challenging scenario with a distant subject, viewed from above, with approximately calibrated and moving cameras. We combine multiple state-of-the-art 2D joint detectors with a 3D human body model and a powerful prior on human pose. We jointly optimize for 3D body pose and camera pose to robustly fit the 2D measurements. To our knowledge, this is the first successful demonstration of outdoor, full-body, markerless motion capture from autonomous flying vehicles.

Code Data Video Paper Manuscript DOI BibTeX

Perceiving Systems Article Deep Neural Network-based Cooperative Visual Tracking through Multiple Micro Aerial Vehicles Price, E., Lawless, G., Ludwig, R., Martinovic, I., Buelthoff, H. H., Black, M. J., Ahmad, A. IEEE Robotics and Automation Letters, Robotics and Automation Letters, 3(4):3193-3200, IEEE, October 2018, Also accepted and presented in the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (Published)

Abstract ›

Multi-camera tracking of humans and animals in outdoor environments is a relevant and challenging problem. Our approach to it involves a team of cooperating micro aerial vehicles (MAVs) with on-board cameras only. DNNs often fail at objects with small scale or far away from the camera, which are typical characteristics of a scenario with aerial robots. Thus, the core problem addressed in this paper is how to achieve on-board, online, continuous and accurate vision-based detections using DNNs for visual person tracking through MAVs. Our solution leverages cooperation among multiple MAVs and active selection of most informative regions of image. We demonstrate the efficiency of our approach through simulations with up to 16 robots and real robot experiments involving two aerial robots tracking a person, while maintaining an active perception-driven formation. ROS-based source code is provided for the benefit of the community.

Published Version DOI URL BibTeX