AirCap: Perception-Based Control

Flying mocap systems use aerial robots with on-board cameras that can localize and navigate autonomously. These robots must detect, track and follow the subject (human or animal) in real time. A key component of such systems is motion planning and control of multiple robots that ensures optimal perception of the subject while obeying other constraints, e.g., inter-robot and static obstacle collision avoidance.

We address this with decentralized convex model-predictive control (MPC) []. The micro air vehicles (MAVs) actively compute local motion plans producing optimal view-point configurations, which minimize uncertainty in the tracked estimate. We achieve this by decoupling the goal of active tracking into a quadratic objective and non-convex constraints corresponding to angular configurations of the MAVs w.r.t. the person. We derive this decoupling using Gaussian observation model assumptions within the cooperative tracking algorithm [] []. We preserve convexity in optimization by embedding all the non-convex constraints, including those for dynamic obstacle avoidance, as external control inputs in the MPC dynamics []. We evaluate this with 3 MAVs in challenging scenarios.

Tracking animals is difficult with multi-rotor vehicles due to their noise and short flight time. Consequently, we developed a novel framework for modeling, simulation and control of helium-based blimps []. This addresses several unique properties of blimps, such as, i) dynamic deformation in response to aerodynamic and control forces, ii) susceptibility to wind and turbulence at low airspeed, iii) variability in airship designs regarding placement, direction and vectoring of thrusters and control surfaces. Based on simulated wind and deformations due to changes in ambient pressure and temperature as well as helium leakage, we predict substantial effects on controllability, verified in real world flight experiments. We are currently developing formation control strategies for blimps, where we address their non-linear, non-holonomic and time-delayed dynamics.

Members

Perceiving Systems

Aamir Ahmad

Tenure-track Professor, University of Stuttgart, Research Group Leader (mpi-is)

Perceiving Systems

Rahul Tallamraju

Doctoral Researcher

Perceiving Systems

Eric Price

Guest Scientist

Perceiving Systems

Roman Ludwig

Student Assistant

Perceiving Systems

Michael Black

Director

Perceiving Systems

Aamir Ahmad

Tenure-track Professor, University of Stuttgart, Research Group Leader (mpi-is)

Perceiving Systems

Nowfal Manakkaparambil Ali

Perceiving Systems

Halil Acet

Perceiving Systems

Yu-Tang Liu

Guest Scientist

Perceiving Systems

Igor Martinovic

Affiliated Researcher

Perceiving Systems

Elia Bonetto

Guest Scientist

Publications

Perceiving Systems Article AirCapRL: Autonomous Aerial Human Motion Capture Using Deep Reinforcement Learning Tallamraju, R., Saini, N., Bonetto, E., Pabst, M., Liu, Y. T., Black, M., Ahmad, A. IEEE Robotics and Automation Letters, 5(4):6678-6685, IEEE, October 2020, Also accepted and presented in the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (Published)

Abstract ›

In this letter, we introduce a deep reinforcement learning (DRL) based multi-robot formation controller for the task of autonomous aerial human motion capture (MoCap). We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose, and shape of a single moving person using multiple micro aerial vehicles. State-of-the-art solutions to this problem are based on classical control methods, which depend on hand-crafted system, and observation models. Such models are difficult to derive, and generalize across different systems. Moreover, the non-linearities, and non-convexities of these models lead to sub-optimal controls. In our work, we formulate this problem as a sequential decision making task to achieve the vision-based motion capture objectives, and solve it using a deep neural network-based RL method. We leverage proximal policy optimization (PPO) to train a stochastic decentralized control policy for formation control. The neural network is trained in a parallelized setup in synthetic environments. We performed extensive simulation experiments to validate our approach. Finally, real-robot experiments demonstrate that our policies generalize to real world conditions.

DOI URL BibTeX

Perceiving Systems Conference Paper AirCap – Aerial Outdoor Motion Capture Ahmad, A., Price, E., Tallamraju, R., Saini, N., Lawless, G., Ludwig, R., Martinovic, I., Bülthoff, H. H., Black, M. J. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), Workshop on Aerial Swarms, November 2019

Abstract ›

This paper presents an overview of the Grassroots project Aerial Outdoor Motion Capture (AirCap) running at the Max Planck Institute for Intelligent Systems. AirCap's goal is to achieve markerless, unconstrained, human motion capture (mocap) in unknown and unstructured outdoor environments. To that end, we have developed an autonomous flying motion capture system using a team of aerial vehicles (MAVs) with only on-board, monocular RGB cameras. We have conducted several real robot experiments involving up to 3 aerial vehicles autonomously tracking and following a person in several challenging scenarios using our approach of active cooperative perception developed in AirCap. Using the images captured by these robots during the experiments, we have demonstrated a successful offline body pose and shape estimation with sufficiently high accuracy. Overall, we have demonstrated the first fully autonomous flying motion capture system involving multiple robots for outdoor scenarios.

Talk slides BibTeX

Perceiving Systems Article Active Perception based Formation Control for Multiple Aerial Vehicles Tallamraju, R., Price, E., Ludwig, R., Karlapalem, K., Bülthoff, H. H., Black, M. J., Ahmad, A. IEEE Robotics and Automation Letters, Robotics and Automation Letters, 4(4):4491-4498, IEEE, October 2019

Abstract ›

We present a novel robotic front-end for autonomous aerial motion-capture (mocap) in outdoor environments. In previous work, we presented an approach for cooperative detection and tracking (CDT) of a subject using multiple micro-aerial vehicles (MAVs). However, it did not ensure optimal view-point configurations of the MAVs to minimize the uncertainty in the person's cooperatively tracked 3D position estimate. In this article, we introduce an active approach for CDT. In contrast to cooperatively tracking only the 3D positions of the person, the MAVs can actively compute optimal local motion plans, resulting in optimal view-point configurations, which minimize the uncertainty in the tracked estimate. We achieve this by decoupling the goal of active tracking into a quadratic objective and non-convex constraints corresponding to angular configurations of the MAVs w.r.t. the person. We derive this decoupling using Gaussian observation model assumptions within the CDT algorithm. We preserve convexity in optimization by embedding all the non-convex constraints, including those for dynamic obstacle avoidance, as external control inputs in the MPC dynamics. Multiple real robot experiments and comparisons involving 3 MAVs in several challenging scenarios are presented.

pdf DOI BibTeX

Perceiving Systems Conference Paper Decentralized MPC based Obstacle Avoidance for Multi-Robot Target Tracking Scenarios Tallamraju, R., Rajappa, S., Black, M. J., Karlapalem, K., Ahmad, A. 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 1-8, IEEE, August 2018 (Published)

Abstract ›

In this work, we consider the problem of decentralized multi-robot target tracking and obstacle avoidance in dynamic environments. Each robot executes a local motion planning algorithm which is based on model predictive control (MPC). The planner is designed as a quadratic program, subject to constraints on robot dynamics and obstacle avoidance. Repulsive potential field functions are employed to avoid obstacles. The novelty of our approach lies in embedding these non-linear potential field functions as constraints within a convex optimization framework. Our method convexifies nonconvex constraints and dependencies, by replacing them as pre-computed external input forces in robot dynamics. The proposed algorithm additionally incorporates different methods to avoid field local minima problems associated with using potential field functions in planning. The motion planner does not enforce predefined trajectories or any formation geometry on the robots and is a comprehensive solution for cooperative obstacle avoidance in the context of multi-robot target tracking. We perform simulation studies for different scenarios to showcase the convergence and efficacy of the proposed algorithm.

Published Version DOI URL BibTeX

Perceiving Systems Article An Online Scalable Approach to Unified Multirobot Cooperative Localization and Object Tracking Ahmad, A., Lawless, G., Lima, P. IEEE Transactions on Robotics (T-RO), 33:1184 - 1199, October 2017 (Published)

Abstract ›

In this article we present a unified approach for multi-robot cooperative simultaneous localization and object tracking based on particle filters. Our approach is scalable with respect to the number of robots in the team. We introduce a method that reduces, from an exponential to a linear growth, the space and computation time requirements with respect to the number of robots in order to maintain a given level of accuracy in the full state estimation. Our method requires no increase in the number of particles with respect to the number of robots. However, in our method each particle represents a full state hypothesis, leading to the linear dependency on the number of robots of both space and time complexity. The derivation of the algorithm implementing our approach from a standard particle filter algorithm and its complexity analysis are presented. Through an extensive set of simulation experiments on a large number of randomized datasets, we demonstrate the correctness and efficacy of our approach. Through real robot experiments on a standardized open dataset of a team of four soccer playing robots tracking a ball, we evaluate our method's estimation accuracy with respect to the ground truth values. Through comparisons with other methods based on i) nonlinear least squares minimization and ii) joint extended Kalman filter, we further highlight our method's advantages. Finally, we also present a robustness test for our approach by evaluating it under scenarios of communication and vision failure in teammate robots.

Published Version DOI URL BibTeX

Perceiving Systems Article Moving-horizon Nonlinear Least Squares-based Multirobot Cooperative Perception Ahmad, A., Bülthoff, H. Robotics and Autonomous Systems, 83:275-286, 2016

Abstract ›

In this article we present an online estimator for multirobot cooperative localization and target tracking based on nonlinear least squares minimization. Our method not only makes the rigorous optimization-based approach applicable online but also allows the estimator to be stable and convergent. We do so by employing a moving horizon technique to nonlinear least squares minimization and a novel design of the arrival cost function that ensures stability and convergence of the estimator. Through an extensive set of real robot experiments, we demonstrate the robustness of our method as well as the optimality of the arrival cost function. The experiments include comparisons of our method with i) an extended Kalman filter-based online-estimator and ii) an offline-estimator based on full-trajectory nonlinear least squares.

DOI BibTeX