Institute Homepage
Institute Homepage EN Sign In

Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation

Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group

Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award

Empirical Inference Conference Paper Necessary and sufficient conditions for causal feature selection in time series with latent common causes Mastakouri, A. A., Schölkopf, B., Janzing, D. Proceedings of 38th International Conference on Machine Learning (ICML), 139:7502-7511, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, July 2021 (Published) arXiv URL BibTeX
Empirical Inference Conference Paper Neighborhood Contrastive Learning Applied to Online Patient Monitoring Yèche, H., Dresdner, G., Locatello, F., Hüser, M., Rätsch, G. Proceedings of 38th International Conference on Machine Learning, 139:11964-11974, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, The 38th International Conference on Machine Learning (ICML 2021), July 2021 (Published) URL BibTeX
Empirical Inference Conference Paper Neural Symbolic Regression that Scales Biggio*, L., Bendinelli*, T., Neitz, A., Lucchi, A., Parascandolo, G. Proceedings of 38th International Conference on Machine Learning (ICML 2021), 139:936-945, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, The 38th International Conference on Machine Learning (ICML 2021), July 2021, *equal contribution (Published) URL BibTeX
Autonomous Learning Conference Paper Neuro-algorithmic Policies Enable Fast Combinatorial Generalization Vlastelica, M., Rolinek, M., Martius, G. In Proceedings of the 2021 International Conference on Machine Learning (ICML), The Thirty-eighth International Conference on Machine Learning (ICML), July 2021
Although model-based and model-free approa\-ches to learning the control of systems have achieved impressive results on standard benchmarks, generalization to task variations is still lacking. Recent results suggest that generalization for standard architectures improves only after obtaining exhaustive amounts of data. We give evidence that generalization capabilities are in many cases bottlenecked by the inability to generalize on the combinatorial aspects of the problem. We show that, for a certain subclass of the MDP framework, this can be alleviated by a neuro-algorithmic policy architecture that embeds a time-dependent shortest path solver in a deep neural network. Trained end-to-end via blackbox-differentiation, this method leads to considerable improvement in generalization capabilities in the low-data regime.
arXiv Spotlight PDF BibTeX
Empirical Inference Conference Paper On Disentangled Representations Learned From Correlated Data Träuble, F., Creager, E., Kilbertus, N., Locatello, F., Dittadi, A., Goyal, A., Schölkopf, B., Bauer, S. Proceedings of 38th International Conference on Machine Learning (ICML), 139:10401-10412, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, July 2021 (Published) URL BibTeX
Haptic Intelligence Article Piezoresistive Textile Layer and Distributed Electrode Structure for Soft Whole-Body Tactile Skin Lee, H., Park, K., Kim, J., Kuchenbecker, K. J. Smart Materials and Structures, 30(8):085036, July 2021, Hyosang Lee and Kyungseo Park contributed equally to this publication (Published)
Tactile sensors based on electrical resistance tomography (ERT) provide pressure sensing over a large area using only a few electrodes, which is a promising property for robotic tactile skin. Most ERT-based tactile sensors employ electrodes only on the sensor's edge to avoid undesirable artifacts caused by electrode contact. The distribution of these electrodes is critical, as electrode location largely determines the sensitive regions, but only a few studies have positioned electrodes in the sensor's central region to improve the sensitivity. Establishing the use of internal electrodes on a stretchable textile needs further investigation into piezoresistive structure fabrication, measurement strategy, and calibration. This article presents a comprehensive study of an ERT-based tactile sensor with distributed electrodes. We describe key fabrication details of a layered textile-based piezoresistive structure, an iterative method for choosing the current injection pathways that yields pairwise optimal patterns, and a calibration process to account for the spatially varying sensitivity of such sensors. We demonstrate two sample sensors with electrodes located only on the boundary or distributed across the surface, and we evaluate their performance via three methods widely used to test tactile sensing in biological systems: single-point localization, two-point discrimination, and contact force estimation.
DOI BibTeX
Haptic Intelligence Conference Paper PrendoSim: Proxy-Hand-Based Robot Grasp Generator Abdlkarim, D., Ortenzi, V., Pardi, T., Filipovica, M., Wing, A. M., Kuchenbecker, K. J., Di Luca, M. In Proceedings of the International Conference on Informatics in Control, Automation and Robotics (ICINCO), 60-68, (Editors: Gusikhin, Oleg and Nijmeijer, Henk and Madani, Kurosh), SciTePress, Virtual, July 2021 (Published)
The synthesis of realistic robot grasps in a simulated environment is pivotal in generating datasets that support sim-to-real transfer learning. In a step toward achieving this goal, we propose PrendoSim, an open-source grasp generator based on a proxy-hand simulation that employs NVIDIA's physics engine (PhysX) and the recently released articulated-body objects developed by Unity (https://prendosim.github.io). We present the implementation details, the method used to generate grasps, the approach to operationally evaluate stability of the generated grasps, and examples of grasps obtained with two different grippers (a parallel jaw gripper and a three-finger hand) grasping three objects selected from the YCB dataset (a pair of scissors, a hammer, and a screwdriver). Compared to simulators proposed in the literature, PrendoSim balances grasp realism and ease of use, displaying an intuitive interface and enabling the user to produce a large and varied dataset of stable grasps.
DOI BibTeX
Physics for Inference and Optimization Article Principled network extraction from images Baptista, D., Bacco, C. D. Royal Society Open Science, 8(7):210025, July 2021 (Published)
Images of natural systems may represent patterns of network-like structure, which could reveal important information about the topological properties of the underlying subject. However, the image itself does not automatically provide a formal definition of a network in terms of sets of nodes and edges. Instead, this information should be suitably extracted from the raw image data. Motivated by this, we present a principled model to extract network topologies from images that is scalable and efficient. We map this goal into solving a routing optimization problem where the solution is a network that minimizes an energy function which can be interpreted in terms of an operational and infrastructural cost. Our method relies on recent results from optimal transport theory and is a principled alternative to standard image-processing techniques that are based on heuristics. We test our model on real images of the retinal vascular system, slime mold and river networks and compare with routines combining image-processing techniques. Results are tested in terms of a similarity measure related to the amount of information preserved in the extraction. We find that our model finds networks from retina vascular network images that are more similar to hand-labeled ones, while also giving high performance in extracting networks from images of rivers and slime mold for which there is no ground truth available. While there is no unique method that fits all the images the best, our approach performs consistently across datasets, its algorithmic implementation is efficient and can be fully automatized to be run on several datasets with little supervision.
Preprint Code DOI BibTeX
Empirical Inference Conference Paper Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction Mastouri, A., Zhu, Y., Gultchin, L., Korba, A., Silva, R., Kusner, M., Gretton, A., Muandet, K. Proceedings of 38th International Conference on Machine Learning (ICML), 139:7512-7523, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, July 2021 (Published) URL BibTeX
Empirical Inference Conference Paper Representation Learning for Out-of-distribution Generalization in Reinforcement Learning Learning Träuble*, F., Dittadi*, A., Wüthrich, M., Widmaier, F., Gehler, P., Winther, O., Locatello, F., Bachem, O., Schölkopf, B., Bauer, S. ICML 2021 Workshop on Unsupervised Reinforcement Learning (ICML 2021) , ICML 2021 Workshop on Unsupervised Reinforcement Learning (ICML 2021) , July 2021, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Robust Value Iteration for Continuous Control Tasks Lutter, M., Mannor, S., Peters, J., Fox, D., Garg, A. Robotics: Science and Systems XVII (R:SS 2021), (Editors: Dylan A. Shell and Marc Toussaint and M. Ani Hsieh), July 2021 (Published) DOI BibTeX
Empirical Inference Conference Paper Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning Immer, A., Bauer, M., Fortuin, V., Rätsch, G., Khan, M. E. Proceedings of 38th International Conference on Machine Learning (ICML), 139:4563-4573, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, July 2021 (Published) arXiv URL BibTeX
Movement Generation and Control Conference Paper Stochastic and robust mpc for bipedal locomotion: A comparative study on robustness and performance Gazar, A., Khadiv, M., DelPrete, A., Righetti, L. 1-8, IEEE, IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), July 2021 (Published)
Linear Model Predictive Control (MPC) has been successfully used for generating feasible walking motions for humanoid robots. However, the effect of uncertainties on constraints satisfaction has only been studied using Robust MPC (RMPC) approaches, which account for the worst-case realization of bounded disturbances at each time instant. In this letter, we propose for the first time to use linear stochastic MPC (SMPC) to account for uncertainties in bipedal walking. We show that SMPC offers more flexibility to the user (or a high level decision maker) by tolerating small (user-defined) probabilities of constraint violation. Therefore, SMPC can be tuned to achieve a constraint satisfaction probability that is arbitrarily close to 100%, but without sacrificing performance as much as tube-based RMPC. We compare SMPC against RMPC in terms of robustness (constraint satisfaction) and performance (optimality). Our results highlight the benefits of SMPC and its interest for the robotics community as a powerful mathematical tool for dealing with uncertainties.
DOI BibTeX
Autonomous Learning Conference Paper The dynamical regime and its importance for evolvability, task performance and generalization Prosi, J., Khajehabdollahi, S., Giannakakis, E., Martius, G., Levina, A. In The 2021 Conference on Artificial Life, MIT Press, July 2021 PDF DOI URL BibTeX
Empirical Inference Conference Paper Value Iteration in Continuous Actions, States and Time Lutter, M., Mannor, S., Peters, J., Fox, D., Garg, A. Proceedings of 38th International Conference on Machine Learning (ICML), 139:7224-7234, Proceedings of Machine Learning Research, (Editors: Meila, Marina and Zhang, Tong), PMLR, July 2021 (Published) URL BibTeX
Haptic Intelligence Miscellaneous Vibrotactile Playback for Teaching Sensorimotor Skills in Medical Procedures Gourishetti, R., Kuchenbecker, K. J. Hands-on demonstration presented at the IEEE World Haptics Conference (WHC), July 2021 (Published) BibTeX
Empirical Inference Article SKID RAW: Skill Discovery From Raw Trajectories Tanneberg, D., Ploeger, K., Rueckert, E., Peters, J. IEEE Robotics and Automation Letters, 6(3):4696-4703, IEEE, New York, NY, July 2021 (Published) DOI BibTeX
Modern Magnetic Systems Ph.D. Thesis Röntgenmikroskopische Untersuchungen der Magnetisierungsdynamik in nanoskaligen magnonischen Wellenleiterstrukturen Träger, N. Universität Stuttgart, Stuttgart (und Cuvillier Verlag, Göttingen), June 2021 (Published) BibTeX
Micro, Nano, and Molecular Systems Article Comment on “Using NMR to Test Molecular Mobility during a Chemical Reaction” Fillbrook, L. L., Günther, J., Majer, G., Price, W. S., Fischer, P., Beves, J. E. The Journal of Physical Chemistry Letters, 12(25):5932-5937, June 2021
A study reported in The Journal of Physical Chemistry Letters (Wang et al., 2021, 12, 2370) of “boosted mobility” measured by diffusion NMR experiments contains significant errors in data analysis and interpretation. We carefully reanalyzed the same data and find no evidence of boosted mobility, and we identify several sources of error.
DOI URL BibTeX
Materials Article Synthesis of magnetic Fe and Co nano-whiskers and platelets via physical vapor deposition Huang, W., Gatel, C., Li, Z., Richter, G. Materials & Design, 208:109914, June 2021 (Published) DOI BibTeX
Dynamic Locomotion Article Hybrid Parallel Compliance Allows Robots to Operate With Sensorimotor Delays and Low Control Frequencies Milad Shafiee Ashtiani, , Alborz Aghamaleki Sarvestani, , Badri-Spröwitz, A. Frontiers in Robotics and AI, 8(na):645748, (Editors: Dai Owaki, Tohoku University, Japan), June 2021 (Published)
Animals locomote robustly and agile, albeit significant sensorimotor delays of their nervous system and the harsh loading conditions resulting from repeated, high-frequent impacts. The engineered sensorimotor control in legged robots is implemented with high control frequencies, often in the kilohertz range. Consequently, robot sensors and actuators can be polled within a few milliseconds. However, especially at harsh impacts with unknown touch-down timing, controllers of legged robots can become unstable, while animals are seemingly not affected. We examine this discrepancy and suggest and implement a hybrid system consisting of a parallel compliant leg joint with varying amounts of passive stiffness and a virtual leg length controller. We present systematic experiments both in computer simulation and robot hardware. Our system shows previously unseen robustness, in the presence of sensorimotor delays up to 60 ms, or control frequencies as low as 20 Hz, for a drop landing task from 1.3 leg lengths high and with a compliance ratio (fraction of physical stiffness of the sum of virtual and physical stiffness) of 0.7. In computer simulations, we report successful drop-landings from 3.8 leg lengths (1.2 m) for a 2 kg quadruped robot with 100 Hz control frequency and a sensorimotor delay of 35 ms.
CAD spring-mount DOI URL BibTeX
Micro, Nano, and Molecular Systems Article Dynamic Acoustic Levitator Based On Subwavelength Aperture Control Lu, X., Twiefel, J., Ma, Z., Yu, T., Wallaschek, J., Fischer, P. Advanced Science, 8(15):2100888, June 2021
Acoustic levitation provides a means to achieve contactless manipulation of fragile materials and biological samples. Most acoustic levitators rely on complex electronic hardware and software to shape the acoustic field and realize their dynamic operation. Here, the authors introduce a dynamic acoustic levitator that is based on mechanically controlling the opening and (partial) closing of subwavelength apertures. This simple approach relies on the use of a single ultrasonic transducer and is shown to permit the facile and reliable manipulation of a variety targets ranging from solid particles, to fluid and ferrofluidic drops. Experimental observations agree well with numerical simulations of the Gor'kov potential. Remarkably, this system even enables the generation of time-varying potentials and induces oscillatory and rotational motion in the levitated objects via a feedback mechanism between the trapped object and the trapping potential. This is shown to result in long distance translation, in-situ rotation and self-modulated oscillation of the trapped particles. In addition, dense ferrofluidic droplets are levitated and transformed inside the levitator. Controlling subwavelength apertures opens the possibility to realize simple powerful levitators that nevertheless allow for the versatile dynamic manipulation of levitated matter.
DOI URL BibTeX
Intelligent Control Systems Conference Paper Probabilistic robust linear quadratic regulators with Gaussian processes von Rohr, A., Neumann-Brosig, M., Trimpe, S. Proceedings of the 3rd Conference on Learning for Dynamics and Control, 324-335, Proceedings of Machine Learning Research (PMLR), Vol. 144, (Editors: Jadbabaie, Ali and Lygeros, John and Pappas, George J. and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.), PMLR, Brookline, MA 02446 , 3rd Annual Conference on Learning for Dynamics and Control (L4DC), June 2021 (Published)
Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. While learning-based control has the potential to yield superior performance in demanding applications, robustness to uncertainty remains an important challenge. Since Bayesian methods quantify uncertainty of the learning results, it is natural to incorporate these uncertainties in a robust design. In contrast to most state-of-the-art approaches that consider worst-case estimates, we leverage the learning methods’ posterior distribution in the controller synthesis. The result is a more informed and thus efficient trade-off between performance and robustness. We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin. The formulation is based on a recently proposed algorithm for linear quadratic control synthesis, which we extend by giving probabilistic robustness guarantees in the form of credibility bounds for the system’s stability. Comparisons to existing methods based on worst-case and certainty-equivalence designs reveal superior performance and robustness properties of the proposed method.
DOI URL BibTeX
Micro, Nano, and Molecular Systems Modern Magnetic Systems Article Light- and magnetically actuated FePt microswimmers Kadiri, V. M., Günther, J., Kottapalli, S. N., Goyal, R., Peter, F., Alarcon-Correa, M., Son, K., Barad, H., Börsch, M., Fischer, P. The European Physical Journal E, 44(6):74, June 2021
Externally controlled microswimmers offer prospects for transport in biological research and medical applications. This requires biocompatibility of the swimmers and the possibility to tailor their propulsion mechanisms to the respective low Reynolds number environment. Here, we incorporate low amounts of the biocompatible alloy of iron and platinum (FePt) in its L10 phase in microstructures by a versatile one-step physical vapor deposition process. We show that the hard magnetic properties of L10 FePt are beneficial for the propulsion of helical micropropellers with rotating magnetic fields. Finally, we find that the FePt coatings are catalytically active and also make for Janus microswimmers that can be light-actuated and magnetically guided.
DOI URL BibTeX
Autonomous Motion Conference Paper Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12 Leziart, P. F. T. G. F. M. N. S. P. Proceedings IEEE International Conference on Robotics and Automation (ICRA), 5007-5013, IEEE, IEEE International Conference on Robotics and Automation (ICRA), June 2021 (Published) DOI BibTeX
Micro, Nano, and Molecular Systems Article Panoramic imaging assessment of different bladder phantoms – an evaluation study Hackner, R., Suarez-Ibarrola, I., Qiu, T., Lemke, N., Pohlmann, P., Wilhelm, K., Fischer, P., Miernik, A., Wittenberg, T. Urology, 156:e103-e110, June 2021
Objective: To evaluate “panoramic image stitching” for cystoscopy, a novel technique to augment an urologist's field of view transoperatively in real-time during a cystoscopic “keyhole” procedure, 3-D bladder phantoms provide a suitable setting. Thus, the objective is the evaluation of different 3-D printed bladder phantoms with respect to their ability to be used for extended experiments of panoramic cystoscopy. Results: Panoramas of all phantom and endoscope combinations were computed. Using landmarks (south pole, north pole, equator) in the phantoms, maximum extension of the panoramas was assessed. The computed panoramas yield maximum extensions between 270o (0-degree cystoscope) and 330o (video cystoscope). Deformable phantoms yield larger panoramas than the rigid models.
DOI URL BibTeX
Empirical Inference Conference Paper Approximate Distributionally Robust Nonlinear Optimization with Application to Model Predictive Control: A Functional Approach Nemmour, Y., Schölkopf, B., Zhu, J. Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC), 144:1255-1269, Proceedings of Machine Learning Research, (Editors: Jadbabaie, Ali and Lygeros, John and Pappas, George J. and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.), PMLR, June 2021 (Published) URL BibTeX
Empirical Inference Article Automated Morphometric Analysis of the Hip Joint on MRI from the German National Cohort Study Fischer, M., Walter, S. S., Hepp, T., Zimmer, M., Notohamiprodjo, M., Schick, F., Yang, B. Radiology: Artificial Intelligence, 3(5), June 2021 (Published) DOI BibTeX
Empirical Inference Ph.D. Thesis Causal Inference in Vision Meding, K. Eberhard Karls Universität Tübingen, Tübingen, June 2021 (Published) BibTeX
Empirical Inference Article Deep Learning-Based Automated Abdominal Organ Segmentation in the UK Biobank and German National Cohort Magnetic Resonance Imaging Studies Kart, T., Fischer, M., Küstner, T., Hepp, T., Bamberg, F., Winzeck, S., Glocker, B., Rueckert, D., Gatidis, S. Investigative Radiology, 56(6):401-408, June 2021 (Published) DOI BibTeX
Movement Generation and Control Conference Paper DeepQ Stepper: A framework for reactive dynamic walking on uneven terrain Meduri, A., Khadiv, M., Righetti, L. The 2021 International Conference on Robotics and Automation (ICRA 2021), June 2021 (Published)
Reactive stepping and push recovery for biped robots is often restricted to flat terrains because of the difficulty in computing capture regions for nonlinear dynamic models. In this paper, we address this limitation by using reinforcement learning to approximately learn the 3D capture region for such systems. We propose a novel 3D reactive stepper, The DeepQ stepper, that computes optimal step locations for walking at different velocities using the 3D capture regions approximated by the action-value function. We demonstrate the ability of the approach to learn stepping with a simplified 3D pendulum model and a full robot dynamics. Further, the stepper achieves a higher performance when it learns approximate capture regions while taking into account the entire dynamics of the robot that are often ignored in existing reactive steppers based on simplified models. The DeepQ stepper can handle non convex terrain with obstacles, walk on restricted surfaces like stepping stones and recover from external disturbances for a constant computational cost.
URL BibTeX
Empirical Inference Article Discrimination of simple objects decoded from the output of retinal ganglion cells upon sinusoidal electrical stimulation Corna, A., Ramesh, P., Jetter, F., Lee, M., Macke, J. H., Zeck, G. Journal of Neural Engineering, 18(4), June 2021 (Published) DOI BibTeX
Empirical Inference Conference Paper Distilling Audio-Visual Knowledge by Compositional Contrastive Learning Chen, Y., Xian, Y., Koepke, A. S., Akata, Z. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7016-7025, Computer Vision Foundation / IEEE, CVPR, June 2021 (Published) URL BibTeX
Haptic Intelligence Article Free and Forced Vibration Modes of the Human Fingertip Serhat, G., Kuchenbecker, K. J. Applied Sciences, 11(12):5709, June 2021 (Published)
Computational analysis of free and forced vibration responses provides crucial information on the dynamic characteristics of deformable bodies. Although such numerical techniques are prevalently used in many disciplines, they have been underutilized in the quest to understand the form and function of human fingers. We addressed this opportunity by building DigiTip, a detailed three-dimensional finite element model of a representative human fingertip that is based on prior anatomical and biomechanical studies. Using the developed model, we first performed modal analyses to determine the free vibration modes with associated frequencies up to about 250 Hz, the frequency at which humans are most sensitive to vibratory stimuli on the fingertip. The modal analysis results reveal that this typical human fingertip exhibits seven characteristic vibration patterns in the considered frequency range. Subsequently, we applied distributed harmonic forces at the fingerprint centroid in three principal directions to predict forced vibration responses through frequency-response analyses; these simulations demonstrate that certain vibration modes are excited significantly more efficiently than the others under the investigated conditions. The results illuminate the dynamic behavior of the human fingertip in haptic interactions involving oscillating stimuli, such as textures and vibratory alerts, and they show how the modal information can predict the forced vibration responses of the soft tissue.
DOI BibTeX
Autonomous Vision Conference Paper GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Niemeyer, M., Geiger, A. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11448-11459 , IEEE, Conference on Computer Vision and Pattern Recognition (CVPR), June 2021 (Published)
Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and hence ignore that our world is three-dimensional. Further, only few works consider the compositional nature of scenes. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. Representing scenes as compositional generative neural feature fields allows us to disentangle one or multiple objects from the background as well as individual objects' shapes and appearances while learning from unstructured and unposed image collections without any additional supervision. Combining this scene representation with a neural rendering pipeline yields a fast and realistic image synthesis model. As evidenced by our experiments, our model is able to disentangle individual objects and allows for translating and rotating them in the scene as well as changing the camera pose.
pdf suppmat video Project Page DOI URL BibTeX
Movement Generation and Control Conference Paper High-frequency nonlinear model predictive control of a manipulator Kleff, S., Meduri, A., Budhiraja, R., Mansard, N., Righetti, L. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 7330-7336 , The 2021 International Conference on Robotics and Automation (ICRA 2021), June 2021 (Published)
Model Predictive Control (MPC) promises to endow robots with enough reactivity to perform complex tasks in dynamic environments by frequently updating their motion plan based on measurements. Despite its appeal, it has seldom been deployed on real machines because of scaling constraints. This paper presents the first hardware implementation of closed-loop nonlinear MPC on a 7-DoF torque-controlled robot. Our controller leverages a state-of-the art optimal control solver, namely Differential Dynamic Programming (DDP), in order to replan state and control trajectories at real-time rates (1kHz). In addition to this experimental proof of concept, we present exhaustive performance analysis on the iconic pick-and-place task and show that our controller outperforms open-loop MPC. We also exhibit the importance of a sufficient preview horizon and full robot dynamics in the controller performance through comparisons with inverse dynamics and kinematic optimization.
DOI BibTeX
Autonomous Motion Autonomous Learning Article How to Train Your Differentiable Filter Kloss, A., Martius, G., Bohg, J. Autonomous Robots, 45(4):561-578, Springer, June 2021 (Published)
In many robotic applications, it is crucial to maintain a belief about the state of a system, which serves as input for planning and decision making and provides feedback during task execution. Bayesian Filtering algorithms address this state estimation problem, but they require models of process dynamics and sensory observations and the respective noise characteristics of these models. Recently, multiple works have demonstrated that these models can be learned by end-to-end training through differentiable versions of recursive filtering algorithms. In this work, we investigate the advantages of differentiable filters (DFs) over both unstructured learning approaches and manually-tuned filtering algorithms, and provide practical guidance to researchers interested in applying such differentiable filters. For this, we implement DFs with four different underlying filtering algorithms and compare them in extensive experiments. Specifically, we (i) evaluate different implementation choices and training approaches, (ii) investigate how well complex models of uncertainty can be learned in DFs, (iii) evaluate the effect of end-to-end training through DFs and (iv) compare the DFs among each other and to unstructured LSTM models.
arXiv paper DOI URL BibTeX
Rationality Enhancement Talk Improving Human Decision-Making by Discovering Efficient Strategies for Hierarchical Planning Heindrich, L., Consul, S., Stojcheski, J., Lieder, F. Tübingen, Germany, The first edition of Life Improvement Science Conference, June 2021 (Accepted)
The discovery of decision strategies is an essential part of creating effective cognitive tutors that teach planning and decision-making skills to humans. In the context of bounded rationality, this requires weighing the benefits of different planning operations compared to their computational costs. For small decision problems, it has already been shown that near-optimal decision strategies can be discovered automatically and that the discovered strategies can be taught to humans to increase their performance. Unfortunately, these near-optimal strategy discovery algorithms have not been able to scale well to larger problems due to their computational complexity. In this talk, we will present recent work at the Rationality Enhancement Group to overcome the computational bottleneck of existing strategy discovery algorithms. Our approach makes use of the hierarchical structure of human behavior by decomposing sequential decision problems into two sub-problems: setting a goal and planning how to achieve it. An additional metacontroller component is introduced to switch the current goal when it becomes beneficial. The hierarchical decomposition enables us to discover near-optimal strategies for human planning in larger and more complex tasks than previously possible. We then show in online experiments that teaching the discovered strategies to humans improves their performance in complex sequential decision-making tasks.
BibTeX
Empirical Inference Conference Paper Learning Decision Trees Recurrently Through Communication Alaniz, S., Marcos, D., Schiele, B., Akata, Z. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13518-13527, Computer Vision Foundation / IEEE, CVPR, June 2021 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Graph Embeddings for Compositional Zero-shot Learning Naeem, M. F., Xian, Y., Tombari, F., Akata, Z. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 953-962, Computer Vision Foundation / IEEE, CVPR, June 2021 (Published) URL BibTeX
Movement Generation and Control Conference Paper Learning a Centroidal Motion Planner for Legged Locomotion Viereck, J., Righetti, L. 2021 IEEE International Conference on Robotics and Automation (ICRA), 4905-4911 , IEEE, The 2021 International Conference on Robotics and Automation (ICRA 2021), June 2021 (Published)
Whole-body optimizers have been successful at automatically computing complex dynamic locomotion behaviors. However they are often limited to offline planning as they are computationally too expensive to replan with a high frequency. Simpler models are then typically used for online replanning. In this paper we present a method to generate whole body movements in real-time for locomotion tasks. Our approach consists in learning a centroidal neural network that predicts the desired centroidal motion given the current state of the robot and a desired contact plan. The network is trained using an existing whole body motion optimizer. Our approach enables to learn with few training samples dynamic motions that can be used in a complete whole-body control framework at high frequency, which is usually not attainable with typical full-body optimizers. We demonstrate our method to generate a rich set of walking and jumping motions on a real quadruped robot.
DOI BibTeX
Rationality Enhancement Conference Paper Leveraging AI to support the self-directed learning of disadvantaged youth in developing countries Teo, J., Pauly, R., Heindrich, L., Amo, V., Lieder, F. The first Life Improvement Science Conference, Tübingen, Germany, The first Life Improvement Science Conference, June 2021 (Accepted)
Globally 258 million children and youth do not have access to school (Unesco, 2019), while 600 million receive ineffective education (Unesco, 2017). Solve Education! (SE!) is a non-profit organization committed to enable these young people to empower themselves through education, and currently operates in over 7 countries. Their team includes educationists, technologists, and business executives, who work together with governments and local communities to reach young people with disadvantaged backgrounds. Solve Education!’s main mobile application “The Dawn of Civilisation” (DoC), is an open platform that can deliver different learning content, with the focus on English literacy. It is designed to support lower end devices, as well as offline learning. At the Rationality Enhancement Group, we are laying the scientific foundation for helping people do more good in better ways. We combine methods from computational cognitive science, psychology, human-computer interaction, and artificial intelligence for the development of practical tools, strategies, and interventions that support people in their personal growth. In our collaboration with SE!, we aim at learning from and contributing to real-world challenges by applying our research to enhance SE!’s learning platform. We are currently working on two projects. The first project’s goal is to develop a principled approach to incentivize efficient self-directed learning with digital educational resources and to evaluate its effectiveness regarding learners’ behaviors and success in cooperation with SE!. Specifically, SE!’s DoC serves as the digital educational resource and allows to evaluate the approach with very high ecological validity. The planned intervention is based on the concept of optimal brain points developed by Xu, Wirzberger & Lieder (2019). The core idea is to incentivize effort and smart study choices rather than performance and to do so in a way that learners cannot exploit shortcuts to accumulate game points without also moving closer to their actual learning goals. If successful, SE! can build upon the intervention to further enhance the benefits their users draw from DoC. The second project is based on hierarchical goal setting and consists of a digital assistant that helps users set real-world goals and make progress towards them by reaching milestones with DoC. In this talk, in addition to introducing our work together with SE, we will highlight the mutual benefits of the collaboration between scientists and socially impactful organizations.
BibTeX
Movement Generation and Control Conference Paper Leveraging Forward Model Prediction Error for Learning Control Bechtle, S., Hammoud, B., Rai, A., Meier, F., Righetti, L. 2021 IEEE International Conference on Robotics and Automation (ICRA) , 4445-4451 , IEEE International Conference on Robotics and Automation (ICRA 2021), June 2021 (Published)
Learning for model based control can be sample-efficient and generalize well, however successfully learning models and controllers that represent the problem at hand can be challenging for complex tasks. Using inaccurate models for learning can lead to sub-optimal solutions, that are unlikely to perform well in practice. In this work, we present a learning approach which iterates between model learning and data collection and leverages forward model prediction error for learning control. We show how using the controller's prediction as input to a forward model can create a differentiable connection between the controller and the model, allowing us to formulate a loss in the state space. This lets us include forward model prediction error during controller learning and we show that this creates a loss objective that significantly improves learning on different motor control tasks. We provide empirical and theoretical results that show the benefits of our method and present evaluations in simulation for learning control on a 7 DoF manipulator and an underactuated 12 DoF quadruped. We show that our approach successfully learns controllers for challenging motor control tasks involving contact switching.
URL BibTeX
Empirical Inference Conference Paper Neural Lyapunov Redesign Mehrjou, A., Ghavamzadeh, M., Schölkopf, B. Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC), 144:459-470, Proceedings of Machine Learning Research, (Editors: Jadbabaie, Ali and Lygeros, John and Pappas, George J. and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.), PMLR, 3rd Annual Conference on Learning for Dynamics and Control (L4DC) , June 2021 (Published) URL BibTeX
Perceiving Systems Conference Paper On Self-Contact and Human Pose Müller, L., Osman, A. A. A., Tang, S., Huang, C. P., Black, M. J. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 9985-9994, IEEE, Piscataway, NJ, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), June 2021 (Published)
People touch their face 23 times an hour, they cross their arms and legs, put their hands on their hips, etc. While many images of people contain some form of self-contact, current 3D human pose and shape (HPS) regression methods typically fail to estimate this contact. To address this, we develop new datasets and methods that significantly improve human pose estimation with self-contact. First, we create a dataset of 3D Contact Poses (3DCP) containing SMPL-X bodies fit to 3D scans as well as poses from AMASS, which we refine to ensure good contact. Second, we leverage this to create the Mimic-The-Pose (MTP) dataset of images, collected via Amazon Mechanical Turk, containing people mimicking the 3DCP poses with self-contact. Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images. Fourth, for more image variety, we label a dataset of in-the-wild images with Discrete Self-Contact (DSC) information and use another new optimization method, SMPLify-DC, that exploits discrete contacts during pose optimization. Finally, we use our datasets during SPIN training to learn a new 3D human pose regressor, called TUCH (Towards Understanding Contact in Humans). We show that the new self-contact training data significantly improves 3D human pose estimates on withheld test data and existing datasets like 3DPW. Not only does our method improve results for self-contact poses, but it also improves accuracy for non-contact poses. The code and data are available for research purposes at https://tuch.is.tue.mpg.de.
project arXiv poster video code DOI BibTeX
Intelligent Control Systems Conference Paper On exploration requirements for learning safety constraints Massiani, P., Heim, S., Trimpe, S. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, 905-916, Proceedings of Machine Learning Research (PMLR), Vol. 144, (Editors: Jadbabaie, Ali and Lygeros, John and Pappas, George J. and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie), PMLR, 3rd Annual Conference on Learning for Dynamics and Control (L4DC), June 2021 (Published)
Enforcing safety for dynamical systems is challenging, since it requires constraint satisfaction along trajectory predictions. Equivalent control constraints can be computed in the form of sets that enforce positive invariance, and can thus guarantee safety in feedback controllers without predictions. However, these constraints are cumbersome to compute from models, and it is not yet well established how to infer constraints from data. In this paper, we shed light on the key objects involved in learning control constraints from data in a model-free setting. In particular, we discuss the family of constraints that enforce safety in the context of a nominal control policy, and expose that these constraints do not need to be accurate everywhere. They only need to correctly exclude a subset of the state-actions that would cause failure, which we call the critical set.
URL BibTeX
Physics for Inference and Optimization Article Optimal Transport in Multilayer Networks for Traffic Flow Optimization for Traffic Flow Optimization Ibrahim, A. A., Lonardi, A., Bacco, C. D. Algorithms, 14(7):189, June 2021 (Published)
Modeling traffic distribution and extracting optimal flows in multilayer networks is of the utmost importance to design efficient, multi-modal network infrastructures. Recent results based on optimal transport theory provide powerful and computationally efficient methods to address this problem, but they are mainly focused on modeling single-layer networks. Here, we adapt these results to study how optimal flows distribute on multilayer networks. We propose a model where optimal flows on different layers contribute differently to the total cost to be minimized. This is done by means of a parameter that varies with layers, which allows to flexibly tune the sensitivity to the traffic congestion of the various layers. As an application, we consider transportation networks, where each layer is associated to a different transportation system, and show how the traffic distribution varies as we tune this parameter across layers. We show an example of this result on the real, 2-layer network of the city of Bordeaux with a bus and tram, where we find that in certain regimes, the presence of the tram network significantly unburdens the traffic on the road network. Our model paves the way for further analysis of optimal flows and navigability strategies in real, multilayer networks.
Code Preprint DOI BibTeX
Empirical Inference Ph.D. Thesis Optimization Algorithms for Machine Learning Raj, A. University of Tübingen, Germany, June 2021 (Published) BibTeX
Empirical Inference Conference Paper Orthogonal Over-Parameterized Training Liu, W., Lin, R., Liu, Z., Rehg, J., Paull, L., Xiong, L., Song, L., Weller, A. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7251-7260, Computer Vision Foundation / IEEE, CVPR, June 2021 (Published) URL BibTeX
Perceiving Systems Conference Paper Populating 3D Scenes by Learning Human-Scene Interaction Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M. J. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 14703-14713, IEEE, Piscataway, NJ, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), June 2021 (Published)
Humans live within a 3D space and constantly interact with it to perform tasks. Such interactions involve physical contact between surfaces that is semantically meaningful. Our goal is to learn how humans interact with scenes and leverage this to enable virtual characters to do the same. To that end, we introduce a novel Human-Scene Interaction (HSI) model that encodes proximal relationships, called POSA for “Pose with prOximitieS and contActs”. The representation of interaction is body-centric, which enables it to generalize to new scenes. Specifically, POSA augments the SMPL-X parametric human body model such that, for every mesh vertex, it encodes (a) the contact probability with the scene surface and (b) the corresponding semantic scene label. We learn POSA with a VAE conditioned on the SMPL-X vertices, and train on the PROX dataset, which contains SMPL-X meshes of people interacting with 3D scenes, and the corresponding scene semantics from the PROX-E dataset. We demonstrate the value of POSA with two applications. First, we automatically place 3D scans of people in scenes. We use a SMPL-X model fit to the scan as a proxy and then find its most likely placement in 3D. POSA provides an effective representation to search for “affordances” in the scene that match the likely contact relationships for that pose. We perform a perceptual study that shows significant improvement over the state of the art on this task. Second, we show that POSA’s learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art. Our model and code are available for research purposes at https://posa.is.tue.mpg.de.
project pdf poster video DOI BibTeX