Institute Homepage
Institute Homepage DE Sign In

Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation

Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group

Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award

Modern Magnetic Systems Thesis Voltage dependent interfacial magnetism in multilayer systems Nacke, R. Universität Stuttgart, Stuttgart, December 2020 BibTeX
Modern Magnetic Systems Master Thesis Hydromagnonics: Manipulation of magnonic systems with hydrogen Sauter, R. Universität Stuttgart, Stuttgart, December 2020 BibTeX
Dynamic Locomotion Article Virtual Point Control for Step-down Perturbations and Downhill Slopes in Bipedal Running Drama, Ö., Badri-Spröwitz, A. Frontiers in Bioengineering and Biotechnology, 8:586534, Frontiers Media, December 2020 (Published)
Bipedal running is a difficult task to realize in robots, since the trunk is underactuated and control is limited by intermittent ground contacts. Stabilizing the trunk becomes even more challenging if the terrain is uneven and causes perturbations. One bio-inspired method to achieve postural stability is the virtual point (VP) control, which is able to generate natural motion. However, so far it has only been studied for level running. In this work, we investigate whether the VP control method can accommodate single step-down perturbations and downhill terrains. We provide guidelines on the model and controller parameterizations for handling varying terrain conditions. Next, we show that the VP method is able to stabilize single step-down perturbations up to 40 cm, and downhill grades up to 20-10° corresponding to running speeds of 2-5 m/s. Our results show that the VP approach leads to asymmetrically bounded ground reaction forces for downhill running, unlike the commonly-used symmetric friction cone constraints. Overall, VP control is a promising candidate for terrain-adaptive running control of bipedal robots.
DOI URL BibTeX
Intelligent Control Systems Ph.D. Thesis Learning and Control Strategies for Cyber-physical Systems: From Wireless Control over Deep Reinforcement Learning to Causal Identification Baumann, D. KTH Royal Institute of Technology, Stockholm, Sweden, December 2020 (Published) PDF URL BibTeX
Empirical Inference Conference Paper A Class of Algorithms for General Instrumental Variable Models Kilbertus, N., Kusner, M. J., Silva, R. Advances of Neural Information Processing Systems 33 (NeurIPS 2020), 33:20108-20119, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates Inc., 34th Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings Park, J., Muandet, K. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 21247-21259, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Probabilistic Learning Group Conference Paper Algorithmic recourse under imperfect causal knowledge: a probabilistic approach Karimi*, A., von Kügelgen*, J., Schölkopf, B., Valera, I. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 265-277, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020, *equal contribution (Published) arXiv URL BibTeX
Empirical Inference Conference Paper Barking up the right tree: an approach to search over molecule synthesis DAGs Bradshaw, J., Paige, B., Kusner, M., Segler, M., Hernández-Lobato, J. M. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 6852-6866, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Ph.D. Thesis Causal Feature Selection in Neuroscience Mastakouri, A. University of Tübingen, Germany, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Causal analysis of Covid-19 Spread in Germany Mastakouri, A., Schölkopf, B. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 3153-3163, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Haptic Intelligence Ph.D. Thesis Delivering Expressive and Personalized Fingertip Tactile Cues Young, E. M. University of Pennsylvania, Philadelphia, PA, December 2020, Department of Mechanical Engineering and Applied Mechanics (Published)
Wearable haptic devices have seen growing interest in recent years, but providing realistic tactile feedback is not a challenge that is soon to be solved. Daily interac- tions with physical objects elicit complex sensations at the fingertips. Furthermore, human fingertips exhibit a broad range of physical dimensions and perceptive abilities, adding increased complexity to the task of simulating haptic interactions in a compelling manner. However, as the applications of wearable haptic feedback grow, concerns of wearability and generalizability often persuade tactile device designers to simplify the complexities associated with rendering realistic haptic sensations. As such, wearable devices tend to be optimized for particular uses and average users, rendering only the most salient dimensions of tactile feedback for a given task and assuming all users interpret the feedback in a similar fashion. We propose that providing more realistic haptic feedback will require in-depth examinations of higher-dimensional tactile cues and personalization of these cues for individual users. In this thesis, we aim to provide hardware and software-based solutions for rendering more expressive and personalized tactile cues to the fingertip. We first explore the idea of rendering six-degree-of-freedom (6-DOF) tactile fingertip feedback via a wearable device, such that any possible fingertip interaction with a flat surface can be simulated. We highlight the potential of parallel continuum manipulators (PCMs) to meet the requirements of such a device, and we refine the design of a PCM for providing fingertip tactile cues. We construct a manually actuated prototype to validate the concept, and then continue to develop a motorized version, named the Fingertip Puppeteer, or Fuppeteer for short. Various error reduction techniques are presented, and the resulting device is evaluated by analyzing system responses to step inputs, measuring forces rendered to a biomimetic finger sensor, and comparing intended sensations to perceived sensations of twenty-four participants in a human-subject study. Once the functionality of the Fuppeteer is validated, we begin to explore how the device can be used to broaden our understanding of higher-dimensional tactile feedback. One such application is using the 6-DOF device to simulate different lower-dimensional devices. We evaluate 1-, 3-, and 6-DOF tactile feedback during shape discrimination and mass discrimination in a virtual environment, also comparing to interactions with real objects. Results from 20 naive study participants show that higher-dimensional tactile feedback may indeed allow completion of a wider range of virtual tasks, but that feedback dimensionality surprisingly does not greatly affect the exploratory techniques employed by the user. To address alternative approaches to improving tactile rendering in scenarios where low-dimensional tactile feedback is appropriate, we then explore the idea of personalizing feedback for a particular user. We present two software-based approaches to personalize an existing data-driven haptic rendering algorithm for fingertips of different sizes. We evaluate our algorithms in the rendering of pre-recorded tactile sensations onto rubber casts of six different fingertips as well as onto the real fingertips of 13 human participants, all via a 3-DOF wearable device. Results show that both personalization approaches significantly reduced force error magnitudes and improved realism ratings.
BibTeX
Empirical Inference Conference Paper Dual Instrumental Variable Regression Muandet, K., Mehrjou, A., Lee, S. K., Raj, A. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2710-2721, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Autonomous Vision Conference Paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis Schwarz, K., Liao, Y., Niemeyer, M., Geiger, A. In Advances in Neural Information Processing Systems 33, 25:20154-20166, (Editors: Larochelle , H. and Ranzato, M. and Hadsell , R. and Balcan , M. F. and Lin, H.), Curran Associates, Inc., Red Hook, NY, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020 (Published)
While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, eg, the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.
pdf suppmat video Project Page URL BibTeX
Rationality Enhancement Article Improving Human Decision-Making using Metalevel-RL and Bayesian Inference Kemtur, A., Jain, Y. R., Mehta, A., Callaway, F., Consul, S., Stojcheski, J., Lieder, F. NeurIPS Workshop on Challenges for Real-World RL, December 2020 (Accepted)
Teaching clever heuristics is a promising approach to improve decision-making. We can leverage machine learning to discover clever strategies automatically. Current methods require an accurate model of the decision problems people face in real life. But most models are misspecified because of limited in-formation and cognitive biases. To address this problem we develop strategy discovery methods that are robust to model misspecification. Robustness is achieved by modeling model-misspecification using common cognitive biases and handling uncertainty about the real-world according to Bayesian inference. We translate our methods into an intelligent tutor that automatically discovers and teaches robust planning strategies. Our robust cognitive tutor significantly improved human decision-making when the model was so biased that conventional cognitive tutors were no longer effective. These findings highlight that our robust strategy discovery methods are a significant step towards leveraging artificial intelligence to improve human decision-making in the real world.Teaching clever heuristics is a promising approach to improve decision-making. We can leverage machine learning to dis- cover clever strategies automatically. Current methods require an accurate model of the decision problems people face in real life. But most models are misspecified because of limited information and cognitive biases. To address this problem we develop strategy discovery methods that are robust to model misspecification. Robustness is achieved by modeling model-misspecification using common cognitive biases and handling uncertainty about the real-world according to Bayesian inference. We translate our methods into an intelligent tutor that automatically discovers and teaches robust planning strategies. Our robust cognitive tutor significantly improved human decision-making when the model was so biased that conventional cognitive tutors were no longer effective. These findings highlight that our robust strategy discovery methods are a significant step towards leveraging artificial intelligence to improve human decision-making in the real world.
Improving Human Decision-Making using Metalevel-RL and Bayesian Inference BibTeX
Empirical Inference Conference Paper Learning Kernel Tests Without Data Splitting Kübler, J. M., Jitkrittum, W., Schölkopf, B., Muandet, K. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 6245-6255, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Modeling Shared responses in Neuroimaging Studies through MultiView ICA Richard, H., Gresele, L., Hyvärinen, A., Thirion, B., Gramfort, A., Ablin, P. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 19149-19162, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., Red Hook, NY, 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Object-Centric Learning with Slot Attention Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., Kipf, T. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 11525-11538, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Movement Generation and Control Article On the use of simulation in robotics: Opportunities challenges, and suggestions for moving forward Choi, H., Crump, C., Duriez, C., Elmquist, A., Hager, G., Han, D., Hearl, F., Hodgins, J., Jain, A., Leve, F., Li, C., Meier, F., Negrut, D., Righetti, L., Rodriguez, A., Tan, J., Trinkle, J. PNAS, 118(1), December 2020 (Published) DOI BibTeX
Empirical Inference Conference Paper Probabilistic Linear Solvers for Machine Learning Wenger, J., Hennig, P. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 6731-6742, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Probabilistic Learning Group Conference Paper Relative gradient optimization of the Jacobian term in unsupervised deep learning Gresele, L., Fissore, G., Javaloy, A., Schölkopf, B., Hyvarinen, A. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 16567-16578, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining Tripp, A., Daxberger, E., Hernández-Lobato, J. M. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 11259-11272, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Self-Paced Deep Reinforcement Learning Klink, P., D’Eramo, C., Peters, J., Pajarinen, J. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 9216-9227, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Empirical Inference Conference Paper Stochastic Stein Discrepancies Gorham, J., Raj, A., Mackey, L. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 17931-17942, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Haptic Intelligence Conference Paper Synchronicity Trumps Mischief in Rhythmic Human-Robot Social-Physical Interaction Fitter, N. T., Kuchenbecker, K. J. In Robotics Research, 10:269-284, Springer Proceedings in Advanced Robotics, (Editors: Amato, Nancy M. and Hager, Greg and Thomas, Shawna and Torres-Torriti, Miguel), Springer Cham, International Symposium on Robotics Research (ISRR), December 2020 (Published)
Hand-clapping games and other forms of rhythmic social-physical interaction might help foster human-robot teamwork, but the design of such interactions has scarcely been explored. We leveraged our prior work to enable the Rethink Robotics Baxter Research Robot to competently play one-handed tempo-matching hand-clapping games with a human user. To understand how such a robot’s capabilities and behaviors affect user perception, we created four versions of this interaction: the hand clapping could be initiated by either the robot or the human, and the non-initiating partner could be either cooperative, yielding synchronous motion, or mischievously uncooperative. Twenty adults tested two clapping tempos in each of these four interaction modes in a random order, rating every trial on standardized scales. The study results showed that having the robot initiate the interaction gave it a more dominant perceived personality. Despite previous results on the intrigue of misbehaving robots, we found that moving synchronously with the robot almost always made the interaction more enjoyable, less mentally taxing, less physically demanding, and lower effort for users than asynchronous interactions caused by robot or human mischief. Taken together, our results indicate that cooperative rhythmic social-physical interaction has the potential to strengthen human-robot partnerships.
DOI BibTeX
Haptic Intelligence Patent System and Method for Simultaneously Sensing Contact Force and Lateral Strain Lee, H., Kuchenbecker, K. J. (EP20000480.2), December 2020
A tactile sensing system having a sensor component which comprises a plurality of layers stacked along a normal axis Z and a detection unit electrically connected to the sensor component, wherein the sensor component comprises a first layer, designed as a piezoresistive layer, a third layer, designed as a conductive layer which is electrically connected to the detection unit, and a second layer, designed as a spacing layer between the first layer and the third layer, wherein the first layer comprises a plurality of electrodes In electrically connected to the detection unit, wherein at least one contact force along the normal axis Z on the sensor component is detectable by the detection unit due to a change of a current distribution between the first layer and the third layer, wherein at least one lateral strain on the sensor component is detectable by the detection unit due to a change of the resistance distribution change in the piezoresistive first layer.
BibTeX
Empirical Inference Conference Paper Worst-Case Risk Quantification under Distributional Ambiguity using Kernel Mean Embedding in Moment Problem Zhu, J., Jitkrittum, W., Diehl, M., Schölkopf, B. In 59th IEEE Conference on Decision and Control (CDC), 3457-3463, IEEE, December 2020 (Published) arXiv DOI BibTeX
Empirical Inference Conference Paper MATE: Plugging in Model Awareness to Task Embedding for Meta Learning Chen, X., Wang, Z., Tang, S., Muandet, K. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 11865-11877, (Editors: H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin), Curran Associates, Inc., 34th Annual Conference on Neural Information Processing Systems, December 2020 (Published) URL BibTeX
Dynamic Locomotion Article Postural stability in human running with step-down perturbations: an experimental and numerical study Drama, Ö., Vielemeyer, J., Badri-Spröwitz, A., Müller, R. Royal Society Open Science, 7(11):200570, November 2020 (Published)
Postural stability is one of the most crucial elements in bipedal locomotion. Bipeds are dynamically unstable and need to maintain their trunk upright against the rotations induced by the ground reaction forces (GRFs), especially when running. Gait studies report that the GRF vectors focus around a virtual point above the center of mass (VPA), while the trunk moves forward in pitch axis during the stance phase of human running. However, a recent simulation study suggests that a virtual point below the center of mass (VPB) might be present in human running, since a VPA yields backward trunk rotation during the stance phase. In this work, we perform a gait analysis to investigate the existence and location of the VP in human running at 5 m s−1, and support our findings numerically using the spring-loaded inverted pendulum model with a trunk (TSLIP). We extend our analysis to include perturbations in terrain height (visible and camouflaged), and investigate the response of the VP mechanism to step-down perturbations both experimentally and numerically. Our experimental results show that the human running gait displays a VPB of ≈ −30 cm and a forward trunk motion during the stance phase. The camouflaged step-down perturbations affect the location of the VPB. Our simulation results suggest that the VPB is able to encounter the step-down perturbations and bring the system back to its initial equilibrium state.
DOI URL BibTeX
Empirical Inference Conference Paper Action-Conditional Recurrent Kalman Networks For Forward and Inverse Dynamics Learning Shaj, V., Becker, P., Büchler, D., Pandya, H., van Duijkeren, N., Taylor, C. J., Hanheide, M., Neumann, G. Proceedings of the 4th Conference on Robot Learning (CoRL), 155:765-781, Proceedings of Machine Learning Research, (Editors: Jens Kober and Fabio Ramos and Claire J. Tomlin), PMLR, November 2020 (Published) PDF URL BibTeX
Empirical Inference Conference Paper Advances in Human-Robot Handshaking Prasad, V., Stock-Homburg, R., Peters, J. Social Robotics - 12th International Conference (ICSR), 12483:478-489, Lecture Notes in Computer Science, (Editors: Wager, A. R. and Feil-Seifer, D. and Haring, K. S. and Rossi, S. and Willians, T. and He, H. and Sam Ge, S.), Springer, November 2020 (Published) DOI BibTeX
Empirical Inference Ph.D. Thesis Enforcing and Discovering Structure in Machine Learning Locatello, F. ETH Zurich, Switzerland, November 2020, (CLS Fellowship Program) (Published) BibTeX
Perceiving Systems Empirical Inference Conference Paper Grasping Field: Learning Implicit Representations for Human Grasps Karunratanakul, K., Yang, J., Zhang, Y., Black, M., Muandet, K., Tang, S. In 2020 International Conference on 3D Vision (3DV 2020), 333-344, IEEE, Piscataway, NJ, International Conference on 3D Vision (3DV 2020), November 2020 (Published)
Robotic grasping of house-hold objects has made remarkable progress in recent years. Yet, human grasps are still difficult to synthesize realistically. There are several key reasons: (1) the human hand has many degrees of freedom (more than robotic manipulators); (2) the synthesized hand should conform to the surface of the object; and (3) it should interact with the object in a semantically and physically plausible manner. To make progress in this direction, we draw inspiration from the recent progress on learning-based implicit representations for 3D object reconstruction. Specifically, we propose an expressive representation for human grasp modelling that is efficient and easy to integrate with deep neural networks. Our insight is that every point in a three-dimensional space can be characterized by the signed distances to the surface of the hand and the object, respectively. Consequently, the hand, the object, and the contact area can be represented by implicit surfaces in a common space, in which the proximity between the hand and the object can be modelled explicitly. We name this 3D to 2D mapping as Grasping Field, parameterize it with a deep neural network, and learn it from data. We demonstrate that the proposed grasping field is an effective and expressive representation for human grasp generation. Specifically, our generative model is able to synthesize high-quality human grasps, given only on a 3D object point cloud. The extensive experiments demonstrate that our generative model compares favorably with a strong baseline and approaches the level of natural human grasps. Furthermore, based on the grasping field representation, we propose a deep network for the challenging task of 3D hand-object interaction reconstruction from a single RGB image. Our method improves the physical plausibility of the hand-object contact reconstruction and achieves comparable performance for 3D hand reconstruction compared to state-of-the-art methods. Our model and code are available for research purpose at https://github.com/korrawe/grasping_field.
pdf arXiv code DOI BibTeX
Empirical Inference Conference Paper High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewards Ploeger, K., Lutter, M., Peters, J. Proceedings of the 4th Conference on Robot Learning (CoRL), 155:642-653, Proceedings of Machine Learning Research, (Editors: Jens Kober and Fabio Ramos and Claire J. Tomlin), PMLR, November 2020 (Published) URL BibTeX
Autonomous Learning Haptic Intelligence Robotics Patent Method for Force Inference of a Sensor Arrangement, Methods for Training Networks, Force Inference Module and Sensor Arrangement Sun, H., Martius, G., Lee, H., Spiers, A., Fiene, J. (PCT/EP2020/083261), Max Planck Institute for Intelligent Systems, Max Planck Ring 4, November 2020
The present invention relates to a method for force inference of a sensor arrangement, to related methods for training of networks, to a force inference module for performing such methods, and to a sensor arrangement for sensing forces. When developing applications such as robots, sensing of forces applied on a robot hand or another part of a robot such as a leg or a manipulation device is crucial in giving robots increased capabilities to move around and/or manipulate objects. Known implementations for sensor arrangements that can be used in robotic applications in order to have feedback with regard to applied forces are quite expensive and do not have sufficient resolution. Sensor arrangements may be used to measure forces. However, known sensor arrangements need a high density of sensors to provide for a high special resolution. It is thus an object of the present invention to provide for a method for force inference of a sensor arrangement and related methods that are different or optimized with regard to the prior art. It is a further object to provide for a force inference module to perform such methods. It is a further object to provide for a sensor arrangement for sensing forces with such a force inference module.
BibTeX
Physics for Inference and Optimization Article Network extraction by routing optimization Baptista, T. D., Leite, D., Facca, E., Putti, M., De Bacco, C. Scientific Reports, 10:20806, November 2020 (Published)
Routing optimization is a relevant problem in many contexts. Solving directly this type of optimization problem is often computationally unfeasible. Recent studies suggest that one can instead turn this problem into one of solving a dynamical system of equations, which can instead be solved efficiently using numerical methods. This results in enabling the acquisition of optimal network topologies from a variety of routing problems. However, the actual extraction of the solution in terms of a final network topology relies on numerical details which can prevent an accurate investigation of their topological properties. In this context, theoretical results are fully accessible only to an expert audience and ready-to-use implementations for non-experts are rarely available or insufficiently documented. In particular, in this framework, final graph acquisition is a challenging problem in-and-of-itself. Here we introduce a method to extract networks topologies from dynamical equations related to routing optimization under various parameters’ settings. Our method is made of three steps: first, it extracts an optimal trajectory by solving a dynamical system, then it pre-extracts a network and finally, it filters out potential redundancies. Remarkably, we propose a principled model to address the filtering in the last step, and give a quantitative interpretation in terms of a transport-related cost function. This principled filtering can be applied to more general problems such as network extraction from images, thus going beyond the scenarios envisioned in the first step. Overall, this novel algorithm allows practitioners to easily extract optimal network topologies by combining basic tools from numerical methods, optimization and network theory. Thus, we provide an alternative to manual graph extraction which allows a grounded extraction from a large variety of optimal topologies.
Code Preprint DOI URL BibTeX
Perceiving Systems Article Occlusion Boundary: A Formal Definition & Its Detection via Deep Exploration of Context Wang, C., Fu, H., Tao, D., Black, M. J. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2641-2656, November 2020 (Published)
Occlusion boundaries contain rich perceptual information about the underlying scene structure and provide important cues in many visual perception-related tasks such as object recognition, segmentation, motion estimation, scene understanding, and autonomous navigation. However, there is no formal definition of occlusion boundaries in the literature, and state-of-the-art occlusion boundary detection is still suboptimal. With this in mind, in this paper we propose a formal definition of occlusion boundaries for related studies. Further, based on a novel idea, we develop two concrete approaches with different characteristics to detect occlusion boundaries in video sequences via enhanced exploration of contextual information (e.g., local structural boundary patterns, observations from surrounding regions, and temporal context) with deep models and conditional random fields. Experimental evaluations of our methods on two challenging occlusion boundary benchmarks (CMU and VSB100) demonstrate that our detectors significantly outperform the current state-of-the-art. Finally, we empirically assess the roles of several important components of the proposed detectors to validate the rationale behind these approaches.
official version DOI BibTeX
Autonomous Learning Haptic Intelligence Robotics Patent Sensor Arrangement for Sensing Forces and Method for Farbricating a Sensor Arrangement Spiers, A., Sun, H., Lee, H., Martius, G., Fiene, J., Seo, W. H. (PCT/EP2020/083260), November 2020 BibTeX
Empirical Inference Conference Paper Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis Xing, X., Jin, Z., Jin, D., Wang, B., Zhang, Q., Huang, X. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3594-3605, (Editors: Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu), Association for Computational Linguistics, Online, November 2020 (Published) PDF DOI URL BibTeX
Empirical Inference Autonomous Motion Movement Generation and Control Conference Paper TriFinger: An Open-Source Robot for Learning Dexterity Wüthrich, M., Widmaier, F., Grimminger, F., Akpo, J., Joshi, S., Agrawal, V., Hammoud, B., Khadiv, M., Bogdanovic, M., Berenz, V., Viereck, J., Naveau, M., Righetti, L., Schölkopf, B., Bauer, S. Proceedings of the 4th Conference on Robot Learning (CoRL), 155:1871-1882, Proceedings of Machine Learning Research, (Editors: Jens Kober and Fabio Ramos and Claire J. Tomlin), PMLR, November 2020 (Published) PDF URL BibTeX
Haptic Intelligence Miscellaneous Utilizing Interviews and Thematic Analysis to Uncover Specifications for a Companion Robot Burns, R. B., Seifi, H., Lee, H., Kuchenbecker, K. J. Workshop paper (2 pages) presented at the ICSR Workshop on Enriching HRI Research with Qualitative Methods, Virtual, November 2020 (Published)
We will share our experiences designing and conducting structured video-conferencing interviews with autism specialists and utilizing thematic analysis to create qualitative requirements and quantitative specifications for a touch-perceiving robot companion tailored for children with autism. We will also explain how we wrote about our qualitative approaches for a journal setting.
URL BibTeX
Perceiving Systems Conference Paper GIF: Generative Interpretable Faces Ghosh, P., Gupta, P. S., Uziel, R., Ranjan, A., Black, M. J., Bolkart, T. In 2020 International Conference on 3D Vision (3DV 2020), 1:868-878, IEEE, Piscataway, NJ, International Conference on 3D Vision (3DV 2020), November 2020 (Published)
Photo-realistic visualization and animation of expressive human faces have been a long standing challenge. 3D face modeling methods provide parametric control but generates unrealistic images, on the other hand, generative 2D models like GANs (Generative Adversarial Networks) output photo-realistic face images, but lack explicit control. Recent methods gain partial control, either by attempting to disentangle different factors in an unsupervised manner, or by adding control post hoc to a pre-trained model. Unconditional GANs, however, may entangle factors that are hard to undo later. We condition our generative model on pre-defined control parameters to encourage disentanglement in the generation process. Specifically, we condition StyleGAN2 on FLAME, a generative 3D face model. While conditioning on FLAME parameters yields unsatisfactory results, we find that conditioning on rendered FLAME geometry and photometric details works well. This gives us a generative 2D face model named GIF (Generative Interpretable Faces) that offers FLAME's parametric control. Here, interpretable refers to the semantic meaning of different parameters. Given FLAME parameters for shape, pose, expressions, parameters for appearance, lighting, and an additional style vector, GIF outputs photo-realistic face images. We perform an AMT based perceptual study to quantitatively and qualitatively evaluate how well GIF follows its conditioning. The code, data, and trained model are publicly available for research purposes at http://gif.is.tue.mpg.de
pdf project code video DOI BibTeX
Perceiving Systems Conference Paper PLACE: Proximity Learning of Articulation and Contact in 3D Environments Zhang, S., Zhang, Y., Ma, Q., Black, M. J., Tang, S. In 2020 International Conference on 3D Vision (3DV 2020), 1:642-651, IEEE, Piscataway, NJ, International Conference on 3D Vision (3DV 2020), November 2020 (Published)
High fidelity digital 3D environments have been proposed in recent years, however, it remains extremely challenging to automatically equip such environment with realistic human bodies. Existing work utilizes images, depth or semantic maps to represent the scene, and parametric human models to represent 3D bodies. While being straight-forward, their generated human-scene interactions often lack of naturalness and physical plausibility. Our key observation is that humans interact with the world through body-scene contact. To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world. To that end, we propose a novel interaction generation method, named PLACE(Proximity Learning of Articulation and Contact in 3D Environments), which explicitly models the proximity between the human body and the 3D scene around it. Specifically, given a set of basis points on a scene mesh, we leverage a conditional variational autoencoder to synthesize the minimum distances from the basis points to the human body surface. The generated proximal relationship exhibits which region of the scene is in contact with the person. Furthermore, based on such synthesized proximity, we are able to effectively obtain expressive 3D human bodies that interact with the 3D scene naturally. Our perceptual study shows that PLACE significantly improves the state-of-the-art method, approaching the realism of real human-scene interaction. We believe our method makes an important step towards the fully automatic synthesis of realistic 3D human bodies in 3D scenes. The code and model are available for research at https://sanweiliti.github.io/PLACE/PLACE.html
pdf arXiv project code DOI BibTeX
Dynamic Locomotion Article 3D Anatomy of the Quail Lumbosacral Spinal Canal—Implications for Putative Mechanosensory Function Kamska, V., Daley, M., Badri-Spröwitz, A. Integrative Organismal Biology, 2(1):obaa037, Oxford University Press, October 2020 (Published)
Birds are diverse and agile vertebrates capable of aerial, terrestrial, aquatic, and arboreal locomotion. Evidence suggests that birds possess a novel balance sensing organ in the lumbosacral spinal canal, a structure referred to as the “lumbosacral organ” (LSO), which may contribute to their locomotor agility and evolutionary success. The mechanosensing mechanism of this organ remains unclear. Here we quantify the 3D anatomy of the lumbosacral region of the common quail, focusing on establishing the geometric and biomechanical properties relevant to potential mechanosensing functions. We combine digital and classic dissection to create a 3D anatomical model of the quail LSO and estimate the capacity for displacement and deformation of the soft tissues. We observe a hammock-like network of denticulate ligaments supporting the lumbosacral spinal cord, with a close association between the accessory lobes and ligamentous intersections. The relatively dense glycogen body has the potential to apply loads sufficient to pre-stress denticulate ligaments, enabling external accelerations to excite tuned oscillations in the LSO soft tissue, leading to strain-based mechanosensing in the accessory lobe neurons. Considering these anatomical features together, the structure of the LSO is reminiscent of a mass-spring-based accelerometer.
3d model Youtube DOI BibTeX
Movement Generation and Control Conference Paper Enabling Remote Whole-Body Control with 5G Edge Computing Zhu, H., Sharma, M., Pfeiffer, K., Mezzavilla, M., Shen, J., Rangan, S., Righetti, L. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3553-3560, IEEE, Piscataway, NJ, USA, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 2020 (Accepted)
Real-world applications require light-weight, energy-efficient, fully autonomous robots. Yet, increasing auton- omy is oftentimes synonymous with escalating computational requirements. It might thus be desirable to offload intensive computation—not only sensing and planning, but also low- level whole-body control—to remote servers in order to reduce on-board computational needs. Fifth Generation (5G) wireless cellular technology, with its low latency and high bandwidth capabilities, has the potential to unlock cloud-based high per- formance control of complex robots. However, state-of-the-art control algorithms for legged robots can only tolerate very low control delays, which even ultra-low latency 5G edge computing can sometimes fail to achieve. In this work, we investigate the problem of cloud-based whole-body control of legged robots over a 5G link. We propose a novel approach that consists of a standard optimization-based controller on the network edge and a local linear, approximately optimal controller that significantly reduces on-board computational needs while increasing robustness to delay and possible loss of commu- nication. Simulation experiments on humanoid balancing and walking tasks that includes a realistic 5G communication model demonstrate significant improvement of the reliability of robot locomotion under jitter and delays likely to be experienced in 5G wireless links.
DOI URL BibTeX
Dynamic Locomotion Conference Paper Simulating the response of a neuro-musculoskeletal model to assistive forces: implications for the design of wearables compensating for motor control deficits Stollenmaier, K., Rist, I., Izzi, F., Haeufle, D. F. In 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob 2020), 779-784, IEEE, Piscataway, NJ, 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob 2020), October 2020 (Published)
Models of the human arm may help to estimate design parameters like peak torque and power of wearable assistive devices by predicting required forces to compensate for motor control impairments. This work focuses on the idea of compensating hypermetria (overshoot)-a motor control deficit that may occur in neurodegenerative diseases-by a simple assistive device. As musculoskeletal dynamics play an important role in the interaction between an assistive device and the neuro-musculoskeletal system, we hypothesized that their consideration in the model might influence the predicted design parameters. To test this, we simulated two-degree-of-freedom point-to-point arm movements. By introducing inconsistent neuronal control parameters, we induced hypermetria. We implemented mechanical and low-level assistive torque strategies in simulation which lead to a reduction of hypermetria. We found that-depending on the type of assistance-the predicted torques and powers can differ by more than a factor of 10 between musculoskeletal and torque-driven arm models. We conclude that the magnitude of torque and power required to reduce hypermetria by simple wearable assistive devices may be significantly underestimated if muscle-tendon characteristics are not considered.
DOI BibTeX
Perceiving Systems Article 3D Morphable Face Models - Past, Present and Future Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., Theobalt, C., Blanz, V., Vetter, T. ACM Transactions on Graphics, 39(5):157, October 2020 (Published)
In this paper, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed. The challenges in building and applying these models, namely capture, modeling, image formation, and image analysis, are still active research topics, and we review the state-of-the-art in each of these areas. We also look ahead, identifying unsolved challenges, proposing directions for future research and highlighting the broad range of current and future applications.
project page pdf preprint DOI BibTeX
Empirical Inference Master Thesis A Comprehensive Benchmark Evaluation of Synthetic Data Generation for Biomedical Imaging DuMont Schütte, A. ETH Zurich, Switzerland, October 2020 (Published) BibTeX
Dynamic Locomotion Intelligent Control Systems Article A Learnable Safety Measure Heim, S., Rohr, A. V., Trimpe, S., Badri-Spröwitz, A. Proceedings of the Conference on Robot Learning, 100:627-639, Proceedings of Machine Learning Research, (Editors: Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei), PMLR, Conference on Robot Learning, October 2020 (Published) Arxiv BibTeX
Empirical Inference Master Thesis A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning Ahmed, O. ETH Zurich, Switzerland, October 2020 (Published) BibTeX
Perceiving Systems Article AirCapRL: Autonomous Aerial Human Motion Capture Using Deep Reinforcement Learning Tallamraju, R., Saini, N., Bonetto, E., Pabst, M., Liu, Y. T., Black, M., Ahmad, A. IEEE Robotics and Automation Letters, 5(4):6678-6685, IEEE, October 2020, Also accepted and presented in the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (Published)
In this letter, we introduce a deep reinforcement learning (DRL) based multi-robot formation controller for the task of autonomous aerial human motion capture (MoCap). We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose, and shape of a single moving person using multiple micro aerial vehicles. State-of-the-art solutions to this problem are based on classical control methods, which depend on hand-crafted system, and observation models. Such models are difficult to derive, and generalize across different systems. Moreover, the non-linearities, and non-convexities of these models lead to sub-optimal controls. In our work, we formulate this problem as a sequential decision making task to achieve the vision-based motion capture objectives, and solve it using a deep neural network-based RL method. We leverage proximal policy optimization (PPO) to train a stochastic decentralized control policy for formation control. The neural network is trained in a parallelized setup in synthetic environments. We performed extensive simulation experiments to validate our approach. Finally, real-robot experiments demonstrate that our policies generalize to real world conditions.
DOI URL BibTeX