Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation


Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group


Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award


Empirical Inference Article Self-supervised contrastive learning with random walks for medical image segmentation with limited annotations Fischer, M., Hepp, T., Gatidis, S., Yang, B. Computerized Medical Imaging and Graphics, 104, Elsevier, Amsterdam, March 2023 (Published) DOI BibTeX

Empirical Inference Article Compact holographic sound fields enable rapid one-step assembly of matter in 3D Melde, K., Kremer, H., Shi, M., Seneca, S., Frey, C., Platzman, I., Degel, C., Schmitt, D., Schölkopf, B., Fischer, P. Science Advances, 9(6), AAAS, Washington, DC, February 2023 (Published) DOI URL BibTeX

Perceiving Systems Ph.D. Thesis Reconstruction and Synthesis of Human-Scene Interaction Hassan, M. University of Tübingen, February 2023 (Published)
In this thesis, we argue that the 3D scene is vital for understanding, reconstructing, and synthesizing human motion. We present several approaches which take the scene into consideration in reconstructing and synthesizing Human-Scene Interaction (HSI). We first observe that state-of-the-art pose estimation methods ignore the 3D scene and hence reconstruct poses that are inconsistent with the scene. We address this by proposing a pose estimation method that takes the 3D scene explicitly into account. We call our method PROX for Proximal Relationships with Object eXclusion. We leverage the data generated using PROX and build a method to automatically place 3D scans of people with clothing in scenes. The core novelty of our method is encoding the proximal relationships between the human and the scene in a novel HSI model, called POSA for Pose with prOximitieS and contActs. POSA is limited to static HSI, however. We propose a real-time method for synthesizing dynamic HSI, which we call SAMP for Scene-Aware Motion Prediction. SAMP enables virtual humans to navigate cluttered indoor scenes and naturally interact with objects. Data-driven kinematic models, like SAMP, can produce high-quality motion when applied in environments similar to those shown in the dataset. However, when applied to new scenarios, kinematic models can struggle to generate realistic behaviors that respect scene constraints. In contrast, we present InterPhys which uses adversarial imitation learning and reinforcement learning to train physically-simulated characters that perform scene interaction tasks in a physical and life-like manner.
download pdf DOI BibTeX

Empirical Inference Article Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets Choe, J., Oh, S. J., Chun, S., Lee, S., Akata, Z., Shim, H. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1732-1748, IEEE, New York, NY, February 2023 (Published) DOI URL BibTeX

Materials Article Dry Synthesis of Pure and Ultrathin Nanoporous Metallic Films Kwon, H., Barad, H., Olaya, A. R. S., Alarcón-Correa, M., Hahn, K., Richter, G., Wittstock, G., Fischer, P. ACS Applied Materials and Interfaces, 15(4):5620-5627, February 2023 (Published) DOI BibTeX

Empirical Inference Article GRASP: Scalable Graph Alignment by Spectral Corresponding Functions Hermanns, J., Skitsas, K., Tsitsulin, A., Munkhoeva, M., Kyster, A., Nielsen, S., Bronstein, A. M., Mottin, D., Karras, P. ACM Transactions on Knowledge Discovery from Data, 17(4), February 2023 (Published) DOI BibTeX

Social Foundations of Computation Conference Paper Human-Guided Fair Classification for Natural Language Processing Dorner, F. E., Peychev, M., Konstantinov, N., Goel, N., Ash, E., Vechev, M. In The Eleventh International Conference on Learning Representations (ICLR 2023), February 2023 (Published)
Text classifiers have promising applications in high-stake tasks such as resume screening and content moderation. These classifiers must be fair and avoid discriminatory decisions by being invariant to perturbations of sensitive attributes such as gender or ethnicity. However, there is a gap between human intuition about these perturbations and the formal similarity specifications capturing them. While existing research has started to address this gap, current methods are based on hardcoded word replacements, resulting in specifications with limited expressivity or ones that fail to fully align with human intuition (e.g., in cases of asymmetric counterfactuals). This work proposes novel methods for bridging this gap by discovering expressive and intuitive individual fairness specifications. We show how to leverage unsupervised style transfer and GPT-3's zero-shot capabilities to automatically generate expressive candidate pairs of semantically similar sentences that differ along sensitive attributes. We then validate the generated pairs via an extensive crowdsourcing study, which confirms that a lot of these pairs align with human intuition about fairness in the context of toxicity classification. Finally, we show how limited amounts of human feedback can be leveraged to learn a similarity specification that can be used to train downstream fairness-aware models.
arXiv URL BibTeX

Dynamic Locomotion Article Slack-based tunable damping leads to a trade-off between robustness and efficiency in legged locomotion Mo, A., Izzi, F., Gönen, E. C., Häufle, D., Badri-Spröwitz, A. Scientific Reports, 13:3290, Nature Publishing Group, February 2023 (Published)
Animals run robustly in diverse terrain. This locomotion robustness is puzzling because axon conduction velocity is limited to a few ten meters per second. If reflex loops deliver sensory information with significant delays, one would expect a destabilizing effect on sensorimotor control. Hence, an alternative explanation describes a hierarchical structure of low-level adaptive mechanics and high-level sensorimotor control to help mitigate the effects of transmission delays. Motivated by the concept of an adaptive mechanism triggering an immediate response, we developed a tunable physical damper system. Our mechanism combines a tendon with adjustable slackness connected to a physical damper. The slack damper allows adjustment of damping force, onset timing, effective stroke, and energy dissipation. We characterize the slack damper mechanism mounted to a legged robot controlled in open-loop mode. The robot hops vertically and planar over varying terrains and perturbations. During forward hopping, slack-based damping improves faster perturbation recovery (up to 170\%) at higher energetic cost (27\%). The tunable slack mechanism auto-engages the damper during perturbations, leading to a perturbation-trigger damping, improving robustness at minimum energetic cost. With the results from the slack damper mechanism, we propose a new functional interpretation of animals' redundant muscle tendons as tunable dampers.
arxiv Video Journal URL CAD and data DOI URL BibTeX

Empirical Inference Article SphereFace Revived: Unifying Hyperspherical Face Recognition Liu, W., Wen, Y., Raj, B., Singh, R., Weller, A. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2458-2474, February 2023 (Published) DOI BibTeX

Haptic Intelligence Article The S-BAN: Insights into the Perception of Shape-Changing Haptic Interfaces via Virtual Pedestrian Navigation Spiers, A. J., Young, E., Kuchenbecker, K. J. ACM Transactions on Computer-Human Interaction, 30(1):1-31, February 2023 (Published)
Screen-based pedestrian navigation assistance can be distracting or inaccessible to users. Shape-changing haptic interfaces can overcome these concerns. The S-BAN is a new handheld haptic interface that utilizes a parallel kinematic structure to deliver 2-DOF spatial information over a continuous workspace, with a form factor suited to integration with other travel aids. The ability to pivot, extend and retract its body opens possibilities and questions around spatial data representation. We present a static study to understand user perception of absolute pose and relative motion for two spatial mappings, showing highest sensitivity to relative motions in the cardinal directions. We then present an embodied navigation experiment in virtual reality. User motion efficiency when guided by the S-BAN was statistically equivalent to using a vision-based tool (a smartphone proxy). Although haptic trials were slower than visual trials, participants' heads were more elevated with the S-BAN, allowing greater visual focus on the environment.
DOI BibTeX

Empirical Inference Conference Paper Towards Empirical Process Theory for Vector-Valued Functions: Metric Entropy of Smooth Function Classes Park, J., Muandet, K. Proceedings of the 34th International Conference on Algorithmic Learning Theory (ALT), 201:1216-1260, Proceedings of Machine Learning Research, (Editors: Agrawal, Shipra and Orabona, Francesco), PMLR, February 2023 (Published) URL BibTeX

Empirical Inference Master Thesis Towards Generative Machine Teaching Qui, Z. Technical University of Munich, Germany, February 2023 (Published) BibTeX

Empirical Inference Article ViViT: Curvature Access Through The Generalized Gauss-Newton’s Low-Rank Structure Dangel*, F., Tatzel*, L., Hennig, P. Transactions on Machine Learning Research, February 2023, *equal contribution (Published) URL BibTeX

Social Foundations of Computation Conference Paper What Makes ImageNet Look Unlike LAION Shirali, A., Hardt, M. The Twelfth International Conference on Learning Representations (ICLR 2024), February 2023 (Submitted)
ImageNet was famously created from Flickr image search results. What if we recreated ImageNet instead by searching the massive LAION dataset based on image captions alone? In this work, we carry out this counterfactual investigation. We find that the resulting ImageNet recreation, which we call LAIONet, looks distinctly unlike the original. Specifically, the intra-class similarity of images in the original ImageNet is dramatically higher than it is for LAIONet. Consequently, models trained on ImageNet perform significantly worse on LAIONet. We propose a rigorous explanation for the discrepancy in terms of a subtle, yet important, difference in two plausible causal data-generating processes for the respective datasets, that we support with systematic experimentation. In a nutshell, searching based on an image caption alone creates an information bottleneck that mitigates the selection bias otherwise present in image-based filtering. Our explanation formalizes a long-held intuition in the community that ImageNet images are stereotypical, unnatural, and overly simple representations of the class category. At the same time, it provides a simple and actionable takeaway for future dataset creation efforts.
arXiv URL BibTeX

Perceiving Systems MPI Year Book Virtuelle Körper ermöglichen Blick in eine gesunde Zukunft / Virtual bodies provide a glimpse into a healthy future Behrens, S. C., Tesch, J., Sun, P. J. B., Starke, S., Black, M. J., Schneider, H., Pruccoli, J., Zipfel, S., Giel, K. January 2023 (Published)
Menschen mit Magersucht leben in ständiger Angst, zuzunehmen. Betroffene tun oft alles dafür, um keinesfalls mehr zu wiegen – selbst wenn sie bereits unter den gesundheitlichen Folgen leiden oder im Alltag eingeschränkt sind. Forschende des Max-Planck-Instituts für Intelligente Systeme und des Universitätsklinikums Tübingen haben ein Virtual-Reality-Tool entwickelt, mit dem sich eine Gewichtszunahme simulieren lässt. Die Forschungsergebnisse deuten darauf hin, dass die wiederholte Auseinandersetzung mit virtuellem gesundem Gewicht Personen mit Magersucht hilft, ihre Angst vor einer Gewichtzunahme zu reduzieren. ENGLISH: People with anorexia live in constant fear of gaining weight. They often do everything in their power to avoid putting on weight, even if they are already suffering from the health consequences or are restricted in their daily lives. Computer Vision researchers at the Max Planck Institute for Intelligent Systems and the University Hospital Tübingen have developed a virtual reality tool that can be used to simulate weight gain. The research suggests that repeated exposure to a healthy bodyweight in virtual reality helps people with anorexia nervosa reduce their fear of gaining weight. The full text can be found in German on the website of the Max Planck Society.
URL BibTeX

Empirical Inference Article A machine learning route between band mapping and band structure Xian*, R. P., Stimper*, V., Zacharias, M., Dendzik, M., Dong, S., Beaulieu, S., Schölkopf, B., Wolf, M., Rettig, L., Carbogno, C., Bauer, S., Ernstorfer, R. Nature Computational Science, 3(1):101-114, January 2023, *equal contribution (Published) arXiv DOI BibTeX

Empirical Inference Master Thesis ArchiSound: Audio Generation with Diffusion Schneider, F. ETH Zurich, Switzerland, January 2023, external supervision (Published) BibTeX

Empirical Inference Article Audio Retrieval With Natural Language Queries: A Benchmark Study Koepke, A. S., Oncescu, A., Henriques, J. F., Akata, Z., Albanie, S. IEEE Transactions on Multimedia, 25:2675-2685, January 2023 (Published) DOI BibTeX

Rationality Enhancement Article Automatic discovery and description of human planning strategies Skirzynski, J., Jain, Y. R., Lieder, F. Behavior Research Methods, January 2023 (Published)
Scientific discovery concerns finding patterns in data and creating insightful hypotheses that explain these patterns. Traditionally, each step of this process required human ingenuity. But the galloping development of computer chips and advances in artificial intelligence (AI) make it increasingly more feasible to automate some parts of scientific discovery. Understanding human planning is one of the fields in which AI has not yet been utilized. State-of-the-art methods for discovering new planning strategies still rely on manual data analysis. Data about the process of human planning is often used to group similar behaviors together. Researchers then use this data to formulate verbal descriptions of the strategies which might underlie those groups of behaviors. In this work we leverage AI to automate these two steps of scientific discovery. We introduce a method for the automatic discovery and description of human planning strategies from process-tracing data collected with the Mouselab-MDP paradigm. Our algorithm, called Human-Interpret, uses imitation learning to describe data gathered in the experiment in terms of a procedural formula and then translates that formula to natural language using a pre-defined predicate dictionary. We test our method on a benchmark data set that researchers have previously scrutinized manually. We find that the descriptions of human planning strategies that we obtain automatically are about as understandable as human-generated descriptions. They also cover a substantial proportion of all types of human planning strategies that had been discovered manually. Our method saves scientists' time and effort as all the reasoning about human planning is done automatically. This might make it feasible to more rapidly scale up the search for yet undiscovered cognitive strategies that people use for planning and decision-making to many new decision environments, populations, tasks, and domains. Given these results, we believe that the presented work may accelerate scientific discovery in psychology, and due to its generality, extend to problems from other fields.
DOI URL BibTeX

Empirical Inference Article Learning Dynamical Systems using Local Stability Priors Mehrjou, A., Iannelli, A., Schölkopf, B. Journal of Computational Dynamics, 10(1):175-198, January 2023, Special issue "Computation of Lyapunov functions and contraction metrics" (Published) DOI BibTeX

Empirical Inference Article Learning to Control Highly Accelerated Ballistic Movements on Muscular Robots Büchler, D., Calandra, R., Peters, J. Robotics and Autonomous Systems, 159, Elsevier, Amsterdam, January 2023 (Published)
High-speed and high-acceleration movements are inherently hard to control. Applying learning to the control of such motions on anthropomorphic robot arms can improve the accuracy of the control but might damage the system. The inherent exploration of learning approaches can lead to instabilities and the robot reaching joint limits at high speeds. Having hardware that enables safe exploration of high-speed and high-acceleration movements is therefore desirable. To address this issue, we propose to use robots actuated by Pneumatic Artificial Muscles (PAMs). In this paper, we present a four degrees of freedom (DoFs) robot arm that reaches high joint angle accelerations of up to 28000 °/s^2 while avoiding dangerous joint limits thanks to the antagonistic actuation and limits on the air pressure ranges. With this robot arm, we are able to tune control parameters using Bayesian optimization directly on the hardware without additional safety considerations. The achieved tracking performance on a fast trajectory exceeds previous results on comparable PAM-driven robots. We also show that our system can be controlled well on slow trajectories with PID controllers due to careful construction considerations such as minimal bending of cables, lightweight kinematics and minimal contact between PAMs and PAMs with the links. Finally, we propose a novel technique to control the the co-contraction of antagonistic muscle pairs. Experimental results illustrate that choosing the optimal co-contraction level is vital to reach better tracking performance. Through the use of PAM-driven robots and learning, we do a small step towards the future development of robots capable of more human-like motions.
Arxiv Video DOI URL BibTeX

Materials Article Nanoscale bimetallic strip: Atomistic bending mechanisms of AuPd bimetallic nanowhiskers Bisht, A., Kalina, M., Suadiye, E., Richter, G., Rabkin, E. Acta Materialia, 243:118504, Elsevier Science, Kidlington, January 2023 (Published) DOI BibTeX

Autonomous Learning Haptic Intelligence Empirical Inference Article Predicting the Force Map of an ERT-Based Tactile Sensor Using Simulation and Deep Networks Lee, H., Sun, H., Park, H., Serhat, G., Javot, B., Martius, G., Kuchenbecker, K. J. IEEE Transactions on Automation Science and Engineering, 20(1):425-439, January 2023 (Published)
Electrical resistance tomography (ERT) can be used to create large-scale soft tactile sensors that are flexible and robust. Good performance requires a fast and accurate mapping from the sensor's sequential voltage measurements to the distribution of force across its surface. However, particularly with multiple contacts, this task is challenging for both previously developed approaches: physics-based modeling and end-to-end data-driven learning. Some promising results were recently achieved using sim-to-real transfer learning, but estimating multiple contact locations and accurate contact forces remains difficult because simulations tend to be less accurate with a high number of contact locations and/or high force. This paper introduces a modular hybrid method that combines simulation data synthesized from an electromechanical finite element model with real measurements collected from a new ERT-based tactile sensor. We use about 290,000 simulated and 90,000 real measurements to train two deep neural networks: the first (Transfer-Net) captures the inevitable gap between simulation and reality, and the second (Recon-Net) reconstructs contact forces from voltage measurements. The number of contacts, contact locations, force magnitudes, and contact diameters are evaluated for a manually collected multi-contact dataset of 150 measurements. Our modular pipeline's results outperform predictions by both a physics-based model and end-to-end learning.
DOI BibTeX

Empirical Inference Article Pyfectious: An individual-level simulator to discover optimal containment policies for epidemic diseases Mehrjou*, A., Soleymani*, A., Abyaneh, A., Bhatt, S., Schölkopf, B., Bauer, S. PLOS Computational Biology, 19(1):1-41, January 2023, *equal contribution (Published) DOI BibTeX

Empirical Inference Article Quantum machine learning beyond kernel methods Jerbi, S., Fiderer, L. J., Poulsen Nautrup, H., Kübler, J. M., Briegel, H. J., Dunjko, V. Nature Communications, 14(1), January 2023 (Published) DOI BibTeX

Software Workshop Article Resonant Kushi-comb-like multi-frequency radiation of oscillating two-color soliton molecules Melchert, O., Willms, S., Oreshnikov, I., Yulin, A., Morgner, U., Babushkin, I., Demircan, A. New Journal of Physics, 25, January 2023 (Published) DOI URL BibTeX

Haptic Intelligence Article The Utility of Synthetic Reflexes and Haptic Feedback for Upper-Limb Prostheses in a Dexterous Task Without Direct Vision Thomas, N., Fazlollahi, F., Kuchenbecker, K. J., Brown, J. D. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:169-179, January 2023 (Published)
Individuals who use myoelectric upper-limb prostheses often rely heavily on vision to complete their daily activities. They thus struggle in situations where vision is overloaded, such as multitasking, or unavailable, such as poor lighting conditions. Able-bodied individuals can easily accomplish such tasks due to tactile reflexes and haptic sensation guiding their upper-limb motor coordination. Based on these principles, we developed and tested two novel prosthesis systems that incorporate autonomous controllers and provide the user with touch-location feedback through either vibration or distributed pressure. These capabilities were made possible by installing a custom contact-location sensor on the fingers of a commercial prosthetic hand, along with a custom pressure sensor on the thumb. We compared the performance of the two systems against a standard myoelectric prosthesis and a myoelectric prosthesis with only autonomous controllers in a difficult reach-to-pick-and-place task conducted without direct vision. Results from 40 able-bodied participants in this between-subjects study indicated that vibrotactile feedback combined with synthetic reflexes proved significantly more advantageous than the standard prosthesis in several of the task milestones. In addition, vibrotactile feedback and synthetic reflexes improved grasp placement compared to only synthetic reflexes or pressure feedback combined with synthetic reflexes. These results indicate that autonomous controllers and haptic feedback together facilitate success in dexterous tasks without vision, and that the type of haptic display matters.
DOI BibTeX

Modern Magnetic Systems Article 20 K H2 physisorption on metal−organic frameworks with enhanced dormancy compared to liquid hydrogen storage Park, J., Ha, J., Muhammad, R., Lee, H. K., Balderas-Xicohtencatl, R., Cheng, Y., Ramirez-Cuesta, A. J., Streppel, B., Hirscher, M., Moon, H. R., Oh, H. ACS Applied Energy Materials, 6(18):9057-9064, American Chemical Society, Washington, DC, 2023 (Published) DOI BibTeX

Empirical Inference Article A Deterministic Approximation to Neural SDEs Look, A., Kandemir, M., Rakitsch, B., Peters, J. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4023-4037, 2023 (Published) DOI BibTeX

Learning and Dynamical Systems Conference Paper A Dynamical Systems Perspective on Discrete Optimization Tong, G., Muehlebach, M. In Conference on Learning for Dynamics and Control, 211:1373-1386, PMLR, 2023 (Published) URL BibTeX

Robotic Materials Article A Soft, Fast and Versatile Electrohydraulic Gripper with Capacitive Object Size Detection Yoder, Z., Macari, D., Kleinwaks, G., Schmidt, I., Acome, E., Keplinger, C. Advanced Functional Materials, 23(3):2209080, 2023 (Published)
Soft robotic grippers achieve increased versatility and reduced complexity through intelligence embodied in their flexible and conformal structures. The most widely used soft grippers are pneumatically driven; they are simple and effective but require bulky air compressors that limit their application space and external sensors or computationally expensive vision systems for pick verification. In this study, a multi-material architecture for self-sensing electrohydraulic bending actuators is presented that enables a new class of highly versatile and reconfigurable soft grippers that are electrically driven and feature capacitive pick verification and object size detection. These elec-trohydraulic grippers are fast (step input results in finger closure in 50 ms), draw low power (6.5 mW per finger to hold grasp), and can pick a wide variety of objects with simple binary electrical control. Integrated high-voltage driving electronics are presented that greatly increase the application space of the grippers and make them readily compatible with commercially available robotic arms.
A Soft, Fast and Versatile Electrohydraulic Gripper with Capacitive Object Size Detection YouTube video DOI BibTeX

Physical Intelligence Article A ferroelectric/ferroelastic energy harvester: Load impedance and frequency effects Kang, W., Cain, C., Paynter, R., Huber, J. E. Energy Conversion and Management, 277, 2023 (Published) DOI URL BibTeX

Modern Magnetic Systems Article A squarate-pillared titanium oxide quantum sieve towards practical hydrogen isotope separation Yan, Q., Wang, J., Zhang, L., Liu, J., Wahiduzzaman, M., Yan, N., Yu, L., Dupuis, R., Wang, H., Maurin, G., Hirscher, M., Guo, P., Wang, S., Du, J. Nature Communications, 14:4189, Nature Publishing Group, London, 2023 (Published) DOI BibTeX

Empirical Inference Article Altered brain dynamics index levels of arousal in complete locked-in syndrome Zilio, F., Gomez-Pilar, J., Chaudhary, U., Fogel, S., Fomina, T., Synofzik, M., Schöls, L., Cao, S., Zhang, J., Huang, Z., Birbaumer, N., Northoff, G. Communications Biology, 6(1), 2023 (Published) DOI BibTeX

Dynamic Locomotion Unpublished An Open-Source Modular Treadmill for Dynamic Force Measurement with Load Dependant Range Adjustment Sarvestani, A., Ruppert, F., Badri-Spröwitz, A. 2023 (Submitted)
Ground reaction force sensing is one of the key components of gait analysis in legged locomotion research. To measure continuous force data during locomotion, we present a novel compound instrumented treadmill design. The treadmill is 1.7 m long, with a natural frequency of 170 Hz and an adjustable range that can be used for humans and small robots alike. Here, we present the treadmill’s design methodology and characterize it in its natural frequency, noise behavior and real-life performance. Additionally, we apply an ISO 376 norm conform calibration procedure for all spatial force directions and center of pressure position. We achieve a force accuracy of ≤ 5.6 N for the ground reaction forces and ≤ 13 mm in center of pressure position.
arXiv DOI URL BibTeX

Materials Article Atomic-resolution observations of silver segregation in a [111] tilt grain boundary in copper Langeohl, L., Brink, T., Richter, G., Dehm, G., Liebscher, C. H. Physical Review B, 107(13):134112, American Physical Society, Woodbury, NY, 2023 (Published) DOI BibTeX