Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation


Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group


Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award


Embodied Vision Conference Paper Context-Conditional Navigation with a Learning-Based Terrain- and Robot-Aware Dynamics Model Guttikonda, S., Achterhold, J., Li, H., Boedecker, J., Stueckler, J. In Proceedings of the European Conference on Mobile Robots (ECMR), 2023 (Published)
In autonomous navigation settings, several quantities can be subject to variations. Terrain properties such as friction coefficients may vary over time depending on the location of the robot. Also, the dynamics of the robot may change due to, e.g., different payloads, changing the system's mass, or wear and tear, changing actuator gains or joint friction. An autonomous agent should thus be able to adapt to such variations. In this paper, we develop a novel probabilistic, terrain- and robot-aware forward dynamics model, termed TRADYN, which is able to adapt to the above-mentioned variations. It builds on recent advances in meta-learning forward dynamics models based on Neural Processes. We evaluate our method in a simulated 2D navigation setting with a unicycle-like robot and different terrain layouts with spatially varying friction coefficients. In our experiments, the proposed model exhibits lower prediction error for the task of long-horizon trajectory prediction, compared to non-adaptive ablation models. We also evaluate our model on the downstream task of navigation planning, which demonstrates improved performance in planning control-efficient paths by taking robot and terrain properties into account.
preprint code DOI URL BibTeX

Neural Capture and Synthesis Conference Paper DINER: Depth-aware Image-based Neural Radiance Fields Prinzler, M., Hilliges, O., Thies, J. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), CVPR, 2023 (Accepted)
We present Depth-aware Image-based NEural Radiance fields (DINER). Given a sparse set of RGB input views, we predict depth and feature maps to guide the reconstruction of a volumetric scene representation that allows us to render 3D objects under novel views. Specifically, we propose novel techniques to incorporate depth information into feature fusion and efficient scene sampling. In comparison to the previous state of the art, DINER achieves higher synthesis quality and can process input views with greater disparity. This allows us to capture scenes more completely without changing capturing hardware requirements and ultimately enables larger viewpoint changes during novel view synthesis. We evaluate our method by synthesizing novel views, both for human heads and for general objects, and observe significantly improved qualitative results and increased perceptual metrics compared to the previous state of the art. The code is publicly available for research purposes.
Video Code Arxiv URL BibTeX

Physical Intelligence Article Deployable Soft Origami Modular Robotic Arm With Variable Stiffness Using Facet Buckling Park, M., Kim, W., Yu, S., Cho, J., Kang, W., Byun, J., Jeong, U., Cho, K. IEEE Robotics and Automation Letters, 8(2):864-871, 2023 (Published) DOI BibTeX

Physical Intelligence Article Designing Covalent Organic Framework-based Light-driven Microswimmers towards Intraocular Theranostic Applications Sridhar, V., Yildiz, E., Rodrı́guez-Camargo, A., Lyu, X., Yao, L., Wrede, P., Aghakhani, A., Akolpoglu, M. B., Podjaski, F., Lotsch, B. V., Sitti, M. Advanced Materials, 35(25), 2023 (Published)
While micromachines with tailored functionalities enable therapeutic applications in biological environments, their controlled motion and targeted drug delivery in biological media require sophisticated designs for practical applications. Covalent organic frameworks (COFs), a new generation of crystalline and nanoporous polymers, offer new perspectives for light-driven microswimmers in heterogeneous biological environments including intraocular fluids, thus setting the stage for biomedical applications such as retinal drug delivery. Two different types of COFs, uniformly spherical TABP-PDA-COF sub-micrometer particles and texturally nanoporous, micrometer-sized TpAzo-COF particles are described and compared as light-driven microrobots. They can be used as highly efficient visible-light-driven drug carriers in aqueous ionic and cellular media. Their absorption ranging down to red light enables phototaxis even in deeper and viscous biological media, while the organic nature of COFs ensures their biocompatibility. Their inherently porous structures with ≈2.6 and ≈3.4 nm pores, and large surface areas allow for targeted and efficient drug loading even for insoluble drugs, which can be released on demand. Additionally, indocyanine green (ICG) dye loading in the pores enables photoacoustic imaging, optical coherence tomography, and hyperthermia in operando conditions. This real-time visualization of the drug-loaded COF microswimmers enables unique insights into the action of photoactive porous drug carriers for therapeutic applications.
DOI BibTeX

Modern Magnetic Systems Article Direct observation of Néel-type skyrmions and domain walls in a ferrimagnetic DyCo3 thin film Luo, C., Chen, K., Ukleev, V., Wintz, S., Weigand, M., Abrudan, R., Prokes, K., Radu, F. Communications Physics, 6:2018, Nature Publishing Group, London, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Article Direct observation of propagating spin waves in the 2D van der Waals ferromagnet Fe5GeTe2 Schulz, F., Litzius, K., Powalla, L., Birch, M. T., Gallardo, R. A., Satheesh, S., Weigand, M., Scholz, T., Lotsch, B. V., Schütz, G., Burghard, M., Wintz, S. Nano Letters, 23(22):10126-10131, American Chemical Society, Washington, DC, 2023 DOI BibTeX

Autonomous Learning Article Discovering causal relations and equations from data Camps-Valls, G., Gerhardus, A., Ninad, U., Varando, G., Martius, G., Balaguer-Ballester, E., Vinuesa, R., Diaz, E., Zanna, L., Runge, J. Physics Reports, 1044:1-68, 2023
Physics is a field of science that has traditionally used the scientific method to answer questions about why natural phenomena occur and to make testable models that explain the phenomena. Discovering equations, laws, and principles that are invariant, robust, and causal has been fundamental in physical sciences throughout the centuries. Discoveries emerge from observing the world and, when possible, performing interventions on the system under study. With the advent of big data and data-driven methods, the fields of causal and equation discovery have developed and accelerated progress in computer science, physics, statistics, philosophy, and many applied fields. This paper reviews the concepts, methods, and relevant works on causal and equation discovery in the broad field of physics and outlines the most important challenges and promising future lines of research. We also provide a taxonomy for data-driven causal and equation discovery, point out connections, and showcase comprehensive case studies in Earth and climate sciences, fluid dynamics and mechanics, and the neurosciences. This review demonstrates that discovering fundamental laws and causal relations by observing natural phenomena is revolutionised with the efficient exploitation of observational data and simulations, modern machine learning algorithms and the combination with domain knowledge. Exciting times are ahead with many challenges and opportunities to improve our understanding of complex systems.
DOI BibTeX

Physical Intelligence Patent Dry adhesives and methods for making dry adhesives M Sitti, M. M. B. A. 2023, US Patent 11,773,298, 2023 BibTeX

Modern Magnetic Systems Article Ferromagnetic order controlled by the magnetic interface of LaNiO3/ La2/3Ca1/3MnO3 superlattices Soltan, S., Macke, S., Ilse, S. E., Pennycook, T., Zhang, Z. L., Christiani, G., Benckiser, E., Schütz, G., Goering, E. Scientific Reports, 13:3847, Nature Publishing Group, London, UK, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Article Imaging of short-wavelength spin waves in a nanometer-thick YIG/Co bilayer Talapatra, A., Qin, H., Schulz, F., Yao, L., Flajsman, L., Weigand, M., Wintz, S., van Dijken, S. Applied Physics Letters, 122(20):202404, American Institute of Physics, Melville, NY, 2023 (Published) DOI BibTeX

Empirical Inference Article Information theoretic measures of causal influences during transient neural events Shao, K., Logothetis, N. K., Besserve, M. Frontiers in Network Physiology, 3, 2023 (Published) DOI URL BibTeX

Autonomous Learning Article Interpretable Symbolic Regression for Data Science: Analysis of the 2022 Competition Franca, F. D., Virgolin, M., Kommenda, M., Majumder, M., Cranmer, M., Espada, G., Ingelse, L., Fonseca, A., Landajuela, M., Petersen, B., Glatt, R., Mundhenk, N., Lee, C., Hochhalter, J., Randall, D., Kamienny, P., Zhang, H., Dick, G., Simon, A., Burlacu, B., et al. arXiv, 2023 URL BibTeX

Embodied Vision Conference Paper Learning-based Relational Object Matching Across Views Elich, C., Armeni, I., Oswald, M. R., Pollefeys, M., Stueckler, J. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023 (Published)
Intelligent robots require object-level scene understanding to reason about possible tasks and interactions with the environment. Moreover, many perception tasks such as scene reconstruction, image retrieval, or place recognition can benefit from reasoning on the level of objects. While keypoint-based matching can yield strong results for finding correspondences for images with small to medium view point changes, for large view point changes, matching semantically on the object-level becomes advantageous. In this paper, we propose a learning-based approach which combines local keypoints with novel object-level features for matching object detections between RGB images. We train our object-level matching features based on appearance and inter-frame and cross-frame spatial relations between objects in an associative graph neural network. We demonstrate our approach in a large variety of views on realistically rendered synthetic images. Our approach compares favorably to previous state-of-the-art object-level matching approaches and achieves improved performance over a pure keypoint-based approach for large view-point changes.
preprint code DOI URL BibTeX

Modern Magnetic Systems Article Magnetic field sensing elements made of quasi-trapezoidal magnetoplasmonic crystals based on thin permalloy films Murzin, D., Belyaev, V. K., Groß, F., Gräfe, J., Perov, N., Komanicky, V., Rodionova, V. Journal of Magnetism and Magnetic Materials, 588, NH, Elsevier, Amsterdam, 2023 DOI BibTeX

Empirical Inference Article Metrizing Weak Convergence with Maximum Mean Discrepancies Simon-Gabriel, C., Barp, A., Schölkopf, B., Mackey, L. Journal of Machine Learning Research, 24(184), 2023 (Published)
This paper characterizes the maximum mean discrepancies (MMD) that metrize the weak convergence of probability measures for a wide class of kernels. More precisely, we prove that, on a locally compact, non-compact, Hausdorff space, the MMD of a bounded continuous Borel measurable kernel k, whose RKHS-functions vanish at infinity (i.e., Hk ⊂ C0), metrizes the weak convergence of probability measures if and only if k is continuous and integrally strictly positive definite (∫ s.p.d.) over all signed, finite, regular Borel measures. We also correct a prior result of Simon-Gabriel and Schölkopf (JMLR 2018, Thm. 12) by showing that there exist both bounded continuous ∫ s.p.d. kernels that do not metrize weak convergence and bounded continuous non-∫ s.p.d. kernels that do metrize it
arXiv URL BibTeX

Empirical Inference Article Mimicking Tumor Cell Heterogeneity of Colorectal Cancer in a Patient-derived Organoid-Fibroblast Model Atanasova, V. S., de Jesus Cardona, C., Hejret, V., Tiefenbacher, A., Mair, T., Tran, L., Pfneissl, J., Draganić, K., Binder, C., Kabiljo, J., Clement, J., Woeran, K., Neudert, B., Wohlhaupter, S., Haase, A., Domazet, S., Hengstschläger, M., Mitterhauser, M., Müllauer, L., Tichý, B., et al. Cellular and molecular gastroenterology and hepatology, 15(6):1391-1419, 2023 (Published) DOI BibTeX

Dynamic Locomotion Article Muscle Preflex Response to Perturbations in locomotion: In-vitro experiments and simulations with realistic boundary conditions Araz, M., Weidner, S., Izzi, F., Badri-Spröwitz, A., Siebert, T., Haeufle, A. D. F. B. Frontiers in Bioengineering and Biotechnology, 11, 2023 (Published)
Neuromuscular control loops feature substantial communication delays, but mammals run robustly even in the most adverse conditions. In-vivo experiments and computer simulation results suggest that muscles’ preflex—an immediate mechanical response to a perturbation—could be the critical contributor. Muscle preflexes act within a few milliseconds, an order of magnitude faster than neural reflexes. Their short-lasting activity makes mechanical preflexes hard to quantify in-vivo. Muscle models, on the other hand, require further improvement of their prediction accuracy during the non-standard conditions of perturbed locomotion. Additionally, muscles mechanically adapt by increased damping force. Our study aims to quantify the mechanical preflex work and test its mechanical force adaptation. We performed in-vitro experiments with biological muscle fibers under physiological boundary conditions, which we determined in computer simulations of perturbed hopping. Our findings show that muscles initially resist impacts with a stereotypical sti↵ness response—identified as short-range sti↵ness—regardless of the exact perturbation condition. We then observe a velocity adaptation to the force related to the amount of perturbation. The main contributor to the preflex work adaptation is not the force di↵erence but the muscle fiber stretch di↵erence. We find that both muscle sti↵ness and damping are activity-dependent properties. These results indicate that neural control could tune the preflex properties of muscles in expectation of ground conditions leading to previously inexplicable neuromuscular adaptation speeds.
DOI URL BibTeX

Modern Magnetic Systems Article Nanoporous adsorbents for hydrogen storage Hirscher, M., Zhang, L., Oh, H. Applied Physics A, 129(2), Springer-Verlag Heidelberg, Heidelberg, 2023 DOI BibTeX

Empirical Inference Book Chapter Natural Language Processing for Policymaking Jin, Z., Mihalcea, R. In Handbook of Computational Social Science for Policy, 141-162, 7, (Editors: Bertoni, E. and Fontana, M. and Gabrielli, L. and Signorelli, S. and Vespe, M.), Springer International Publishing, 2023 (Published) DOI BibTeX

Embodied Vision Ph.D. Thesis Object-Level Dynamic Scene Reconstruction With Physical Plausibility From RGB-D Images Strecke, M. F. Eberhard Karls Universität Tübingen, Tübingen, 2023 (Published)
Humans have the remarkable ability to perceive and interact with objects in the world around them. They can easily segment objects from visual data and have an intuitive understanding of how physics influences objects. By contrast, robots are so far often constrained to tailored environments for a specific task, due to their inability to reconstruct a versatile and accurate scene representation. In this thesis, we combine RGB-D video data with background knowledge of real-world physics to develop such a representation for robots.</br> </br> Our contributions can be separated into two main parts: a dynamic object tracking tool and optimization frameworks that allow for improving shape reconstructions based on physical plausibility. The dynamic object tracking tool "EM-Fusion" detects, segments, reconstructs, and tracks objects from RGB-D video data. We propose a probabilistic data association approach for attributing the image pixels to the different moving objects in the scene. This allows us to track and reconstruct moving objects and the background scene with state-of-the art accuracy and robustness towards occlusions.</br> </br> We investigate two ways of further optimizing the reconstructed shapes of moving objects based on physical plausibility. The first of these, "Co-Section", includes physical plausibility by reasoning about the empty space around an object. We observe that no two objects can occupy the same space at the same time and that the depth images in the input video provide an estimate of observed empty space. Based on these observations, we propose intersection and hull constraints, which we combine with the observed surfaces in a global optimization approach. Compared to EM-Fusion, which only reconstructs the observed surface, Co-Section optimizes watertight shapes. These watertight shapes provide a rough estimate of unseen surfaces and could be useful as initialization for further refinement, e.g., by interactive perception. In the second optimization approach, "DiffSDFSim", we reason about object shapes based on physically plausible object motion. We observe that object trajectories after collisions depend on the object's shape, and extend a differentiable physics simulation for optimizing object shapes together with other physical properties (e.g., forces, masses, friction) based on the motion of the objects and their interactions. Our key contributions are using signed distance function models for representing shapes and a novel method for computing gradients that models the dependency of the time of contact on object shapes. We demonstrate that our approach recovers target shapes well by fitting to target trajectories and depth observations. Further, the ground-truth trajectories are recovered well in simulation using the resulting shape and physical properties. This enables predictions about the future motion of objects by physical simulation.</br> </br> We anticipate that our contributions can be useful building blocks in the development of 3D environment perception for robots. The reconstruction of individual objects as in EM-Fusion is a key ingredient required for interactions with objects. Completed shapes as the ones provided by Co-Section provide useful cues for planning interactions like grasping of objects. Finally, the recovery of shape and other physical parameters using differentiable simulation as in DiffSDFSim allows simulating objects and thus predicting the effects of interactions. Future work might extend the presented works for interactive perception of dynamic environments by comparing these predictions with observed real-world interactions to further improve the reconstructions and physical parameter estimations.
DOI URL BibTeX

Modern Magnetic Systems Article Pump probe x-ray microscopy of photo-induced magnetization dynamics at MHz repetition rates Gerlinger, K., Pfau, B., Hennecke, M., Kern, L., Will, I., Noll, T., Weigand, M., Gräfe, J., Traeger, N., Schneider, M., Günther, C. M., Engel, D., Schütz, G., Eisebitt, S. Structural Dynamics, 10(2):024301, American Institute of Physics, Melville, NY, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Article Quantifying the spin-wave asymmetry in single and double rectangular Ni80Fe20 microstrips by TR-STXM, FMR, and micromagnetic simulations Pile, S., Ney, A., Lenz, K., Narkowicz, R., Lindner, J., Wintz, S., Förster, J., Mayr, S., Weigand, M. IEEE Transactions on Magnetics, 59(11), Published by the Institute of Electrical and Electronics Engineers for the Magnetics Group, New York, NY, 2023 DOI BibTeX

Materials Article Retention of dissolved organic matter during podzolisation: Testing processes in laboratory experiments and at the submicron scale Krettek, A., Höschen, C., Richter, G., Schweizer, S., Thilo, R. Geoderma Regional, 32:e00606, Elsevier Science, Amsterdam, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Article Seeding and emergence of composite skyrmions in a van der Waals magnet Powalla, L., Birch, M. T., Litzius, K., Wintz, S., Yasin, F. S., Turnbull, L. A., Schulz, F., Mayoh, D. A., Balakrishnan, G., Weigand, M., Yu, X., Kern, K., Schütz, G., Burghard, M. Advanced Materials, 35(12):2208930, Wiley-VCH, Weinheim, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Materials Article Site-selective substitution and resulting magnetism in arc-melted perovskite ATiO3-delta (A \textequals Ca, Sr, Ba) Yoon, S., Xie, W., Xiao, X., Checchia, S., Coduri, M., Schützendübe, P., Widenmeyer, M., Ebbinghaus, S. G., Balke, B., Weidenkaff, A., Schütz, G., Son, K. Journal of the American Ceramic Society, 106(11):6778-6786, American Ceramic Society, Westerville, OH, USA, 2023 (Published) DOI BibTeX

Modern Magnetic Systems Article Skyrmion and skyrmionium formation in the two-dimensional magnet Cr2Ge2Te6 Powalla, L., Birch, M. T., Litzius, K., Wintz, S., Satheesh, S., Weigand, M., Goering, E., Schütz, G., Burghard, M. Physical Review B, 108(21), American Physical Society, Woodbury, NY, 2023 DOI BibTeX

Modern Magnetic Systems Article Spatially-resolved dynamic sampling of different phasic magnetic resonances of nanoparticle ensembles in a magnetotactic bacterium Magnetospirillum magnetotacticum Feggeler, T., Lill, J., Günzing, D., Meckenstock, R., Spoddig, D., Efremova, M. V., Wintz, S., Weigand, M., Zingsem, B. W., Farle, M., Wende, H., Ollefs, K. J., Ohldag, H. New Journal of Physics, 25(4):043010, IOP Publishing, Bristol, 2023 (Published) DOI BibTeX