Institute Homepage
Institute Homepage EN Sign In

Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation

Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group

Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award

Physical Intelligence Article Wireless flow-powered miniature robot capable of traversing tubular structures Hong, C., Wu, Y., Wang, C., Ren, Z., Wang, C., Liu, Z., Hu, W., Sitti, M. Science Robotics, 9(88):eadi5155, 2024 (Published)
Wireless millimeter-scale robots capable of navigating through fluid-flowing tubular structures hold substantial potential for inspection, maintenance, or repair use in nuclear, industrial, and medical applications. However, prevalent reliance on external powering constrains these robots’ operational range and applicable environments. Alternatives with onboard powering must trade off size, functionality, and operation duration. Here, we propose a wireless millimeter-scale wheeled robot capable of using environmental flows to power and actuate its long-distance locomotion through complex pipelines. The flow-powering module can convert flow energy into mechanical energy, achieving an impeller speed of up to 9595 revolutions per minute, accompanied by an output power density of 11.7 watts per cubic meter and an efficiency of 33.7%. A miniature gearbox module can further transmit the converted mechanical energy into the robot’s locomotion system, allowing the robot to move against water flow at an average rate of up to 1.05 meters per second. The robot’s motion status (moving against/with flow or pausing) can be switched using an external magnetic field or an onboard mechanical regulator, contingent on different proposed control designs. In addition, we designed kirigami-based soft wheels for adaptive locomotion. The robot can move against flows of various substances within pipes featuring complex geometries and diverse materials. Solely powered by flow, the robot can transport cylindrical payloads with a diameter of up to 55% of the pipe’s diameter and carry devices such as an endoscopic camera for pipeline inspection, a wireless temperature sensor for environmental temperature monitoring, and a leak-stopper shell for infrastructure maintenance.
DOI URL BibTeX
Perceiving Systems Article InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction from Multi-view RGB-D Images Huang, Y., Taheri, O., Black, M. J., Tzionas, D. International Journal of Computer Vision (IJCV), 132(7):2551-2566, 2024 (Published)
Humans constantly interact with objects to accomplish tasks. To understand such interactions, computers need to reconstruct these in 3D from images of whole bodies manipulating objects, e.g., for grasping, moving and using the latter. This involves key challenges, such as occlusion between the body and objects, motion blur, depth ambiguities, and the low image resolution of hands and graspable object parts. To make the problem tractable, the community has followed a divide-and-conquer approach, focusing either only on interacting hands, ignoring the body, or on interacting bodies, ignoring the hands. However, these are only parts of the problem. On the contrary, recent work focuses on the whole problem. The GRAB dataset addresses whole-body interaction with dexterous hands but captures motion via markers and lacks video, while the BEHAVE dataset captures video of body-object interaction but lacks hand detail. We address the limitations of prior work with InterCap, a novel method that reconstructs interacting whole-bodies and objects from multi-view RGB-D data, using the parametric whole-body SMPL-X model and known object meshes. To tackle the above challenges, InterCap uses two key observations: (i) Contact between the body and object can be used to improve the pose estimation of both. (ii) Consumer-level Azure Kinect cameras let us set up a simple and flexible multi-view RGB-D system for reducing occlusions, with spatially calibrated and temporally synchronized cameras. With our InterCap method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 daily objects of various sizes and affordances, including contact with the hands or feet. To this end, we introduce a new data-driven hand motion prior, as well as explore simple ways for automatic contact detection based on 2D and 3D cues. In total, InterCap has 223 RGB-D videos, resulting in 67,357 multi-view frames, each containing 6 RGB-D images, paired with pseudo ground-truth 3D body and object meshes. Our InterCap method and dataset fill an important gap in the literature and support many research directions. Data and code are available at https://intercap.is.tue.mpg.de.
Paper DOI URL BibTeX
Perceiving Systems Conference Paper 3D Neural Edge Reconstruction Lil, L., Peng, S., Yu, Z., Liu, S., Pautrat, R., Yin, X., Pollefeys, M. In Proceedings 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 21219-21229, 10.1109/CVPR52733.2024.02005, 2024 (Published) DOI URL BibTeX
Human Aspects of Machine Learning Unpublished A Dynamic Model of Performative Human-ML Collaboration: Theory and Empirical Evidence Sühr, T., Samadi, S., Farronato, C. 2024 (Submitted) BibTeX
Physics for Inference and Optimization Conference Paper A causality-inspired adjusted plus-minus model for player evaluation in team sports De Bacco, C., Wang, Y., Blei, D. M. In Proceedings of Machine Learning Research (PMLR), Proceedings Third Conference on Causal Learning and Reasoning, 236:769-792, Third Conference on Causal Learning and Reasoning, 2024 (Published) URL BibTeX
Empirical Inference Article A phenomenological account for causality in terms of elementary actions Janzing, D., Garrido Mejia, S. Journal of Causal Inference, 12(1), 2024 (Published) DOI URL BibTeX
Empirical Inference Article A temperate super-Jupiter imaged with JWST in the mid-infrared Matthews, E. C., Carter, A. L., Pathak, P., Morley, C. V., Phillips, M. W., S. Krishanth, P. M., Feng, F., Bonse, M. J., Boogaard, L. A., Burt, J. A., Crossfield, I. J. M., Douglas, E. S., Henning, T., Hom, J., Ko, C. -., Kasper, M., Lagrange, A., Petit Dit de la Roche, D., Philipot, F. Nature, 633:789–792, 2024 (Published)
Of the approximately 25 directly imaged planets to date, all are younger than 500 Myr, and all but six are younger than 100 Myr (ref. 1). Eps Ind A (HD209100, HIP108870) is a K5V star of roughly solar age (recently derived as 3.7–5.7 Gyr (ref. 2) and  Gyr (ref. 3)). A long-term radial-velocity trend4,5 and an astrometric acceleration6,7 led to claims of a giant planet2,8,9 orbiting the nearby star (3.6384 ± 0.0013 pc; ref. 10). Here we report JWST coronagraphic images which reveal a giant exoplanet that is consistent with these radial and astrometric measurements but inconsistent with the previously claimed planet properties. The new planet has a temperature of approximately 275 K and is remarkably bright at 10.65 and 15.50 µm. Non-detections between 3.5 and 5.0 µm indicate an unknown opacity source in the atmosphere, possibly suggesting a high-metallicity, high carbon-to-oxygen ratio planet. The best-fitting temperature of the planet is consistent with theoretical thermal evolution models, which were previously untested at this temperature range. The data indicate that this is probably the only giant planet in the system, and therefore we refer to it as b, despite it having significantly different orbital properties than the previously claimed planet b.
DOI URL BibTeX
Safety- and Efficiency- aligned Learning Technical Report AI Risk Management Should Incorporate Both Safety and Security Qi, X., Huang, Y., Zeng, Y., Debenedetti, E., Geiping, J., He, L., Huang, K., Madhushani, U., Sehwag, V., Shi, W., Wei, B., Xie, T., Chen, D., Chen, P., Ding, J., Jia, R., Ma, J., Narayanan, A., Su, W. J., Wang, M., et al. 2024 BibTeX
Perceiving Systems Article Accelerated Video Annotation Driven by Deep Detector and Tracker Price, E., Ahmad, A. INTELLIGENT AUTONOMOUS SYSTEMS 18, 2:141–153, IAS, 2024 (Published)
Annotating object ground truth in videos is vital for several downstream tasks in robot perception and machine learning, such as for evaluating the performance of an object tracker or training an image-based object detector. The accuracy of the annotated instances of the moving objects on every image frame in a video is crucially important. Achieving that through manual annotations is not only very time consuming and labor intensive, but is also prone to high error rate. State-of-the-art annotation methods depend on manually initializing the object bounding boxes only in the first frame and then use classical tracking methods, e.g., adaboost, or kernelized correlation filters, to keep track of those bounding boxes. These can quickly drift, thereby requiring tedious manual supervision. In this paper, we propose a new annotation method which leverages a combination of a learning-based detector (SSD) and a learning-based tracker (RE). Through this, we significantly reduce annotation drifts, and, consequently, the required manual supervision. We validate our approach through annotation experiments using our proposed annotation method and existing baselines on a set of drone video frames. Source code and detailed information on how to run the annotation program can be found at https://github.com/robot-perception-group/smarter-labelme
project DOI URL BibTeX
Empirical Inference Miscellaneous Analyzing Human Questioning Behavior and Causal Curiosity through Natural Queries Ceraolo, R., Kharlapenko, D., Khan, A., Reymond, A., Mihalcea, R., Sachan, M., Schölkopf, B., Jin, Z. 2024 (Published) URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Be like a Goldfish, Don’t Memorize! Mitigating Memorization in Generative LLMs Hans, A., Wen, Y., Jain, N., Kirchenbauer, J., Kazemi, H., Singhania, P., Singh, S., Somepalli, G., Geiping, J., Bhatele, A., Goldstein, T. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX
Learning and Dynamical Systems Article Bi-level Motion Imitation for Humanoid Robots Zhao, W., Zhao, Y., Pajarinen, J., Muehlebach, M. Conference on Robot Learning, 2024 (Published) BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Bring Your Own Data! Self-Sensitivity Evaluation for Large Language Models Jain, N., Saifullah, K., Wen, Y., Kirchenbauer, J., Shu, M., Saha, A., Goldblum, M., Geiping, J., Goldstein, T. In Proceedings of the First Conference on Language Modeling, First Conference on Language Modeling, 2024 (Published) URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper CALVIN: Improved Contextual Video Captioning via Instruction Tuning Somepalli, G., Chowdhury, A., Geiping, J., Basri, R., Goldstein, T., Jacobs, D. W. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX
Human Aspects of Machine Learning Conference Paper Causal Adversarial Perturbations for Individual Fairness and Robustness in Heterogeneous Data Spaces. Ehyaei, A., Mohammadi, K., Karimi, A., Samadi, S., Farnadi, G. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024 (Published) BibTeX
Safety- and Efficiency- aligned Learning Technical Report Coercing LLMs to do and reveal (almost) anything Geiping, J., Stein, A., Shu, M., Saifullah, K., Wen, Y., Goldstein, T. 2024 (Submitted) URL BibTeX
Empirical Inference Article Connectome-constrained networks predict neural activity across the fly visual system Lappalainen, J. K., Tschopp, F. D., Prakhya, S., McGill, M., Nern, A., Shinomiya, K., Takemura, S., Gruntman, E., Macke, J. H., Turaga, S. C. Nature, 634:1132–1140, 2024 (Published)
We can now measure the connectivity of every neuron in a neural circuit, but we cannot measure other biological details, including the dynamical characteristics of each neuron. The degree to which measurements of connectivity alone can inform the understanding of neural computation is an open question10. Here we show that with experimental measurements of only the connectivity of a biological neural network, we can predict the neural activity underlying a specified neural computation. We constructed a model neural network with the experimentally determined connectivity for 64 cell types in the motion pathways of the fruit fly optic lobe but with unknown parameters for the single-neuron and single-synapse properties. We then optimized the values of these unknown parameters using techniques from deep learning, to allow the model network to detect visual motion. Our mechanistic model makes detailed, experimentally testable predictions for each neuron in the connectome. We found that model predictions agreed with experimental measurements of neural activity across 26 studies. Our work demonstrates a strategy for generating detailed hypotheses about the mechanisms of neural circuit function from connectivity measurements. We show that this strategy is more likely to be successful when neurons are sparsely connected—a universally observed feature of biological neural networks across species and brain regions.
DOI URL BibTeX
Learning and Dynamical Systems Article Controlling Participation in Federated Learning with Feedback Cummins, M., Er, G. D., Muehlebach, M. 2024 (Submitted) BibTeX
Empirical Inference Conference Paper DeViL: Decoding Vision features into Language Dani, M., Rio-Torto, I., Alaniz, S., Akata, Z. In Lecture Notes in Computer Science, vol 14264, 363–377, 45th Annual Conference of the German-Association-for-Pattern-Recognition (DAGM GCPR), 2024 (Published) DOI URL BibTeX
Human Aspects of Machine Learning Conference Paper Designing Ambiguity Sets for Distributionally Robust Optimization Using Structural Causal Optimal Transport Ehyaei, A. R., Farnadi, G., Samadi, S. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024 (Published) BibTeX
Perceiving Systems Article Editor’s Note: Special Issue on Computer Vision Approach for Animal Tracking and Modeling Park, H. S., Rhodin, H., Kanazawa, A., Neverova, N., Nobuhara, S., Black, M., Zuffi, S. Journal of Computer Vision, 132, 2024 (Published) DOI URL BibTeX
Learning and Dynamical Systems Article Event-Based Federated Q-Learning Er, D., Muehlebach, M. Workshop on Foundations of RL and Control, International Conference on Machine Learning, 2024 (Published) BibTeX
Neural Capture and Synthesis Conference Paper FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models Aneja, S., Thies, J., Dail, A., Niessner, M. In Proceedings 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 21263-21273, IEEE, CVPR, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Technical Report Generating Potent Poisons and Backdoors from Scratch with Guided Diffusion Souri, H., Bansal, A., Kazemi, H., Fowl, L., Saha, A., Geiping, J., Wilson, A. G., Chellappa, R., Goldstein, T., Goldblum, M. 2024 (Submitted) URL BibTeX
Article Generating realistic neurophysiological time series with denoising diffusion probabilistic models Vetter, J., Macke, J. H., Gao, R. Patterns, 5(9), 2024 (Published) DOI URL BibTeX
Rationality Enhancement Article Identifying Resource-Rational Heuristics for Risky Choice Krueger, P., Callaway, F., Gul, S., Griffiths, T., Lieder, F. Psychological Review, 2024 (Published) DOI URL BibTeX
Physical Intelligence Article Individual and collective manipulation of multifunctional bimodal droplets in three dimensions Sun, M., Sun, B., Park, M., Yang, S., Wu, Y., Zhang, M., Kang, W., Yoon, J., Zhang, L., Sitti, M. Science Advances, 10(19):eadp1439, American Association for the Advancement of Science, 2024 (Published) BibTeX
Embodied Vision Ph.D. Thesis Investigating Shape Priors, Relationships, and Multi-Task Cues for Object-level Scene Understanding Elich, C. ETH Zürich, Zurich, 2024 (Published)
Humans are proficient at intuitively identifying objects and reasoning about their diverse properties from complex visual observations. Despite significant advances in artificial intelligence, computers have yet to achieve a comparable level of understanding, which is crucial for effective reasoning about tasks and interactions within an environment. In this thesis, we explore the benefits of various visual cues when dealing with key challenges in scene understanding, specifically focusing on weak supervision, finding view correspondence, and paradigms for simultaneously learning multiple tasks. We begin by investigating cues that reduce the need for full supervision. In particular, we propose an approach for learning multi-object 3D scene decomposition and object-wise properties from single images with only weak supervision. Our method utilizes a recurrent encoder to infer a latent representation for each object and a differentiable renderer to obtain a training signal. To guide the training process and constrain the search space of possible solutions, we leverage prior knowledge through pre-trained 3D shape spaces. Subsequently, we investigate the benefits of reasoning about relations between objects to learn more distinct object representations that allow for matching object detections across viewpoint changes. To address this, we introduce an approach that employs graph neural networks to learn matching features based on appearance as well as inter- and cross-frame relations. We conduct comparisons with keypoint-based methods and propose a methodology to combine these approaches, aiming to achieve overall improved performance. Finally, we consider the challenge of multi-task learning and analyze related paradigms in the context of basic single-task learning. In particular, we study the impact of the choice of optimizer, the role of gradient conflicts, and the effects on the transferability of features learned through either learning setup on common image corruptions. Our findings reveal surprising similarities between single-task and multi-task learning, suggesting that methods and techniques from one field could be advantageously applied to the other.
DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Investigating Style Similarity in Diffusion Models Somepalli, G., Gupta, A., Gupta, K., Palta, S., Goldblum, M., Geiping, J., Shrivastava, A., Goldstein, T. In European Conference on Computer Vision (ECCV 2024), LNCS, Springer Cham, 2024 (Published) URL BibTeX
Article LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry Chen, W., Chen, L., Wang, R., Pollefeys, M. Proceedings 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19844-19853, 1, 1, YeaakEMl, 2024 (Accepted) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper LMD3: Language Model Data Density Dependence Kirchenbauer, J., Honke, G., Somepalli, G., Geiping, J., Lee, K., Ippolito, D., Goldstein, T., Andre, D. In Proceedings of the First Conference on Language Modeling, First Conference on Language Modeling, 2024 (Published) URL BibTeX
Empirical Inference Miscellaneous Language Model Alignment in Multilingual Trolley Problems Jin, Z., Levine, S., Kleiman-Weiner, M., Piatti, G., Liu, J., Gonzalez, F., Ortu, F., Strausz, A., Sachan, M., Mihalcea, R., Choi, Y., Schölkopf, B. 2024 (Published) URL BibTeX
Empirical Inference Article Low-power scalable multilayer optoelectronic neural networks enabled with incoherent light Song, A., Kottapalli, S. N. M., Goyal, R., Schoelkopf, B., Fischer, P. Nature Communications, 15:10692, 2024 (Published)
Optical approaches have made great strides towards the goal of high-speed, energy-efficient computing necessary for modern deep learning and AI applications. Read-in and read-out of data, however, limit the overall performance of existing approaches. This study introduces a multilayer optoelectronic computing framework that alternates between optical and optoelectronic layers to implement matrix-vector multiplications and rectified linear functions, respectively. Our framework is designed for real-time, parallelized operations, leveraging 2D arrays of LEDs and photodetectors connected via independent analog electronics. We experimentally demonstrate this approach using a system with a three-layer network with two hidden layers and operate it to recognize images from the MNIST database with a recognition accuracy of 92% and classify classes from a nonlinear spiral data with 86% accuracy. By implementing multiple layers of a deep neural network simultaneously, our approach significantly reduces the number of read-ins and read-outs required and paves the way for scalable optical accelerators requiring ultra low energy.
DOI URL BibTeX
Embodied Vision Ph.D. Thesis Methods for Learning Adaptive and Symbolic Forward Models for Control and Planning Achterhold, J. M. Eberhard Karls Universität Tübingen, Tübingen, 2024 (Published)
Learning-based methods for sequential decision making, i.e., methods which leverage data, have shown the ability to solve complex problems in recent years. This includes control of dynamical systems, as well as mastering games such as Go and StarCraft. In addition, these methods often promise to be applicable to a wide variety of problems. A subclass of these methods are model-based methods. They leverage data to learn a model which allows predicting the evolution of a dynamical system to control. In recent research, it was shown that these methods, in contrast to model-free methods, require less data to be trained. In addition, model-based methods allow re-using the dynamics model when the task to be solved has changed, and straightforward adaptation to changes in the system’s dynamics. One particular focus of this thesis is on learning dynamics models which can data-efficiently adapt to changes in the system’s dynamics, as well as the efficient collection of data to adapt a learned model. In this regard, two novel methods are presented. In the application domain of autonomous robot navigation, in which both parameters of the robot and the terrain are subject to change, a novel method comprising an adaptive dynamics model is presented and evaluated on a simulated environment. A further advantage of model-based methods is the ability to incorporate physical prior knowledge for model design. In this thesis, we demonstrate that leveraging physical prior knowledge is advantageous for the task of tracking and predicting the motion of a table tennis ball, respecting its spin. However, model-based methods, in particular planning with learned models, have to cope with certain challenges. For long prediction horizons, which are required if the effect of an action is apparent only far in the future, model errors accumulate. In addition, model-based planning is commonly computationally intensive, which is problematic if high-frequency, reactive control is required. In this thesis, a method is presented to alleviate these problems. To this end, we propose a two-layered hierarchical method. Model-based planning is only applied on the higher layer on symbolic abstractions. On the lower-layer, model-free reactive control is used. We show successful application of this method to board games which can only be interacted with through a robotic manipulator, e.g., a robotic arm, which requires high-frequency reactive control.
DOI URL BibTeX
Physical Intelligence Article Micro- and nanofabrication of dynamic hydrogels with multichannel information Zhang, M., Lee, Y., Zheng, Z., Khan, M. T. A., Lyu, X., Byun, J., Giessen, H., Sitti, M. 2024 (Published) BibTeX
Empirical Inference Article Neonatal apnea and hypopnea prediction in infants with Robin sequence with neural additive models for time series Vetter, J., Lim, K., Dijkstra, T. M. H., Dargaville, P. A., Kohlbacher, O., Macke, J. H., Poets, C. F. PLOS Health Digital, 3(12):e0000678, 2024 (Accepted) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Object Recognition as Next Token Prediction Yue, K., Chen, B., Geiping, J., Li, H., Goldstein, T., Lim, S. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), CVPR, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper On the Reliability of Watermarks for Large Language Models Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., Goldstein, T. In The Twelfth International Conference on Learning Representations, ICLR 2024, The Twelfth International Conference on Learning Representations, 2024 (Published) URL BibTeX
Learning and Dynamical Systems Article Online Performance Optimization of Nonlinear Systems: A Gray-Box Approach He, Z., Muehlebach, M., Bolognani, S., Dörfler, F. Workshop on Foundations of RL and Control, International Conference on Machine Learning, 2024 (Published) BibTeX
Learning and Dynamical Systems Article Optimal Transport with Constraints: From Mirror Descent to Classical Mechanics Ibrahim, A. A., Muehlebach, M., De Bacco, C. Physical Review Letters, 133:057401 , 2024 (Published) DOI BibTeX
Physical Intelligence Article Perturbing Dynamics of Active Emulsions and Their Collectives Khan, M. T. A., Gardi, G., Soon, R. H., Zhang, M., Sitti, M. arXiv preprint arXiv:2405.05889, 2024 (Submitted) BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models Wen, Y., Marchyok, L., Hong, S., Geiping, J., and Goldstein, T., Carlini, N. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX
Article Reinforcement learning based autonomous multi-rotor landing on moving platforms Ahmad, P. G. &. A. 2024 (Published) BibTeX
Article Results from the autoPET challenge on fully automated lesion\nsegmentation in oncologic PET/CT imaging Gatidis, S., Frueh, M., Fabritius, M. P., Gu, S., Nikolaou, K., La Fougere, C., Ye, J., He, J., Peng, Y., Bi, L., Ma, J., Wang, B., Zhang, J., Huang, Y., Heiliger, L., Marinov, Z., Stiefelhagen, R., Egger, J., Kleesiek, J., Sibille, L., et al. Nature Machine Intelligence, 6(11):1396–1405, 2024 (Published)
Automated detection of tumour lesions on positron emission tomography–computed tomography (PET/CT) image data is a clinically relevant but highly challenging task. Progress in this field has been hampered in the past owing to the lack of publicly available annotated data and limited availability of platforms for inter-institutional collaboration. Here we describe the results of the autoPET challenge, a biomedical image analysis challenge aimed to motivate research in the field of automated PET/CT image analysis. The challenge task was the automated segmentation of metabolically active tumour lesions on whole-body 18F-fluorodeoxyglucose PET/CT. Challenge participants had access to a large publicly available annotated PET/CT dataset for algorithm training. All algorithms submitted to the final challenge phase were based on deep learning methods, mostly using three-dimensional U-Net architectures. Submitted algorithms were evaluated on a private test set composed of 150 PET/CT studies from two institutions. An ensemble model of the highest-ranking algorithms achieved favourable performance compared with individual algorithms. Algorithm performance was dependent on the quality and quantity of data and on algorithm design choices, such as tailored post-processing of predicted segmentations. Future iterations of this challenge will focus on generalization and clinical translation.
DOI URL BibTeX
Autonomous Learning Conference Paper SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models Sancaktar, C., Gumbsch, C., Zadaianchuk, A., Kolev, P., Martius, G. In The Training Agents with Foundation Models Workshop at RLC, 2024, indicates equal contribution (Published)
Exploring useful behavior is a keystone of reinforcement learning (RL). Existing approaches to intrinsic motivation, following general principles such as information gain, mostly uncover low-level interactions. In contrast, children’s play suggests that they engage in semantically meaningful high-level behavior by imitating or interacting with their caregivers. Recent work has focused on using foundation models to inject these semantic biases into exploration. However, these methods often rely on unrealistic assumptions, such as environments already embedded in language or access to high-level actions. To bridge this gap, we propose SEmaNtically Sensible ExploratIon (Sensei), a framework to equip model-based RL agents with intrinsic motivation for semantically meaningful behavior. To do so, we distill an intrinsic reward signal of interestingness from Vision Language Model (VLM) annotations. The agent learns to predict and maximize these intrinsic rewards using a world model learned directly from intrinsic rewards, image observations, and low-level actions. We show that in both robotic and video game-like simulations Sensei manages to discover a variety of meaningful behaviors. We believe Sensei provides a general tool for integrating feedback from foundation models into autonomous agents, a crucial research direction as openly available VLMs become more powerful.
URL BibTeX
Physics for Inference and Optimization Article Similarity and economy of scale in urban transportation networks and\noptimal transport-based infrastructures Leite, D., De Bacco, C. Nature Communications, 15:7981, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., Goldstein, T. In Proceedings of Machine Learning Research, Proceedings of the Forty-First International Conference on Machine Learning , Forty-First International Conference on Machine Learning , 2024 (Published) URL BibTeX
Conference Paper Terminating Differentiable Tree Experts Jonathan Thomm, M. H. G. C. A. T. B. S. &. A. R. In 2024 (Published) BibTeX
Learning and Dynamical Systems Article Toward a Systems Theory of Algorithms Doerfler, F., He, Z., Belgioioso, G., Bolognani, S., Lygeros, J., Muehlebach, M. IEEE CONTROL SYSTEMS LETTERS, 8:1198 - 1210, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Transformers Can Do Arithmetic with the Right Embeddings McLeish, S. M., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B. R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A., Goldstein, T. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX