Institute Homepage
Institute Homepage DE Sign In

Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation

Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group

Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award

Physical Intelligence Article Micro- and nanofabrication of dynamic hydrogels with multichannel information Zhang, M., Lee, Y., Zheng, Z., Khan, M. T. A., Lyu, X., Byun, J., Giessen, H., Sitti, M. 2024 (Published) BibTeX
Empirical Inference Article Neonatal apnea and hypopnea prediction in infants with Robin sequence with neural additive models for time series Vetter, J., Lim, K., Dijkstra, T. M. H., Dargaville, P. A., Kohlbacher, O., Macke, J. H., Poets, C. F. PLOS Health Digital, 3(12):e0000678, 2024 (Accepted) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Object Recognition as Next Token Prediction Yue, K., Chen, B., Geiping, J., Li, H., Goldstein, T., Lim, S. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), CVPR, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper On the Reliability of Watermarks for Large Language Models Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., Goldstein, T. In The Twelfth International Conference on Learning Representations, ICLR 2024, The Twelfth International Conference on Learning Representations, 2024 (Published) URL BibTeX
Learning and Dynamical Systems Article Online Performance Optimization of Nonlinear Systems: A Gray-Box Approach He, Z., Muehlebach, M., Bolognani, S., Dörfler, F. Workshop on Foundations of RL and Control, International Conference on Machine Learning, 2024 (Published) BibTeX
Learning and Dynamical Systems Article Optimal Transport with Constraints: From Mirror Descent to Classical Mechanics Ibrahim, A. A., Muehlebach, M., De Bacco, C. Physical Review Letters, 133:057401 , 2024 (Published) DOI BibTeX
Physical Intelligence Article Perturbing Dynamics of Active Emulsions and Their Collectives Khan, M. T. A., Gardi, G., Soon, R. H., Zhang, M., Sitti, M. arXiv preprint arXiv:2405.05889, 2024 (Submitted) BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models Wen, Y., Marchyok, L., Hong, S., Geiping, J., and Goldstein, T., Carlini, N. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX
Article Reinforcement learning based autonomous multi-rotor landing on moving platforms Ahmad, P. G. &. A. 2024 (Published) BibTeX
Article Results from the autoPET challenge on fully automated lesion\nsegmentation in oncologic PET/CT imaging Gatidis, S., Frueh, M., Fabritius, M. P., Gu, S., Nikolaou, K., La Fougere, C., Ye, J., He, J., Peng, Y., Bi, L., Ma, J., Wang, B., Zhang, J., Huang, Y., Heiliger, L., Marinov, Z., Stiefelhagen, R., Egger, J., Kleesiek, J., Sibille, L., et al. Nature Machine Intelligence, 6(11):1396–1405, 2024 (Published)
Automated detection of tumour lesions on positron emission tomography–computed tomography (PET/CT) image data is a clinically relevant but highly challenging task. Progress in this field has been hampered in the past owing to the lack of publicly available annotated data and limited availability of platforms for inter-institutional collaboration. Here we describe the results of the autoPET challenge, a biomedical image analysis challenge aimed to motivate research in the field of automated PET/CT image analysis. The challenge task was the automated segmentation of metabolically active tumour lesions on whole-body 18F-fluorodeoxyglucose PET/CT. Challenge participants had access to a large publicly available annotated PET/CT dataset for algorithm training. All algorithms submitted to the final challenge phase were based on deep learning methods, mostly using three-dimensional U-Net architectures. Submitted algorithms were evaluated on a private test set composed of 150 PET/CT studies from two institutions. An ensemble model of the highest-ranking algorithms achieved favourable performance compared with individual algorithms. Algorithm performance was dependent on the quality and quantity of data and on algorithm design choices, such as tailored post-processing of predicted segmentations. Future iterations of this challenge will focus on generalization and clinical translation.
DOI URL BibTeX
Autonomous Learning Conference Paper SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models Sancaktar, C., Gumbsch, C., Zadaianchuk, A., Kolev, P., Martius, G. In The Training Agents with Foundation Models Workshop at RLC, 2024, indicates equal contribution (Published)
Exploring useful behavior is a keystone of reinforcement learning (RL). Existing approaches to intrinsic motivation, following general principles such as information gain, mostly uncover low-level interactions. In contrast, children’s play suggests that they engage in semantically meaningful high-level behavior by imitating or interacting with their caregivers. Recent work has focused on using foundation models to inject these semantic biases into exploration. However, these methods often rely on unrealistic assumptions, such as environments already embedded in language or access to high-level actions. To bridge this gap, we propose SEmaNtically Sensible ExploratIon (Sensei), a framework to equip model-based RL agents with intrinsic motivation for semantically meaningful behavior. To do so, we distill an intrinsic reward signal of interestingness from Vision Language Model (VLM) annotations. The agent learns to predict and maximize these intrinsic rewards using a world model learned directly from intrinsic rewards, image observations, and low-level actions. We show that in both robotic and video game-like simulations Sensei manages to discover a variety of meaningful behaviors. We believe Sensei provides a general tool for integrating feedback from foundation models into autonomous agents, a crucial research direction as openly available VLMs become more powerful.
URL BibTeX
Physics for Inference and Optimization Article Similarity and economy of scale in urban transportation networks and\noptimal transport-based infrastructures Leite, D., De Bacco, C. Nature Communications, 15:7981, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., Goldstein, T. In Proceedings of Machine Learning Research, Proceedings of the Forty-First International Conference on Machine Learning , Forty-First International Conference on Machine Learning , 2024 (Published) URL BibTeX
Conference Paper Terminating Differentiable Tree Experts Jonathan Thomm, M. H. G. C. A. T. B. S. &. A. R. In 2024 (Published) BibTeX
Learning and Dynamical Systems Article Toward a Systems Theory of Algorithms Doerfler, F., He, Z., Belgioioso, G., Bolognani, S., Lygeros, J., Muehlebach, M. IEEE CONTROL SYSTEMS LETTERS, 8:1198 - 1210, 2024 (Published) DOI URL BibTeX
Safety- and Efficiency- aligned Learning Conference Paper Transformers Can Do Arithmetic with the Right Embeddings McLeish, S. M., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B. R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A., Goldstein, T. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) URL BibTeX
Empirical Inference Conference Paper Waffling around for Performance: Visual Classification with Random Words and Broad Concepts Roth, K., Kim, J. M., Koepke, A. S., Vinyals, O., Schmid, C., Akata, Z. In Proceedings 2023 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2023), 15700-15711, 2023 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2023), 2024 (Published) DOI URL BibTeX
Human Aspects of Machine Learning Conference Paper Wasserstein Distributionally Robust Optimization Through the Lens of Structural Causal Models and Individual Fairness. Ehyaei, A. R., Farnadi, G., Samadi, S. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 (Published) BibTeX
Perceiving Systems Conference Paper Emotional Speech-Driven Animation with Content-Emotion Disentanglement Daněček, R., Chhatre, K., Tripathi, S., Wen, Y., Black, M. J., Bolkart, T. In SIGGRAPH Asia 2023 Conference Papers, Association for Computing Machinery , New York, NY, SIGGRAPH Asia, December 2023 (Published)
To be widely adopted, 3D facial avatars must be animated easily, realistically, and directly from speech signals. While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions. Realistic facial animation requires lip-sync together with the natural expression of emotion. To that end, we propose EMOTE (Expressive Model Optimized for Talking with Emotion), which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. To achieve this, we supervise EMOTE with decoupled losses for speech (i.e., lip-sync) and emotion. These losses are based on two key observations: (1) deformations of the face due to speech are spatially localized around the mouth and have high temporal frequency, whereas (2) facial expressions may deform the whole face and occur over longer intervals. Thus, we train EMOTE with a per-frame lip-reading loss to preserve the speech-dependent content, while supervising emotion at the sequence level. Furthermore, we employ a content-emotion exchange mechanism in order to supervise different emotions on the same audio, while maintaining the lip motion synchronized with the speech. To employ deep perceptual losses without getting undesirable artifacts, we devise a motion prior in the form of a temporal VAE. Due to the absence of high-quality aligned emotional 3D face datasets with speech, EMOTE is trained with 3D pseudo-ground-truth extracted from an emotional video dataset (i.e., MEAD). Extensive qualitative and perceptual evaluations demonstrate that EMOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data, while offering additional, high-quality emotional control.
arXiv DOI URL BibTeX
Autonomous Learning Conference Paper On Imitation in Mean-field Games Ramponi, G., Kolev, P., Olivier, P., He, N., Laurière, M., Geist, M. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 1-12, Curran Associates Inc., NeurIPS, December 2023 (Published)
We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs), where the goal is to imitate the behavior of a population of agents following a Nash equilibrium policy according to some unknown payoff function. IL in MFGs presents new challenges compared to single-agent IL, particularly when both the reward function and the transition kernel depend on the population distribution. In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap. Then we show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. However, when the dynamics is population-dependent, we provide a novel upper-bound that suggests IL is harder in this setting. To address this issue, we propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control (MFC) problem, suggesting progress in IL within MFGs may have to build upon MFC.
DOI URL BibTeX
Empirical Inference Conference Paper Optimistic Active Exploration of Dynamical Systems Sukhija, B., Treven, L., Sancaktar, C., Blaes, S., Coros, S., Krause, A. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 1-32, Curran Associates Inc. , NeurIPS, December 2023 (Published) DOI URL BibTeX
Empirical Inference Article Machine-Learning-Aided Prediction of Brain Metastases Development in Non-Small-Cell Lung Cancers Visonà, G., Spiller, L. M., Hahn, S., Hattingen, E., Vogl, T. J., Schweikert, G., Bankov, K., Demes, M., Reis, H., Wild, P., Zeiner, P. S., Acker, F., Sebastian, M., Wenger, K. J. Clinical lung cancer, 24(8):e311-e322, December 2023 (Published) DOI BibTeX
Empirical Inference Conference Paper A Measure-Theoretic Axiomatisation of Causality Park, J., Buchholz, S., Schölkopf, B., Muandet, K. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:28510-28540, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Beyond Good Intentions: Reporting the Research Landscape of NLP for Social Good Gonzalez*, F., Jin*, Z., Schölkopf, B., Hope, T., Sachan, M., Mihalcea, R. Findings of the Association for Computational Linguistics: EMNLP 2023, 415-438, (Editors: Houda Bouamor and Juan Pino and Kalika Bali), Association for Computational Linguistics, December 2023, *equal contribution (Published) DOI BibTeX
Physics for Inference and Optimization Article Bilevel Optimization for Traffic Mitigation in Optimal Transport Networks Lonardi, A., De Bacco, C. Physical Review Letter, 131:267401, December 2023 (Published) Preprint Code Paper DOI URL BibTeX
Empirical Inference Conference Paper CLadder: Assessing Causal Reasoning in Language Models Jin*, Z., Chen*, Y., Leeb*, F., Gresele*, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez, F., Kleiman-Weiner, M., Sachan, M., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:31038-31065, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *main contributors (Published) URL BibTeX
Empirical Inference Conference Paper Can semi-supervised learning use all the data effectively? A lower bound perspective Tifrea*, A., Yüce*, G., Sanyal, A., Yang, F. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:21960-21982, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Causal Component Analysis Liang, W., Kekić, A., von Kügelgen, J., Buchholz, S., Besserve, M., Gresele*, L., Schölkopf*, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:32481-32520, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *shared last author (Published) URL BibTeX
Empirical Inference Conference Paper Causal Modeling with Stationary Diffusions Lorch, L., Krause*, A., Schölkopf*, B. Causal Representation Learning Workshop at NeurIPS 2023, December 2023, *equal supervision (Published) URL BibTeX
Empirical Inference Conference Paper Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data Guo*, S., Tóth*, V., Schölkopf, B., Huszár, F. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:36463-36475, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Causal normalizing flows: from theory to practice Javaloy, A., Sanchez-Martin, P., Valera, I. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:58833-58864, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Social Foundations of Computation Algorithms and Society Conference Paper Collaborative Learning via Prediction Consensus Fan, D., Mendler-Dünner, C., Jaggi, M. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Curran Associates, Inc., The Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS), December 2023 (Published)
We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost the performance of individual models in the target domain from which the auxiliary data is sampled. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods. At the same time, it can probably mitigate the negative impact of bad models on the collective.
ArXiv URL BibTeX
Empirical Inference Perceiving Systems Conference Paper Controlling Text-to-Image Diffusion by Orthogonal Finetuning Qiu*, Z., Liu*, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., Schölkopf, B. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:79320-79362, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems , December 2023, *equal contribution (Published)
Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.
Home Code URL BibTeX
Empirical Inference Master Thesis Denoising Representation Learning for Causal Discovery Sakenyte, U. Université de Genèva, Switzerland, December 2023, external supervision (Published) BibTeX
Social Foundations of Computation Poster Do Personality Tests Generalize to Large Language Models Dorner, F. E., Sühr, T., Samadi, S., Kelava, A. Socially Responsible Language Modelling Research (SoLaR) Workshop, The Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS), December 2023, *equal contribution (Published)
With large language models (LLMs) appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate various properties of these models using tests originally designed for humans. While re-using existing tests is a resource-efficient way to evaluate LLMs, careful adjustments are usually required to ensure that test results are even valid across human sub-populations. Thus, it is not clear to what extent different tests’ validity generalizes to LLMs. In this work, we provide evidence that LLMs’ responses to personality tests systematically deviate from typical human responses, implying that these results cannot be interpreted in the same way as human test results. Concretely, reverse-coded items (e.g. “I am introverted” vs “I am extraverted”) are often both answered affirmatively by LLMs. In addition, variation across different prompts designed to “steer” LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe it is important to pay more attention to tests’ validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs’ “personality”.
URL BibTeX
Social Foundations of Computation Book Fairness and Machine Learning: Limitations and Opportunities Barocas, S., Hardt, M., Narayanan, A. MIT Press, December 2023 (Published)
An introduction to the intellectual foundations and practical utility of the recent work on fairness and machine learning. Fairness and Machine Learning introduces advanced undergraduate and graduate students to the intellectual foundations of this recently emergent field, drawing on a diverse range of disciplinary perspectives to identify the opportunities and hazards of automated decision-making. It surveys the risks in many applications of machine learning and provides a review of an emerging set of proposed solutions, showing how even well-intentioned applications may give rise to objectionable results. It covers the statistical and causal measures used to evaluate the fairness of machine learning models as well as the procedural and substantive aspects of decision-making that are core to debates about fairness, including a review of legal and philosophical perspectives on discrimination. This incisive textbook prepares students of machine learning to do quantitative work on fairness while reflecting critically on its foundations and its practical utility.• Introduces the technical and normative foundations of fairness in automated decision-making• Covers the formal and computational methods for characterizing and addressing problems• Provides a critical assessment of their intellectual foundations and practical utility• Features rich pedagogy and extensive instructor resources
URL BibTeX
Empirical Inference Conference Paper Flow Matching for Scalable Simulation-Based Inference Wildberger*, J., Dax*, M., Buchholz*, S., Green, S. R., Macke, J. H., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:16837-16864, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Perceiving Systems Article From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans Keller, M., Werling, K., Shin, S., Delp, S., Pujades, S., Liu, C. K., Black, M. J. ACM Transactions on Graphics (TOG), ACM Transactions on Graphics (TOG), 42(6):253:1-253:15, ACM New York, NY, USA, December 2023 (Published)
Great progress has been made in estimating 3D human pose and shape from images and video by training neural networks to directly regress the parameters of parametric human models like SMPL. However, existing body models have simplified kinematic structures that do not correspond to the true joint locations and articulations in the human skeletal system, limiting their potential use in biomechanics. On the other hand, methods for estimating biomechanically accurate skeletal motion typically rely on complex motion capture systems and expensive optimization methods. What is needed is a parametric 3D human model with a biomechanically accurate skeletal structure that can be easily posed. To that end, we develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton. To enable this, we need training data of skeletons inside SMPL meshes in diverse poses. We build such a dataset by optimizing biomechanically accurate skeletons inside SMPL meshes from AMASS sequences. We then learn a regressor from SMPL mesh vertices to the optimized joint locations and bone rotations. Finally, we re-parametrize the SMPL mesh with the new kinematic parameters. The resulting SKEL model is animatable like SMPL but with fewer, and biomechanically-realistic, degrees of freedom. We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods. By fitting SKEL to SMPL meshes we are able to “upgrade" existing human pose and shape datasets to include biomechanical parameters. SKEL provides a new tool to enable biomechanics in the wild, while also providing vision and graphics researchers with a better constrained
Project Page Paper DOI URL BibTeX
Empirical Inference Conference Paper Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation Gao*, R., Deistler*, M., Macke, J. H. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:80191-80219, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Autonomous Learning Conference Paper Goal-conditioned Offline Planning from Curious Exploration Bagatella, M., Martius, G. In Advances in Neural Information Processing Systems 36, December 2023 (Published)
Curiosity has established itself as a powerful exploration strategy in deep reinforcement learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, as well as a robust dynamics model. We consider the challenge of extracting goal-conditioned behavior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal-conditioned value functions, we relate this issue to a specific class of estimation artifacts in learned values. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.
URL BibTeX
Empirical Inference Conference Paper Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures Eschenhagen, R., Immer, A., Turner, R., Schneider, F., Hennig, P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:33624-33655, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Endogenous Representation in Reinforcement Learning via Advantage Estimation Pan, H., Schölkopf, B. Causal Representation Learning Workshop at NeurIPS 2023, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Layer-wise Equivariances Automatically using Gradients van der Ouderaa, T., Immer, A., van der Wilk, M. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:28365-28377, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Linear Causal Representations from Interventions under General Nonlinear Mixing Buchholz*, S., Rajendran*, G., Rosenfeld, E., Aragam, B., Schölkopf, B., Ravikumar, P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:45419-45462, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Meta-learning families of plasticity rules in recurrent spiking networks using simulation-based inference Confavreux*, B., Ramesh*, P., Goncalves, P. J., Macke, J. H., Vogels, T. P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:13545-13558, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Article Multimodal learning in clinical proteomics: enhancing antimicrobial resistance prediction models with chemical information Visonà, G., Duroux, D., Miranda, L., Sükei, E., Li, Y., Borgwardt, K., Oliver, C. Bioinformatics, 39(12), December 2023 (Published) DOI BibTeX
Empirical Inference Conference Paper Neural Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning Munkhoeva, M., Oseledets, I. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:60712-60723, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Nonparametric Identifiability of Causal Representations from Unknown Interventions von Kügelgen, J., Besserve, M., Liang, W., Gresele, L., Kekić, A., Bareinboim, E., Blei, D., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:48603-48638, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Nonparametric Teaching for Multiple Learners Zhang, C., Cao, X., Liu, W., Tsang, I. W., Kwok, J. T. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:7756-7786, (Editors: A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Autonomous Learning Conference Paper Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities Zadaianchuk, A., Seitzer, M., Martius, G. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), Advances in Neural Information Processing Systems 36, December 2023
Unsupervised video-based object-centric learning is a promising avenue to learn structured representations from large, unlabeled video collections, but previous approaches have only managed to scale to real-world datasets in restricted domains. Recently, it was shown that the reconstruction of pre-trained self-supervised features leads to object-centric representations on unconstrained real-world image datasets. Building on this approach, we propose a novel way to use such pre-trained features in the form of a temporal feature similarity loss. This loss encodes semantic and temporal correlations between image patches and is a natural way to introduce a motion bias for object discovery. We demonstrate that this loss leads to state-of-the-art performance on the challenging synthetic MOVi datasets. When used in combination with the feature reconstruction loss, our model is the first object-centric video model that scales to unconstrained video datasets such as YouTube-VIS.
arXiv Website OpenReview URL BibTeX