Institute Homepage
Institute Homepage EN Sign In

Publications

DEPARTMENTS

Emperical Interference

Haptic Intelligence

Modern Magnetic Systems

Perceiving Systems

Physical Intelligence

Robotic Materials

Social Foundations of Computation

Research Groups

Autonomous Vision

Autonomous Learning

Bioinspired Autonomous Miniature Robots

Dynamic Locomotion

Embodied Vision

Human Aspects of Machine Learning

Intelligent Control Systems

Learning and Dynamical Systems

Locomotion in Biorobotic and Somatic Systems

Micro, Nano, and Molecular Systems

Movement Generation and Control

Neural Capture and Synthesis

Physics for Inference and Optimization

Organizational Leadership and Diversity

Probabilistic Learning Group

Topics

Robot Learning

Conference Paper

2022

Autonomous Learning

Robotics

AI

Career

Award

Perceiving Systems Conference Paper Emotional Speech-Driven Animation with Content-Emotion Disentanglement Daněček, R., Chhatre, K., Tripathi, S., Wen, Y., Black, M. J., Bolkart, T. In SIGGRAPH Asia 2023 Conference Papers, Association for Computing Machinery , New York, NY, SIGGRAPH Asia, December 2023 (Published)
To be widely adopted, 3D facial avatars must be animated easily, realistically, and directly from speech signals. While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions. Realistic facial animation requires lip-sync together with the natural expression of emotion. To that end, we propose EMOTE (Expressive Model Optimized for Talking with Emotion), which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. To achieve this, we supervise EMOTE with decoupled losses for speech (i.e., lip-sync) and emotion. These losses are based on two key observations: (1) deformations of the face due to speech are spatially localized around the mouth and have high temporal frequency, whereas (2) facial expressions may deform the whole face and occur over longer intervals. Thus, we train EMOTE with a per-frame lip-reading loss to preserve the speech-dependent content, while supervising emotion at the sequence level. Furthermore, we employ a content-emotion exchange mechanism in order to supervise different emotions on the same audio, while maintaining the lip motion synchronized with the speech. To employ deep perceptual losses without getting undesirable artifacts, we devise a motion prior in the form of a temporal VAE. Due to the absence of high-quality aligned emotional 3D face datasets with speech, EMOTE is trained with 3D pseudo-ground-truth extracted from an emotional video dataset (i.e., MEAD). Extensive qualitative and perceptual evaluations demonstrate that EMOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data, while offering additional, high-quality emotional control.
arXiv DOI URL BibTeX
Autonomous Learning Conference Paper On Imitation in Mean-field Games Ramponi, G., Kolev, P., Olivier, P., He, N., Laurière, M., Geist, M. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 1-12, Curran Associates Inc., NeurIPS, December 2023 (Published)
We explore the problem of imitation learning (IL) in the context of mean-field games (MFGs), where the goal is to imitate the behavior of a population of agents following a Nash equilibrium policy according to some unknown payoff function. IL in MFGs presents new challenges compared to single-agent IL, particularly when both the reward function and the transition kernel depend on the population distribution. In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap. Then we show that when only the reward depends on the population distribution, IL in MFGs can be reduced to single-agent IL with similar guarantees. However, when the dynamics is population-dependent, we provide a novel upper-bound that suggests IL is harder in this setting. To address this issue, we propose a new adversarial formulation where the reinforcement learning problem is replaced by a mean-field control (MFC) problem, suggesting progress in IL within MFGs may have to build upon MFC.
DOI URL BibTeX
Empirical Inference Conference Paper Optimistic Active Exploration of Dynamical Systems Sukhija, B., Treven, L., Sancaktar, C., Blaes, S., Coros, S., Krause, A. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 1-32, Curran Associates Inc. , NeurIPS, December 2023 (Published) DOI URL BibTeX
Empirical Inference Article Machine-Learning-Aided Prediction of Brain Metastases Development in Non-Small-Cell Lung Cancers Visonà, G., Spiller, L. M., Hahn, S., Hattingen, E., Vogl, T. J., Schweikert, G., Bankov, K., Demes, M., Reis, H., Wild, P., Zeiner, P. S., Acker, F., Sebastian, M., Wenger, K. J. Clinical lung cancer, 24(8):e311-e322, December 2023 (Published) DOI BibTeX
Empirical Inference Conference Paper A Measure-Theoretic Axiomatisation of Causality Park, J., Buchholz, S., Schölkopf, B., Muandet, K. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:28510-28540, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Beyond Good Intentions: Reporting the Research Landscape of NLP for Social Good Gonzalez*, F., Jin*, Z., Schölkopf, B., Hope, T., Sachan, M., Mihalcea, R. Findings of the Association for Computational Linguistics: EMNLP 2023, 415-438, (Editors: Houda Bouamor and Juan Pino and Kalika Bali), Association for Computational Linguistics, December 2023, *equal contribution (Published) DOI BibTeX
Physics for Inference and Optimization Article Bilevel Optimization for Traffic Mitigation in Optimal Transport Networks Lonardi, A., De Bacco, C. Physical Review Letter, 131:267401, December 2023 (Published) Preprint Code Paper DOI URL BibTeX
Empirical Inference Conference Paper CLadder: Assessing Causal Reasoning in Language Models Jin*, Z., Chen*, Y., Leeb*, F., Gresele*, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez, F., Kleiman-Weiner, M., Sachan, M., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:31038-31065, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *main contributors (Published) URL BibTeX
Empirical Inference Conference Paper Can semi-supervised learning use all the data effectively? A lower bound perspective Tifrea*, A., Yüce*, G., Sanyal, A., Yang, F. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:21960-21982, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Causal Component Analysis Liang, W., Kekić, A., von Kügelgen, J., Buchholz, S., Besserve, M., Gresele*, L., Schölkopf*, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:32481-32520, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *shared last author (Published) URL BibTeX
Empirical Inference Conference Paper Causal Modeling with Stationary Diffusions Lorch, L., Krause*, A., Schölkopf*, B. Causal Representation Learning Workshop at NeurIPS 2023, December 2023, *equal supervision (Published) URL BibTeX
Empirical Inference Conference Paper Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data Guo*, S., Tóth*, V., Schölkopf, B., Huszár, F. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:36463-36475, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Causal normalizing flows: from theory to practice Javaloy, A., Sanchez-Martin, P., Valera, I. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:58833-58864, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Social Foundations of Computation Algorithms and Society Conference Paper Collaborative Learning via Prediction Consensus Fan, D., Mendler-Dünner, C., Jaggi, M. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Curran Associates, Inc., The Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS), December 2023 (Published)
We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost the performance of individual models in the target domain from which the auxiliary data is sampled. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods. At the same time, it can probably mitigate the negative impact of bad models on the collective.
ArXiv URL BibTeX
Empirical Inference Perceiving Systems Conference Paper Controlling Text-to-Image Diffusion by Orthogonal Finetuning Qiu*, Z., Liu*, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., Schölkopf, B. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:79320-79362, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems , December 2023, *equal contribution (Published)
Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.
Home Code URL BibTeX
Empirical Inference Master Thesis Denoising Representation Learning for Causal Discovery Sakenyte, U. Université de Genèva, Switzerland, December 2023, external supervision (Published) BibTeX
Social Foundations of Computation Poster Do Personality Tests Generalize to Large Language Models Dorner, F. E., Sühr, T., Samadi, S., Kelava, A. Socially Responsible Language Modelling Research (SoLaR) Workshop, The Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS), December 2023, *equal contribution (Published)
With large language models (LLMs) appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate various properties of these models using tests originally designed for humans. While re-using existing tests is a resource-efficient way to evaluate LLMs, careful adjustments are usually required to ensure that test results are even valid across human sub-populations. Thus, it is not clear to what extent different tests’ validity generalizes to LLMs. In this work, we provide evidence that LLMs’ responses to personality tests systematically deviate from typical human responses, implying that these results cannot be interpreted in the same way as human test results. Concretely, reverse-coded items (e.g. “I am introverted” vs “I am extraverted”) are often both answered affirmatively by LLMs. In addition, variation across different prompts designed to “steer” LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe it is important to pay more attention to tests’ validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs’ “personality”.
URL BibTeX
Social Foundations of Computation Book Fairness and Machine Learning: Limitations and Opportunities Barocas, S., Hardt, M., Narayanan, A. MIT Press, December 2023 (Published)
An introduction to the intellectual foundations and practical utility of the recent work on fairness and machine learning. Fairness and Machine Learning introduces advanced undergraduate and graduate students to the intellectual foundations of this recently emergent field, drawing on a diverse range of disciplinary perspectives to identify the opportunities and hazards of automated decision-making. It surveys the risks in many applications of machine learning and provides a review of an emerging set of proposed solutions, showing how even well-intentioned applications may give rise to objectionable results. It covers the statistical and causal measures used to evaluate the fairness of machine learning models as well as the procedural and substantive aspects of decision-making that are core to debates about fairness, including a review of legal and philosophical perspectives on discrimination. This incisive textbook prepares students of machine learning to do quantitative work on fairness while reflecting critically on its foundations and its practical utility.• Introduces the technical and normative foundations of fairness in automated decision-making• Covers the formal and computational methods for characterizing and addressing problems• Provides a critical assessment of their intellectual foundations and practical utility• Features rich pedagogy and extensive instructor resources
URL BibTeX
Empirical Inference Conference Paper Flow Matching for Scalable Simulation-Based Inference Wildberger*, J., Dax*, M., Buchholz*, S., Green, S. R., Macke, J. H., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:16837-16864, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Perceiving Systems Article From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans Keller, M., Werling, K., Shin, S., Delp, S., Pujades, S., Liu, C. K., Black, M. J. ACM Transactions on Graphics (TOG), ACM Transactions on Graphics (TOG), 42(6):253:1-253:15, ACM New York, NY, USA, December 2023 (Published)
Great progress has been made in estimating 3D human pose and shape from images and video by training neural networks to directly regress the parameters of parametric human models like SMPL. However, existing body models have simplified kinematic structures that do not correspond to the true joint locations and articulations in the human skeletal system, limiting their potential use in biomechanics. On the other hand, methods for estimating biomechanically accurate skeletal motion typically rely on complex motion capture systems and expensive optimization methods. What is needed is a parametric 3D human model with a biomechanically accurate skeletal structure that can be easily posed. To that end, we develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton. To enable this, we need training data of skeletons inside SMPL meshes in diverse poses. We build such a dataset by optimizing biomechanically accurate skeletons inside SMPL meshes from AMASS sequences. We then learn a regressor from SMPL mesh vertices to the optimized joint locations and bone rotations. Finally, we re-parametrize the SMPL mesh with the new kinematic parameters. The resulting SKEL model is animatable like SMPL but with fewer, and biomechanically-realistic, degrees of freedom. We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods. By fitting SKEL to SMPL meshes we are able to “upgrade" existing human pose and shape datasets to include biomechanical parameters. SKEL provides a new tool to enable biomechanics in the wild, while also providing vision and graphics researchers with a better constrained
Project Page Paper DOI URL BibTeX
Empirical Inference Conference Paper Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation Gao*, R., Deistler*, M., Macke, J. H. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:80191-80219, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Autonomous Learning Conference Paper Goal-conditioned Offline Planning from Curious Exploration Bagatella, M., Martius, G. In Advances in Neural Information Processing Systems 36, December 2023 (Published)
Curiosity has established itself as a powerful exploration strategy in deep reinforcement learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, as well as a robust dynamics model. We consider the challenge of extracting goal-conditioned behavior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal-conditioned value functions, we relate this issue to a specific class of estimation artifacts in learned values. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.
URL BibTeX
Empirical Inference Conference Paper Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures Eschenhagen, R., Immer, A., Turner, R., Schneider, F., Hennig, P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:33624-33655, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Endogenous Representation in Reinforcement Learning via Advantage Estimation Pan, H., Schölkopf, B. Causal Representation Learning Workshop at NeurIPS 2023, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Layer-wise Equivariances Automatically using Gradients van der Ouderaa, T., Immer, A., van der Wilk, M. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:28365-28377, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Learning Linear Causal Representations from Interventions under General Nonlinear Mixing Buchholz*, S., Rajendran*, G., Rosenfeld, E., Aragam, B., Schölkopf, B., Ravikumar, P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:45419-45462, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Meta-learning families of plasticity rules in recurrent spiking networks using simulation-based inference Confavreux*, B., Ramesh*, P., Goncalves, P. J., Macke, J. H., Vogels, T. P. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:13545-13558, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Article Multimodal learning in clinical proteomics: enhancing antimicrobial resistance prediction models with chemical information Visonà, G., Duroux, D., Miranda, L., Sükei, E., Li, Y., Borgwardt, K., Oliver, C. Bioinformatics, 39(12), December 2023 (Published) DOI BibTeX
Empirical Inference Conference Paper Neural Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning Munkhoeva, M., Oseledets, I. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:60712-60723, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Nonparametric Identifiability of Causal Representations from Unknown Interventions von Kügelgen, J., Besserve, M., Liang, W., Gresele, L., Kekić, A., Bareinboim, E., Blei, D., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:48603-48638, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Empirical Inference Conference Paper Nonparametric Teaching for Multiple Learners Zhang, C., Cao, X., Liu, W., Tsang, I. W., Kwok, J. T. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:7756-7786, (Editors: A. Oh and T. Naumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023 (Published) URL BibTeX
Autonomous Learning Conference Paper Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities Zadaianchuk, A., Seitzer, M., Martius, G. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), Advances in Neural Information Processing Systems 36, December 2023
Unsupervised video-based object-centric learning is a promising avenue to learn structured representations from large, unlabeled video collections, but previous approaches have only managed to scale to real-world datasets in restricted domains. Recently, it was shown that the reconstruction of pre-trained self-supervised features leads to object-centric representations on unconstrained real-world image datasets. Building on this approach, we propose a novel way to use such pre-trained features in the form of a temporal feature similarity loss. This loss encodes semantic and temporal correlations between image patches and is a natural way to introduce a motion bias for object discovery. We demonstrate that this loss leads to state-of-the-art performance on the challenging synthetic MOVi datasets. When used in combination with the feature reconstruction loss, our model is the first object-centric video model that scales to unconstrained video datasets such as YouTube-VIS.
arXiv Website OpenReview URL BibTeX
Empirical Inference Conference Paper On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series Kuznetsova*, R., Pace*, A., Burger*, M., Yèche, H., Rätsch, G. Proceedings of the 3rd Machine Learning for Health Symposium (ML4H) , 225:268-291, Proceedings of Machine Learning Research, (Editors: Hegselmann, S.and Parziale, A. and Shanmugam, D. and Tang, S. and Asiedu, M. N. and Chang, S. and Hartvigsen, T. and Singh, H.), PMLR, December 2023, *equal contribution (Published) URL BibTeX
Autonomous Learning Conference Paper Online Learning under Adversarial Nonlinear Constraints Kolev, P., Martius, G., Muehlebach, M. In Advances in Neural Information Processing Systems 36, December 2023 URL BibTeX
Autonomous Learning Conference Paper Regularity as Intrinsic Reward for Free Play Sancaktar, C., Piater, J., Martius, G. In Advances in Neural Information Processing Systems 37, December 2023 Website Code URL BibTeX
Empirical Inference Conference Paper SE(3) Equivariant Augmented Coupling Flows Midgley*, L. I., Stimper*, V., Antorán*, J., Mathieu*, E., Schölkopf, B., Hernández-Lobato, J. M. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:79200-79225, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published)
Coupling normalizing flows allow for fast sampling and density evaluation, making them the tool of choice for probabilistic modeling of physical systems. However, the standard coupling architecture precludes endowing flows that operate on the Cartesian coordinates of atoms with the SE(3) and permutation invariances of physical systems. This work proposes a coupling flow that preserves SE(3) and permutation equivariance by performing coordinate splits along additional augmented dimensions. At each layer, the flow maps atoms’ positions into learned SE(3) invariant bases, where we apply standard flow transformations, such as monotonic rational-quadratic splines, before returning to the original basis. Crucially, our flow preserves fast sampling and density evaluation, and may be used to produce unbiased estimates of expectations with respect to the target distribution via importance sampling. When trained on the DW4, LJ13 and QM9-positional datasets, our flow is competitive with equivariant continuous normalizing flows, while allowing sampling two orders of magnitude faster. Moreover, to the best of our knowledge, we are the first to learn the full Boltzmann distribution of alanine dipeptide by only modeling the Cartesian positions of its atoms. Lastly, we demonstrate that our flow can be trained to approximately sample from the Boltzmann distribution of the DW4 and LJ13 particle systems using only their energy functions.
arXiv URL BibTeX
Empirical Inference Conference Paper Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent Lin*, J. A., Antorán*, J., Padhy*, S., Janz, D., Hernández-Lobato, J. M., Terenin, A. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:36886-36912, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Spuriosity Didn’t Kill the Classifier: Using Invariant Predictions to Harness Spurious Features Eastwood*, C., Singh*, S., Nicolicioiu, A. L., Vlastelica, M., von Kügelgen, J., Schölkopf, B. In Advances in Neural Information Processing Systems 36, 36:18291-18324, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Conference Paper Spuriosity Didn’t Kill the Classifier: Using Invariant Predictions to Harness Spurious Features Eastwood*, C., Singh*, S., Nicolicioiu, A. L., Vlastelica, M., von Kügelgen, J., Schölkopf, B. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 36:18291-18324, (Editors: A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine), Curran Associates, Inc., 37th Annual Conference on Neural Information Processing Systems, December 2023, *equal contribution (Published) URL BibTeX
Empirical Inference Ph.D. Thesis Stochastic Predictive Control for Legged Robots Gazar, A. University of Tübingen, Germany, December 2023 (Published) DOI BibTeX
Perceiving Systems Article FLARE: Fast learning of Animatable and Relightable Mesh Avatars Bharadwaj, S., Zheng, Y., Hilliges, O., Black, M. J., Fernandez Abrevaya, V. ACM Transactions on Graphics (TOG), ACM Transactions on Graphics (TOG), 42(6):204:1-204:15, ACM New York, NY, USA, December 2023 (Published)
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. While 3D meshes enable efficient processing and are highly portable, they lack realism in terms of shape and appearance. Neural representations, on the other hand, are realistic but lack compatibility and are slow to train and render. Our key insight is that it is possible to efficiently learn high-fidelity 3D mesh representations via differentiable rendering by exploiting highly-optimized methods from traditional computer graphics and approximating some of the components with neural networks. To that end, we introduce FLARE, a technique that enables the creation of animatable and relightable mesh avatars from a single monocular video. First, we learn a canonical geometry using a mesh representation, enabling efficient differentiable rasterization and straightforward animation via learned blendshapes and linear blend skinning weights. Second, we follow physically-based rendering and factor observed colors into intrinsic albedo, roughness, and a neural representation of the illumination, allowing the learned avatars to be relit in novel scenes. Since our input videos are captured on a single device with a narrow field of view, modeling the surrounding environment light is non-trivial. Based on the split-sum approximation for modeling specular reflections, we address this by approximating the pre-filtered environment map with a multi-layer perceptron (MLP) modulated by the surface roughness, eliminating the need to explicitly model the light. We demonstrate that our mesh-based avatar formulation, combined with learned deformation, material, and lighting MLPs, produces avatars with high-quality geometry and appearance, while also being efficient to train and render compared to existing approaches.
Paper Project Page Code DOI URL BibTeX
Physical Intelligence Ph.D. Thesis Active Cargo Delivery Using Microorganism-powered Microswimmers Akolpoglu, M. B. November 2023 (Published) DOI BibTeX
Empirical Inference Learning and Dynamical Systems IT Services Robotics Conference Paper A Robust Open-source Tendon-driven Robot Arm for Learning Control of Dynamic Motions Guist, S., Schneider, J., Ma, H., Berenz, V., Martus, J., Grüninger, F., Muehlebach, M., Fiene, J., Schölkopf, B., Büchler, B. RoboLetics: Workshop on Robot Learning in Athletics @CoRL 2023, November 2023 (Published) URL BibTeX
Empirical Inference Article Data-Efficient Learning via Minimizing Hyperspherical Energy Cao, X., Liu, W., Tsang, I. W. IEEE transactions on pattern analysis and machine intelligence, 45(11):13422-13437, November 2023 (Published) DOI BibTeX
Empirical Inference Article Deep learning-based age estimation from clinical Computed Tomography image data of the thorax and abdomen in the adult population Kerber, B., Hepp, T., Küstner, T., Gatidis, S. PLOS ONE, 18(11), November 2023 (Published) DOI BibTeX
Robotic Materials Article Electrochemically Controlled Hydrogels with Electrotunable Permeability and Uniaxial Actuation Benselfelt, T., Shakya, J., Rothemund, P., Lindström, S. B., Piper, A., Winkler, T. E., Hajian, A., Wågberg, L., Keplinger, C., Hamedi, M. M. Advanced Materials, 35(45):2303255, Wiley-VCH GmbH, November 2023
The unique properties of hydrogels enable the design of life-like soft intelligent systems. However, stimuli-responsive hydrogels still suffer from limited actuation control. Direct electronic control of electronically conductive hydrogels can solve this challenge and allow direct integration with modern electronic systems. An electrochemically controlled nanowire composite hydrogel with high in-plane conductivity that stimulates a uniaxial electrochemical osmotic expansion is demonstrated. This materials system allows precisely controlled shape-morphing at only −1 V, where capacitive charging of the hydrogel bulk leads to a large uniaxial expansion of up to 300%, caused by the ingress of ≈700 water molecules per electron–ion pair. The material retains its state when turned off, which is ideal for electrotunable membranes as the inherent coupling between the expansion and mesoporosity enables electronic control of permeability for adaptive separation, fractionation, and distribution. Used as electrochemical osmotic hydrogel actuators, they achieve an electroactive pressure of up to 0.7 MPa (1.4 MPa vs dry) and a work density of ≈150 kJ m−3 (2 MJ m−3 vs dry). This new materials system paves the way to integrate actuation, sensing, and controlled permeation into advanced soft intelligent systems.
DOI URL BibTeX
Autonomous Learning Conference Paper Improving Behavioural Cloning with Positive Unlabeled Learning Wang, Q., McCarthy, R., Bulens, D. C., McGuinness, K., O’Connor, N. E., Sanchez, F. R., Gürtler, N., Widmaier, F., Redmond, S. J. 7th Annual Conference on Robot Learning (CoRL), November 2023 (Accepted) BibTeX
Empirical Inference Article Kernel-Based Independence Tests for Causal Structure Learning on Functional Data Laumann, F., von Kügelgen, J., Park, J., Schölkopf, B., Barahona, M. Entropy, 25(12), November 2023 (Published) DOI BibTeX
Empirical Inference Bachelor Thesis Navigating the Ocean of Biases: Political Bias Attribution in Language Models via Causal Structures Jenny, D. ETH Zurich, Switzerland, November 2023, external supervision (Published) BibTeX
Haptic Intelligence Article Towards Semi-Automated Pleural Cavity Access for Pneumothorax in Austere Environments L’Orsa, R., Lama, S., Westwick, D., Sutherland, G., Kuchenbecker, K. J. Acta Astronautica, 212:48-53, November 2023 (Published)
Astronauts are at risk for pneumothorax, a condition where injury or disease introduces air between the chest wall and the lungs (i.e., the pleural cavity). In a worst-case scenario, it can rapidly lead to a fatality if left unmanaged and will require prompt treatment in situ if developed during spaceflight. Chest tube insertion is the definitive treatment for pneumothorax, but it requires a high level of skill and frequent practice for safe use. Physician astronauts may struggle to maintain this skill on medium- and long-duration exploration-class missions, and it is inappropriate for pure just-in-time learning or skill refreshment paradigms. This paper proposes semi-automating tool insertion to reduce the risk of complications in austere environments and describes preliminary experiments providing initial validation of an intelligent prototype system. Specifically, we showcase and analyse motion and force recordings from a sensorized percutaneous access needle inserted repeatedly into an ex vivo tissue phantom, along with relevant physiological data simultaneously recorded from the operator. When coupled with minimal just-in-time training and/or augmented reality guidance, the proposed system may enable non-expert operators to safely perform emergency chest tube insertion without the use of ground resources.
DOI BibTeX