Publications

Empirical Inference Conference Paper Neural Posterior Estimation of Terrain Parameters from Radar Sounder Data Dal Corso, J., Kofler, A., Cortellazzi, M., Bruzzone, L., Schölkopf, B. IEEE International Geoscience and Remote Sensing Symposium (IGARSS), August 2026 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Echoes of the Prior: A Computational Phenomenology of Forgetting Gao, G., Schölkopf, B., Geiger, A. Proceedings of the ACM on Computer Graphics and Interactive Techniques: PACM-CGIT, SIGGRAPH, July 2026 (Accepted) Project BibTeX

Empirische Inferenz Conference Paper On the Emergence and Test-Time Use of Structural Information in Large Language Models Chen, M. C., Miller, M., Schölkopf, B., Guo, S. 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), July 2026 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control Paulus*, A., Geist*, A. R., Schumacher*, P., Rappenecker, S., Musil, V., Martius, G. The Fourteenth International Conference on Learning Representations (ICLR), April 2026, *equal contribution (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Estimating Joint Interventional Distributions from Marginal Interventional Data Garrido Mejia, S., Kirschbaum, E., Kekić, A., Schölkopf, B., Mastakouri, A. A. Proceedings of the Fifth Conference on Causal Learning and Reasoning, 323:1-23, PMLR, 5th Conference on Causal Learning and Reasoning, April 2026 (To be published) arXiv BibTeX

Empirical Inference Conference Paper Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies Kekić, A., Schneider, J., Büchler, D., Schölkopf*, B., Besserve*, M. The Fourteenth International Conference on Learning Representations (ICLR), April 2026, *joint supervision (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Position: Science is Collaborative—LLM for Science Should Be Too Zhang, T. J., Jiang, W., Guzman Piedrahita, D., Yang, Y., Lu, S., Schölkopf, B., Jin, Z. ICLR 2026 – 2nd Workshop on Foundation Models for Science: Real-World Impact and Science-First Design , ICLR - Workshop FM4Science, April 2026 (Published) URL BibTeX

Empirical Inference Conference Paper Proper Velocity Neural Networks Chen*, Z., Su*, Z., Schölkopf, B., Sebe, N. The Fourteenth International Conference on Learning Representations (ICLR), April 2026, *equal contribution (Published) URL BibTeX

Empirical Inference Conference Paper Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles Ni*, Z., Li*, Y., Qiu*, Z., Schölkopf, B., Guo, H., Liu, W., Liu, S. The Fourteenth International Conference on Learning Representations (ICLR), April 2026, *equal contribution (Published) arXiv URL BibTeX

Empirical Inference Deep Models and Optimization Conference Paper Scaling Behavior of Discrete Diffusion Language Models von Rütte, D., Fluri, J., Pooladzandi, O., Schölkopf, B., Hofmann, T., Orvieto, A. The Fourteenth International Conference on Learning Representations (ICLR), April 2026 (Published) arXiv URL BibTeX

Empirical Inference Robust Machine Learning Conference Paper Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning Reizinger*, P., Mucsányi*, B., Guo*, S., Eysenbach, B., Schölkopf, B., Brendel, W. The Fourteenth International Conference on Learning Representations (ICLR), April 2026, *equal contribution (Published) arXiv URL BibTeX

Empirical Inference Conference Paper SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests Pandey, P. S., Le, H. S., Bhardwaj, D., Mihalcea, R., Zhijing, J. The Fourteenth International Conference on Learning Representations (ICLR), April 2026 (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Democratic or Authoritarian? Probing a New Dimension of Political Biases in Large Language Models Guzman Piedrahita*, D., Strauss*, I., Schölkopf, B., Mihalcea, R., Jin, Z. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 593-652, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026, *equal contribution (Published) DOI URL BibTeX

Empirical Inference Conference Paper How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities Kassem, A. M., Schölkopf, B., Jin, Z. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 7496-7507, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper Sim-to-Real for Muscle-Actuated Robots via Learned Actuator Models Schneider, J., Mahajan, M., Chen, L., Guist, S., Schölkopf, B., Posner, I., Büchler, D. 2nd German Robotics Conference (GRC), March 2026 (Published) URL BibTeX

Empirical Inference Conference Paper Taming Object Hallucinations with Verified Atomic Confidence Estimation Liu, J., Xuan, W., Jin, Z., Diab, M. T. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 5430-5444, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification He, P., Huang, Y., Sachan, M., Jin, Z. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 1231-1250, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper When Do Language Models Endorse Limitations on Human Rights Principles? Samway, K., Takagi, M. N., Mihalcea, R., Schölkopf, B., Chalkidis, I., Hershcovich, D., Jin, Z. Findings of the Association for Computational Linguistics: EACL, 6597-6623, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures Pandey, P. S., Yang, Y., Liu, J., Jin, Z. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 1251-1266, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper NLP for Social Good: A Survey and Outlook of Challenges, Opportunities and Responsible Deployment Karamolegkou, A., Borah, A., Cho, E., Choudhury, S. R., Galletti, M., Gupta, P., Ignat, O., Kargupta, P., Kotonya, N., Lamba, H., Lee, S., Mangla, A., Mondal, I., Moudakir, F. Z., Nazar, D., Nemkova, P., Pisarevskaya, D., Rizwan, N., Sabri, N., Samway, K., et al. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 5110-5170, (Editors: Demberg, Vera and Inui, Kentaro and Marquez, Lluís), Association for Computational Linguistics, 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL), March 2026 (Published) DOI URL BibTeX

Empirical Inference Conference Paper A data and task-constrained mechanistic model of the mouse outer retina shows robustness to contrast variations Kadhim, K. L., Beck, J., Huang, Z., Macke, J. H., Rieke, F., Euler, T., Deistler, M., Berens, P. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) bioRxiv BibTeX

Empirical Inference Conference Paper Are Language Models Efficient Reasoners? A Perspective from Logic Programming Opedal, A., Zengaffinen, Y., Shirakami, H., Pasti, C., Sachan, M., Saparov, A., Cotterell, R., Schölkopf, B. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper CauSciBench: Assessing LLM Causal Reasoning for Scientific Research Acharya, S., Zhang, T. J., Kim, A., Haghighat, A., Sun, X., Shrestha, R. B., Mordig, M., Danisman, F., Jose, C., Qi, Y., Cobben, P., Schölkopf, B., Sachan, M., Jin, Z. NeurIPS 2025: 5th Workshop on Mathematical Reasoning and AI (Math-AI) and CauScien Workshop, December 2025 (Published) URL BibTeX

Empirical Inference Conference Paper Counterfactual reasoning: an analysis of in-context emergence Miller, M., Schölkopf, B., Guo, S. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Cultural Alien Sampler: Open-ended art generation balancing originality and coherence Hernandez, A., Yakura, H., Brinkmann, L., Sola, M. C., Alhaija, H. A., Serna, I., Rahaman, N., Schölkopf, B., Rahwan, I. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, Creative AI Track, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Do-PFN: In-Context Learning for Causal Effect Estimation Robertson*, J., Reuter*, A., Guo, S., Hollmann, N., Hutter, F., Schölkopf, B. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025, *equal contribution (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models Vetter, J., Gloeckler, M., Gedon, D., Macke, J. H. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper FNOPE: Simulation-based inference on function spaces with Fourier Neural Operators Moss, G., Muhle, L. S., Drews, R., Macke, J. H., Schröder, C. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Forecasting in Offline Reinforcement Learning for Non-stationary Environments Ada, S. E., Martius, G., Ugur, E., Oztop, E. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Identifying multi-compartment Hodgkin-Huxley models with high-density extracellular voltage recordings Tanoh, I. C., Deistler, M., Macke, J. H., Linderman, S. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Reparameterized LLM Training via Orthogonal Equivalence Transformation Qiu, Z., Buchholz, S., Xiao, T., Dax, M., Schölkopf, B., Liu, W. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Root Cause Analysis of Outliers with Missing Structural Knowledge Orchard, W. R., Okati, N., Garrido Mejia, S., Blöbaum, P., Janzing, D. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper SPARTAN: A Sparse Transformer World Model Attending to What Matters Lei, A., Schölkopf, B., Posner, I. Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 39th Annual Conference on Neural Information Processing Systems, December 2025 (Accepted) arXiv BibTeX

Empirical Inference Conference Paper Are Language Models Consequentialist or Deontological Moral Reasoners? Samway, K., Kleiman-Weiner, M., Guzman Piedrahita, D., Mihalcea, R., Schölkopf, B., Jin, Z. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 30699-30726, (Editors: Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet), Association for Computational Linguistics, EMNLP, November 2025 (Published)

Abstract ›

As AI systems increasingly navigate applications in healthcare, law, and governance, understanding how they handle ethically complex scenarios becomes critical. Previous work has mainly examined the moral judgments in large language models (LLMs), rather than their underlying moral reasoning process. In contrast, we focus on a large-scale analysis of the moral reasoning traces provided by LLMs. Furthermore, unlike prior work that attempted to draw inferences from only a handful of moral dilemmas, our study leverages over 600 distinct trolley problems as probes for revealing the reasoning patterns that emerge within different LLMs. We introduce and test a taxonomy of moral rationales to systematically classify reasoning traces according to two main normative ethical theories: consequentialism and deontology. Our analysis reveals that LLM chains-of-thought favor deontological principles based on moral obligations, while post-hoc explanations shift notably toward consequentialist rationales that emphasize utility. Our framework provides a foundation for understanding how LLMs process and articulate ethical considerations, an important step toward safe and interpretable deployment of LLMs in high-stakes decision-making environments."

DOI URL BibTeX

Empirical Inference Conference Paper Improving Large Language Model Safety with Contrastive Representation Learning Simko, S., Sachan, M., Schölkopf, B., Jin, Z. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 28166-28194, (Editors: Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet), Association for Computational Linguistics, November 2025 (Published) arXiv DOI URL BibTeX

Empirical Inference Conference Paper Corrupted by reasoning: Reasoning language models become free-riders in public goods games Guzman Piedrahita, D., Yang, Y., Sachan, M., Ramponi, G., Schölkopf, B., Jin, Z. Second Conference on Language Modeling (COLM 2025), October 2025 (Published) arXiv URL BibTeX

Haptic Intelligence Autonomous Learning Empirical Inference Conference Paper Adding Internal Audio Sensing to Internal Vision Enables Human-Like In-Hand Fabric Recognition with Soft Robotic Fingertips Andrussow, I., Solano, J., Richardson, B. A., Martius, G., Kuchenbecker, K. J. In Proceedings of the IEEE-RAS International Conference on Humanoid Robots (Humanoids), 373-380, Seoul, South Korea, September 2025 (Published)

Abstract ›

Distinguishing the feel of smooth silk from coarse cotton is a trivial everyday task for humans. When exploring such fabrics, fingertip skin senses both spatio-temporal force patterns and texture-induced vibrations that are integrated to form a haptic representation of the explored material. It is challenging to reproduce this rich, dynamic perceptual capability in robots because tactile sensors typically cannot achieve both high spatial resolution and high temporal sampling rate. In this work, we present a system that can sense both types of haptic information, and we investigate how each type influences robotic tactile perception of fabrics. Our robotic hand's middle finger and thumb each feature a soft tactile sensor: one is the open- source Minsight sensor that uses an internal camera to measure fingertip deformation and force at 50 Hz, and the other is our new sensor Minsound that captures vibrations through an internal MEMS microphone with a bandwidth from 50 Hz to 15 kHz. Inspired by the movements humans make to evaluate fabrics, our robot actively encloses and rubs folded fabric samples between its two sensitive fingers. Our results test the influence of each sensing modality on overall classification performance, showing high utility for the audio-based sensor. Our transformer-based method achieves a maximum fabric classification accuracy of 97% on a dataset of 20 common fabrics. Incorporating an external microphone away from Minsound increases our method's robustness in loud ambient noise conditions. To show that this audio-visual tactile sensing approach generalizes beyond the training data, we learn general representations of fabric stretchiness, thickness, and roughness.

DOI BibTeX

Empirical Inference Conference Paper Active Fine-Tuning of Multi-Task Policies Bagatella, M., Hübotter, J., Martius, G., Krause, A. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:2409-2441, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) arXiv URL BibTeX

Empirical Inference Deep Models and Optimization Conference Paper Generalized Interpolating Discrete Diffusion von Rütte, D., Fluri, J., Ding, Y., Orvieto, A., Schölkopf, B., Hofmann, T. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:61810-61843, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Generative Intervention Models for Causal Perturbation Modeling Schneider, N., Lorch, L., Kilbertus, N., Schölkopf, B., Krause, A. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:53388-53412, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Learning Joint Interventional Effects from Single-Variable Interventions in Additive Models Kekić, A., Garrido Mejia, S., Schölkopf, B. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:29651-29669, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) arXiv URL BibTeX

Empirical Inference Conference Paper Position: Probabilistic Modelling is Sufficient for Causal Inference Mlodozeniec, B. K., Krueger, D., Turner, R. E. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:81810-81840, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) URL BibTeX

Empirical Inference Conference Paper Progressive Tempering Sampler with Diffusion Rissanen*, S., OuYang*, R., He*, J., Chen, W., Heinonen, M., Solin, A., Hernández-Lobato, J. M. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:51724-51746, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025, *equal contribution (Published) arXiv URL BibTeX

Empirical Inference Autonomous Learning Conference Paper SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models Sancaktar, C., Gumbsch, C., Zadaianchuk, A., Kolev, P., Martius, G. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:52745-52777, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), International Conference on Machine Learning , July 2025 (Published) arXiv Project website URL BibTeX

Empirical Inference Conference Paper Scalable Gaussian Processes with Latent Kronecker Structure Lin, J. A., Ament, A., Balandat, M., Eriksson, D., Hernández-Lobato, J. M., Bakshy, E. Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:37730-37744, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published) arXiv URL BibTeX

Autonomous Learning Empirical Inference Conference Paper Zero-Shot Offline Imitation Learning via Optimal Transport Rupf, T., Bagatella, M., Gürtler, N., Frey, J., Martius, G. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 267:52345-52381, Proceedings of Machine Learning Research, (Editors: Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry), PMLR, International Conference on Machine Learning, July 2025 (Published)

Abstract ›

Zero-shot imitation learning algorithms hold the promise of reproducing unseen behavior from as little as a single demonstration at test time. Existing practical approaches view the expert demonstration as a sequence of goals, enabling imitation with a high-level goal selector, and a low-level goal-conditioned policy. However, this framework can suffer from myopic behavior: the agent's immediate actions towards achieving individual goals may undermine long-term objectives. We introduce a novel method that mitigates this issue by directly optimizing the occupancy matching objective that is intrinsic to imitation learning. We propose to lift a goal-conditioned value function to a distance between occupancies, which are in turn approximated via a learned world model. The resulting method can learn from offline, suboptimal data, and is capable of non-myopic, zero-shot imitation, as we demonstrate in complex, continuous benchmarks.

arXiv URL BibTeX

Empirical Inference Conference Paper Temporally Consistent Object-Centric Learning by Contrasting Slots Manasyan, A., Seitzer, M., Radovic, F., Martius, G., Zadaianchuk, A. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5401-5411, June 2025 (Published) DOI BibTeX

Empirical Inference Conference Paper VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models Ye, M., Liu, W., He, P. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8679-8688, June 2025 (Published) DOI BibTeX

Empirical Inference Perceiving Systems Conference Paper ChatHuman: Chatting about 3D Humans with Tools Lin, J., Feng, Y., Liu, W., Black, M. J. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8150-8161, June 2025 (Published)

Abstract ›

Numerous methods have been proposed to detect, estimate, and analyze properties of people in images, including 3D pose, shape, contact, human-object interaction, and emotion. While widely applicable in vision and other areas, such methods require expert knowledge to select, use, and interpret the results. To address this, we introduce ChatHuman, a language-driven system that integrates the capabilities of specialized methods into a unified framework. ChatHuman functions as an assistant proficient in utilizing, analyzing, and interacting with tools specific to 3D human tasks, adeptly discussing and resolving related challenges. Built on a Large Language Model (LLM) framework, ChatHuman is trained to autonomously select, apply, and interpret a diverse set of tools in response to user inputs. Our approach overcomes significant hurdles in adapting LLMs to 3D human tasks, including the need for domain-specific knowledge and the ability to interpret complex 3D outputs. The innovations of ChatHuman include leveraging academic publications to instruct the LLM on tool usage, employing a retrieval-augmented generation model to create in-context learning examples for managing new tools, and effectively discriminating between and integrating tool results by transforming specialized 3D outputs into comprehensible formats. Experiments demonstrate that ChatHuman surpasses existing models in both tool selection accuracy and overall performance across various 3D human tasks, and it supports interactive chatting with users. ChatHuman represents a significant step toward consolidating diverse analytical methods into a unified, robust system for 3D human tasks.

project pdf Paper DOI BibTeX

Empirical Inference Conference Paper Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation Sanyal, A., Hu, Y., Yu, Y., Ma, Y., Wang, Y., Schölkopf, B. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), 258:2170-2178, Proceedings of Machine Learning Research, (Editors: Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz), PMLR, May 2025 (Published) URL BibTeX

Publications

Filter by