2023 Intelligent Systems Summer Colloquium & Summer Party

Biography

Ruth Ley received a BA in Integrative Biology from the University of California at Berkeley, a PhD from the University of Colorado, Boulder, and then received an NRC-NASA Fellowship for post-doctoral work with Dr. Norman Pace at CU Boulder. She moved to Washington University School of Medicine to work with Dr. Jeffrey Gordon on the human microbiome in 2004. She was named an Instructor in 2005 and a Research Assistant Professor at Washington University School of Medicine in 2007. In July 2008, Ley joined the Department of Microbiology at Cornell University as an Assistant Professor, and in 2013 became an Associate Professor with tenure in the Department of Molecular Biology and Genetics. She is currently the Director of the Department of Microbiome Science at the Max Planck Institute for Developmental Biology in Tübingen (2016). Since 2018 Ley is a speaker for the Center of Excellence award “Controlling Microbiomes to Fight Infections” with the University of Tübingen.

Abstract

In this talk, we present recent progress on large-scale learning of multimodal video representations. We begin by presenting VideoBert, a joint model for video and language, repurposing the Bert model for multimodal data. This model achieves state-of-the-art results for zero-shot prediction and video captioning. Next, we introduce Vid2Seq, a model for dense video captioning that takes as input video and speech and predicts both temporal boundaries and textual descriptions simultaneously. We then present an approach for video question answering and image captioning that relies on a retrieval-augmented visual language model that learns to encode world knowledge into a large-scale memory and to retrieve from it to answer knowledge-intensive queries. We show that our approach achieves state-of-the-art results on visual question answering and image captioning. We conclude the presentation with recent work on vision guided navigation and robot manipulation given language instructions. This work builds on and extends vision-language transformers by integrating action history and predicting actions. We outperform the state of the art on different vision-language-navigation benchmarks and RLBench, a benchmark for robot manipulation.

Biography

Cordelia Schmid holds a M.S. degree in Computer Science from the University of Karlsruhe and a Doctorate in Computer Science, from the Institut National Polytechnique de Grenoble (INPG). Her doctoral thesis on "Local Greyvalue Invariants for Image Matching and Retrieval" received the best thesis award from INPG in 1996. She received the Habilitation degree in 2001 for her thesis entitled "From Image Matching to Learning Visual Models". Dr. Schmid was a post-doctoral research assistant in the Robotics Research Group of Oxford University in 1996--1997. Since 1997 she has held a permanent research position at Inria, where she is a research director. Dr. Schmid is a member of the German National Academy of Sciences, Leopoldina and a fellow of IEEE and the ELLIS society. She was awarded the Longuet-Higgins prize in 2006, 2014 and 2016 and the Koenderink prize in 2018, both for fundamental contributions in computer vision that have withstood the test of time. She received an ERC advanced grant in 2013, the Humbolt research award in 2015, the Inria & French Academy of Science Grand Prix in 2016, the Royal Society Milner award in 2020, the PAMI distinguished researcher award in 2021 and the Körber European Science Price in 2023. Dr. Schmid has been an Associate Editor for IEEE PAMI (2001--2005) and for IJCV (2004--2012), an editor-in-chief for IJCV (2013--2018), a program chair of IEEE CVPR 2005 and ECCV 2012 as well as a general chair of IEEE CVPR 2015, ECCV 2020 and ICCV 2023. Starting 2018 she holds a joint appointment with Google research.

Abstract

I will present our recent work on developing intrinsically motivated exploration strategies for robots to learn by playing to manipulate objects and to master the control of musculoskeletal models of ostriches and humans. We will see that exploration is crucial to obtain good results. In the talk, I will present a line of research on model-based reinforcement learning that enables zero-shot generalization to new tasks and is a very promising route to efficient learning on real robots. The control of high-dimensional systems is relevant in robotics and in understanding human motor control. We recently achieved learning of natural behavior in realistic musculoskeletal simulations with reinforcement learning and the help of a suitable exploration strategy. I will share some illustrative and funny videos with you and elaborate on the potential of this result.

Biography

Georg Martius is a full professor at the University of Tübingen in Computer Science since 2023. His also leading a research group on Autonomous Learning at the Max Planck Institute for Intelligent Systems in Tübingen. Before joining the MPI in Tübingen, he was a postdoc fellow at the IST Austria and was a postdoctoral researcher at the Max Planck Institute for Mathematics in the Sciences in Leipzig. He received his Ph.D. from the University of Göttingen and his computer science degree from University of Leipzig. His research focus is on machine learning for robotics, including internal model learning, reinforcement learning, intrinsic motivations, representation learning, differentiable combinatorial optimization and haptics.