Events & Talks

Perceiving Systems Talk Michael Zollhoefer 27-07-2022 Complete Codec Telepresence Imagine two people, each of them within their own home, being able to communicate and interact virtually with each other as if they are both present in the same shared physical space. Enabling such an experience, i.e., building a telepresence system that is indistinguishable from reality, is one of the goals of Reality Labs Research (RLR) in Pittsburgh. To this end, we develop key technology that combines fundamental computer vision, machine learning, and graphics techniques based on a novel neural reconstruction and rendering paradigm. In this talk, I will cover our advances towards a neur... Yao Feng

Perceiving Systems Talk Rana Hanocka 13-06-2022 Shape editing, generation, and stylization Manual authoring of 3D content is a laborious and tedious task. In this talk, I present some of 3DL's recent and on-going efforts toward building tools which provide intuitive control for editing, manipulating, and generating 3D shapes. I will discuss how recent advancements, such as joint vision-language embedding spaces can be used to stylize 3D objects, driven by natural language. Finally, I will conclude with ongoing and future work in this direction, as well as other related areas. Omid Taheri

Perceiving Systems Talk Mohammed Hassan 13-06-2022 Title: Synthesizing Physical Character-Scene Interactions Movement is how people interact with and affect their environment. For realistic virtual character animation, it is necessary to realistically synthesize such interactions between virtual characters and their surroundings. Despite recent progress in character animation using machine learning, most systems focus on controlling an agent's movements in fairly simple and homogeneous environments, with limited interactions with other objects. Furthermore, many previous approaches that synthesize human-scene interaction require significant manual labeling of the training data. In contrast, we pre... Nikos Athanasiou

Perceiving Systems Talk Youngjoong Kwon 09-06-2022 Learning to create Digital Humans: Generalizable Radiance Fields for Human Performance Rendering In this work, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human, In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose a no... Yuliang Xiu

Perceiving Systems Talk Tianye Li 07-06-2022 Reconstruction and Synthesis for Dynamic Humans and Scenes This thesis focuses on automated systems to capture realistic 4D visual content for general humans and scenes, such that we can animate and replay the captured content. Firstly, we design a system to reconstruct and register a large quantity of high-quality 4D faces across identities, expressions, and poses by utilizing geometric, photometric, and motion cues. Based on well-curated datasets we propose a lightweight yet expressive face model that works on a wide range of populations, by separately modeling the shape (identity), expression, and poses of human faces. Secondly, we design an inf... Nikos Athanasiou Timo Bolkart

Perceiving Systems Talk Jiashi Feng 02-05-2022 Learning to estimate 3D human poses without labeled data Estimating 3D human poses from images or videos is a fundamental task in computer vision. However, the limitation of training data with high-quality 3D pose annotations largely hinder its development and deployment in real applications. In this talk, I will introduce our recent works on training 3D pose estimation models without requiring 3D labeled data. Our first step is to present PoseAug, a new auto-augmentation framework that learns to augment the available training poses towards a greater diversity and thus improve generalization of the trained 2D-to-3D pose estimator. Specifically, P... Michael Black

Perceiving Systems Talk Lixin Yang 25-04-2022 Leverage Kinematic and Contact constraints for understanding hand-object interaction My works focus on inferring and understanding the human hand’s interaction with objects from visual inputs, which include several tasks like pose estimation, grasping pose generation, and interacting pose transfer. Unlike the single-body pose estimation task, understanding the Hand-object (multi-bodies) interactions in 3D spaces is more challenging, due to its high degree of articulations, the projection ambiguity, self or mutual occlusions, and the complicated physical constraints. Designing algorithms to tackle these challenges is my goal. We find that the mutual contact can provide rich ... Yuliang Xiu

Perceiving Systems Talk Chunlu Li 19-04-2022 Model-based Face Reconstruction and Occlusion Segmentation from Weak Supervision 3D face reconstruction under occlusions is highly challenging because of the large variability of the appearance and location of occluders. Currently, the most successful methods fit a 3D face model through inverse rendering and assume a given segmentation of the occluder to avoid fitting the occluder. However, the segmentation annotations are costly since training an occlusion segmentation model requires large amounts of annotated data. To overcome this, we introduce a model-based approach for 3D face reconstruction that is highly robust to occlusions but does not require any occlusion ann... Victoria Fernandez Abrevaya

Perceiving Systems Talk Angela Yao 12-04-2022 Mixing Synthetic and Real-World Captures for RGB Hand Pose Estimation How can we learn models for hand pose estimation without any (real-world) labels? This talk presents our recent efforts in tackling the challenging scenario of learning from labelled synthetic data and unlabelled real-world data. I will focus on two strategies that we find to be effective: (1) cross-modal consistency and alignment for representation learning and (2) pseudo-label corrections and refinement. The second part of the talk will introduce Assembly101, our newly recorded dataset that tackles 3D hand pose and action understanding over time. Assembly101 is a new procedural activit... Dimitris Tzionas

Perceiving Systems Talk Henry Clever 07-04-2022 Modeling Humans at Rest with Applications to Robotic Assistance Humans spend a large part of their lives resting. Machine perception of this class of body poses would be beneficial to numerous applications, but it is complicated by line-of-sight occlusion from bedding. Pressure sensing mats are a promising alternative, but data is challenging to collect at scale. To overcome this, we use modern physics engines to simulate bodies resting on a soft bed with a pressure sensing mat. This method can efficiently generate data at scale for training deep neural networks. We present a deep model trained on this data that infers 3D human pose and body shape from ... Dimitris Tzionas Chun-Hao Paul Huang

Perceiving Systems Talk Sida Peng 07-04-2022 Reconstructing Static Scenes and Dynamic Humans with Implicit Neural Representations 3D reconstruction is a long-standing problem in computer vision and has a variety of applications such as virtual reality, 3D content generation, and telepresence. In this talk, I will present our progress on 3D reconstruction of static scenes and dynamic humans with implicit neural representations. The first part of the talk introduces an effective regularization when optimizing implicit neural representations on indoor scenes based on the Manhattan-world Assumption. In the second part, I will show some animatable implicit neural representations for modeling dynamic humans from videos. Hongwei Yi

Perceiving Systems Talk Pinelopi Papalampidi 08-02-2022 Structure-aware Narrative Understanding and Summarization In this work, we analyze and summarize full-length movies from multimodal input (i.e., video, text, audio). We first hypothesize that identifying the narrative structure of movies is a precondition for summarizing them. According to screenwriting theory, turning points (e.g., change of plans, major setback, climax) are crucial narrative moments within a movie that define the narrative structure and determine its progression and thematic units. Therefore, we introduce the task of Turning Point (TP) identification and leverage it for movie summarization and trailer generation. Next, we propos... Nikos Athanasiou Chun-Hao Paul Huang

Perceiving Systems Talk Ye Yuan 18-01-2022 Unified Simulation, Perception, and Generation of Human Behavior Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this talk, I will present a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. I will show how the three aspects are deeply connected and how utilizing and improving one aspect can greatly benefit the other aspects. Since humans live in a physical world, we treat simulation as the foundation of our approach and start by developing a fundamental framework for representing human ... Hongwei Yi

Perceiving Systems Talk Tiantian Wang 11-01-2022 Animatable humans from monocular RGB(D) videos We aim to reconstruct animatable humans from monocular RGB(D) videos. Learning user-controlled representation under novel poses remains a challenging problem. To tackle this problem, I will introduce two related methods. First, to reconstruct animatable photo-realistic humans, we integrate observations across frames and encode the appearance at each individual frame by utilizing the human pose and point clouds as the input. In addition, we utilize a temporal transformer to integrate the features of points in the unseen frames and the tracked points in a handful of automatically-selected key... Arjun Chandrasekaran

Perceiving Systems Talk Arianna Rampini 14-12-2021 Discrete inverse spectral geometry for shape analysis Spectral quantities as the eigenvalues of the Laplacian operator are widely used in geometry processing since they provide a very informative summary of the intrinsic geometry of deformable shapes. Typically, the intrinsic properties of shapes are computed from their representation in 3D space and are used to encode compact geometric features, thus adopting a data-reduction principle. On the contrary, this talk focuses on the inverse problem: namely, recovering an extrinsic embedding from a purely intrinsic encoding, like in the classical “hearing the shape of the drum” problem. I will sta... Silvia Zuffi

Perceiving Systems Talk Yajie Zhao 10-12-2021 Next Generation Lifelike Avatar Creation High-fidelity avatar creation for films and games is tied with complex capture equipment, massive data, a long production cycle, and intensive manual labor by a production team. And it may still be in the notorious Uncanny Valley. In this talk, we will explore how to produce a lifelike avatar in a low-cost way. We will show how to leverage deep learning networks to accelerate and simplify the industrial avatar production procedure from data capturing to animation. And bring photorealism to the next level! Timo Bolkart

Perceiving Systems Talk Yandong Wen 05-10-2021 Toward Reconstructing Face from Voice We address a new challenge posed by voice profiling - reconstructing someone’s face from their voice. Specifically, given an audio clip spoken by an unseen person, we aim to reconstruct a face that has as many associations as possible with the speaker in terms of identity. In this talk, I will introduce how we explore and approach the ultimate goal step by step. First, we investigate the audio-visual association by matching voices to faces based on identity, and vice versa. Second, we set up a baseline for reconstructing 2D face images from a voice recording and show reasonable reconstructi... Timo Bolkart

Perceiving Systems Talk Yuxiang Zhang and Yang Zheng 28-09-2021 DeepMultiCap & Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras We propose DeepMultiCap, a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models. To tackle the serious occlusion challenge for close interacting scenes, we combine a recently proposed pixel-aligned implicit function with a parametric model for robust reconstruction of the invisible surface areas. An effective attention-aware module is designed to obtain the fine-grained geometry details from multi-view images, where high-fidelity results can be generated. I... Chun-Hao Paul Huang

Perceiving Systems Talk Meng-Yu Jennifer Kuo 27-09-2021 Refraction and Absorption for Underwater Shape Recovery In this talk the speaker will present her work on the recovery of rigid and deformable 3D shape from underwater images. Silvia Zuffi

Perceiving Systems Talk Tianye Li 23-09-2021 Topologically Consistent Multi-View Face Inference Using Volumetric Sampling High-fidelity face digitization solutions often combine multi-view stereo (MVS) techniques for 3D reconstruction and a non-rigid registration step to establish dense correspondence across identities and expressions. A common problem is the need for manual clean-up after the MVS step, as 3D scans are typically affected by noise and outliers and contain hairy surface regions that need to be cleaned up by artists. Furthermore, mesh registration tends to fail for extreme facial expressions. Most learning-based methods use an underlying 3D morphable model (3DMM) to ensure robustness, but this li... Timo Bolkart

Perceiving Systems Talk Siwei Zhang 22-09-2021 Learning motion priors for 4D human body capture in 3D scenes It is challenging to recover realistic human-scene interactions and high-quality human motions while dealing with occlusions and partial views with a monocular RGB(D) camera. We address this problem by learning motion smoothness and infilling priors from the large scale mocap dataset AMASS, to reduce the jitters, and handle contacts and occlusions, respectively. Furthermore, we combine them into a multi-stage optimization pipeline for the high quality 4D human capture in complex 3D scenes. Chun-Hao Paul Huang

Perceiving Systems Talk Bharat Lal Bhatnagar 09-09-2021 Hybrid surface representations: Leveraging implicit functions and parametric models for 3D human modelling Representing 3D humans with parametric body models (eg: SMPL) allows us to control the 3D appearance of a human with explicit parameters for pose, shape and even clothing (to an extent). Implicit function based representations on the other hand, typically lack such interpretable control but can produce more detailed models as compared to parametric approaches. They also are not constrained by topology and resolution. In this talk I would like to discuss how we can combine these two directions and leverage the best of both worlds to model detailed and controllable 3D humans. I will primari... Ahmed Osman Arjun Chandrasekaran

Perceiving Systems Talk Jiefeng Li 06-09-2021 From skeleton to body: Keypoint Estimation is Helpful for Human Body Reconstruction My works mainly lie in inferring human structures from RGB inputs, which starts from 2D keypoint estimation, towards more complex tasks like 3D skeleton inference and SMPL-based human pose & shape estimation. Along this road, we find that high-level tasks, like human body estimation, can benefit a lot from low-level inferred structures, like 3D skeletons, and vice versa. Furthermore, in our latest work, "Human Pose Regression with Residual Log-likelihood Estimation", we unified all the above HPS tasks in a direct regression paradigm, replacing generally accepted heatmap without loss of accu... Yuliang Xiu

Perceiving Systems Talk Davis Rempe 27-07-2021 Modeling 3D Human Motion for Improved Pose Estimation Though substantial progress has been made in estimating 3D human poses from dynamic observations, recent methods still struggle to recover physically-plausible motions, and the presence of noise and occlusions remains challenging. In this talk, I'll introduce two methods that tackle these issues by leveraging models of 3D human motion - one physics-based and one learned. In the first approach, an initial 3D motion is refined using a physics-based trajectory optimization that leverages automatically-detected foot contacts from RGB video. In the second, a learned generative model is used as a... Muhammed Kocabas

Perceiving Systems Talk Hao Li 26-07-2021 AI SYNTHESIS: FROM AVATARS TO 3D SCENES In this talk I will motivate how digital humans will impact the future of communication, human-machine interaction, and content creation. I will present our latest 3D avatar digitization technology from Pinscreen from a single photo, and give a live demonstration. I will also showcase how we use hybrid CG and neural rendering solutions for real-time applications used in next generation virtual assistant and virtual production pipelines. I will then present a real-time teleportation system that only uses a single webcam as input, and our latest efforts at UC Berkeley in real-time AI synthesi... Yao Feng

Perceiving Systems Talk Maria Kolos 14-07-2021 TRANSPR: Transparency Ray-Accumulating Neural 3D Scene Point Renderer We propose and evaluate a neural point-based graphics method that can model semi-transparent scene parts. Similarly to its predecessor pipeline, ours uses point clouds to model proxy geometry, and augments each point with a neural descriptor. Additionally, a learnable transparency value is introduced in our approach for each point. Our neural rendering procedure consists of two steps. Firstly, the point cloud is rasterized using ray grouping into a multi-channel image. This is followed by the neural rendering step that "translates" the rasterized image into an RGB output using a learnable ... Qianli Ma

Perceiving Systems Talk Nataniel Ruiz 14-06-2021 Using Generative Models for Faces to Test Neural Networks Most machine learning models are validated on fixed datasets. This can give an incomplete picture of the capabilities and weaknesses of the model. Such weaknesses can be revealed at test time in the real world with dire consequences. In order to alleviate this issue, simulators can be controlled in a fine-grained manner using interpretable parameters to explore the semantic image manifold and discover such weaknesses before deploying a model. Also, in recent years there have been important advances in generative models for computer vision resulting in realistic face generation and manipulat... Timo Bolkart

Perceiving Systems Talk Peizhuo Li 10-06-2021 Learning Skeletal Articulations with Neural Blend Shapes Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure which produces high quality pose dependent deformations. Our framework learns to rig and skin characters with the same articulation structure... Hongwei Yi

Perceiving Systems Talk Marc Habermann 01-06-2021 Real-time Deep Dynamic Characters Animatable and photo-realistic virtual 3D characters are of enormous importance nowadays. However, generating realistic characters still requires manual intervention, expensive equipment, and the resulting characters are either difficult to control or not realistic. Therefore, the goal of the work, that is presented within the talk, is to learn digital characters which are both realistic and easy to control and can be learned directly from a multi-view video. To this end, I will introduce a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic a... Yinghao Huang Chun-Hao Paul Huang

Perceiving Systems Talk Chloe LeGendre 11-05-2021 Lighting Virtual Objects using Machine Learning Compositing rendered, virtual objects into photographs or videos is a fundamental technique in mixed reality, visual effects, and film production. For truly convincing and seamless composites, the subjects must be rendered with lighting that matches that of the target footage. For instance, a rendered object that is too bright, too dark, or lit from a direction inconsistent with other objects in the scene will look out of place. As such, in this talk I will introduce two recent machine learning based approaches for lighting estimation used for improving the realism of augmented reality (AR)... Victoria Fernandez Abrevaya

Perceiving Systems Talk Justus Thies 15-04-2021 Neural Capture & Synthesis The main theme of my work is to capture and to (re-)synthesize the real world using commodity hardware. It includes the modeling of the human body, tracking, as well as the reconstruction and interaction with the environment. The digitization is needed for various applications in AR/VR as well as in movie (post-)production. Teleconferencing and remote collaborative working in VR is of high interest since it is the next evolution step of how people communicate. A realistic reproduction of appearances and motions is key for such applications. Capturing natural motions and expressions as well ... Ahmed Osman

Perceiving Systems Talk Gyeongsik Moon 12-04-2021 Expressive Whole-Body 3D Multi-Person Pose and Shape Estimation from a Single Image Human is the most centric and interesting object in our life: many human-centric techniques and studies have been proposed from both industry and academia, such as virtual try-on, 3D personal avatar, and marker-less motion capture in the movie/game industry, including AR/VR. Recovery of accurate 3D geometry of humans (i.e., 3D human pose and shape) is a key component of the human-centric techniques and studies. In particular, the 3D pose and shape of multiple persons can deliver relative 3D location between persons. Also, the 3D pose and shape of the whole body, which includes hands and fac... Chun-Hao Paul Huang

Perceiving Systems Talk Angjoo Kanazawa 08-04-2021 Pushing the Boundaries of Novel View Synthesis 2020 was a turbulent year, but for 3D learning it was a fruitful one with lots of exciting new tools and ideas. In particular, there have been many exciting developments in the area of coordinate based neural networks and novel view synthesis. In this talk I will discuss our recent work on single image view synthesis with pixelNeRF, which aims to predict a Neural Radiance Field (NeRF) from a single image. I will discuss how NeRF representation allows models like pixel-aligned implicit functions (PiFu) to be trained without explicit 3D supervision and the importance of other key design fact... Qianli Ma

Perceiving Systems Talk Meng Zhang 01-04-2021 Hair & garment synthesis using deep learning method For both AR and VR applications, there is a strong motivation to generate virtual avatars with realistic hairs and garments that are the two most significant elements to personify any character. However, due to the complex structures and ever-changing fashion styles, modeling hairs and garments still remain tedious and expensive as they require considerable professional effort. My research interest focuses on deep learning methods in 3D modeling, rendering, and animation, especially to synthesis high-quality hairs and garments with plausible details. In this talk, I will present the progres... Jinlong Yang

Perceiving Systems Talk Garvita Tiwari 25-02-2021 Learning Size-Sensitive Clothing Model From Real Data 3D Human modeling has numerous applications in AR/VR, entertainment, the fashion industry and has been a challenging task, due to variation in human motion, body shape, style of clothing. One of the main challenges in human modeling is clothing, because of the complex behavior of clothing in the real world, lack of large scale dataset, etc. In this talk, I will talk about the motivation of learning from real-world data and present my previous work on size sensitive clothing model and 3d clothing parsing.

Perceiving Systems Talk Leonidas Guibas 22-02-2021 Joint Learning Over Visual and Geometric Data Many challenges remain in applying machine learning to domains where obtaining massive annotated data is difficult. We discuss approaches that aim to reduce supervision load for learning algorithms in the visual and geometric domains by leveraging correlations among data as well as among learning tasks -- what we call joint learning. The basic notion is that inference problems do not occur in isolation but rather in a "social context" that can be exploited to provide self-supervision by enforcing consistency, thus improving performance and increasing sample efficiency. An example is voting ... Qianli Ma

Perceiving Systems Talk Ruilong Li 10-02-2021 AI Choreographer: Learn to dance with AIST++ In this work, we present a transformer-based learning framework for 3D dance generation conditioned on music. We carefully design our network architecture and empirically study the keys for obtaining qualitatively pleasing results. In addition, we propose a new dataset of paired 3D motion and music called AIST++, which contains 1.1M frames of 3D dance motion in 1408 sequences, covering 10 genres of dance choreographies and accompanied with multi-view camera parameters. To our knowledge it is the largest dataset of this kind. Yuliang Xiu

Perceiving Systems Talk Shruti Agarwal 28-01-2021 Creating, Weaponizing, and Detecting Deep Fakes The past few years have seen a startling and troubling rise in the fake-news phenomena in which everyone from individuals to nation-sponsored entities can produce and distribute misinformation. The implications of fake news range from a misinformed public to an existential threat to democracy, and horrific violence. At the same time, recent and rapid advances in machine learning are making it easier than ever to create sophisticated and compelling fake images, videos, and audio recordings, making the fake-news phenomena even more powerful and dangerous. These AI-synthesized media (so-called... Jinlong Yang

Perceiving Systems Talk Yixin Chen 27-01-2021 Towards a more holistic understanding of scene, object, and human Humans, even young infants, are adept at perceiving and understanding complex indoor scenes. Such an incredible vision system relies on not only the data-driven pattern recognition but also roots from the visual reasoning system, known as the core knowledge, that facilitates the 3D holistic scene understanding tasks. This talk discusses how to employ physical common sense and human-object interaction to bridge scene and human understanding and how the part-level 3D affordance perception may lead to a more fine-grained human-object interaction modeling. Future directions may be extended to d... Dimitris Tzionas

Perceiving Systems Talk Zorah Lähner 21-01-2021 Non-Rigid Shape Correspondence through Deformation Solving for 3D correspondences beyond isometries has made tremendous progress in recent years, much of it due to (deep) learning. However, not all applications provide the necessary training data. This talk will focus on how far we can take the results without learning. I will present a line of work that poses the non-rigid shape registration problem in terms of physical and non-physical deformation energies. Our work aims to combine extrinsic and intrinsic measures to overcome typical shortcomings of both. We use Functional Maps and Markov Chain Monte Carlo initialization to handle all kin... Jinlong Yang

Perceiving Systems Talk Raquel Urtasun 14-12-2020 A Future with Self-Driving Vehicles We are on the verge of a new era in which robotics and artificial intelligence will play an important role in our daily lives. Self-driving vehicles have the potential to redefine transportation as we understand it today. Our roads will become safer and less congested, while parking spots will be repurposed as leisure zones and parks. However, many technological challenges remain as we pursue this future. In this talk I will showcase the latest advancements made by Uber Advanced Technologies Group’s in the quest towards self-driving vehicles at scale. Qianli Ma

Perceiving Systems Talk Daniel Haun 05-10-2020 The phenotyping revolution One of the most striking characteristics of human behavior in contrast to all other animal is that we show extraordinary variability across populations. Human cultural diversity is a biological oddity. More specifically, we propose that what makes humans unique is the nature of the individual ontogenetic process, that results in this unparalleled cultural diversity. Hence, our central question is: How is human ontogeny adapted to cultural diversity and how does it contribute to it? This question is critical, because cultural diversity does not only entail our predominant mode of adaptation ... Timo Bolkart

Perceiving Systems Talk Noah Snavely 02-10-2020 Reconstructing the Plenoptic Function Imagine a futuristic version of Google Street View that could dial up any possible place in the world, at any possible time. Effectively, such a service would be a recording of the plenoptic function—the hypothetical function described by Adelson and Bergen that captures all light rays passing through space at all times. While the plenoptic function is completely impractical to capture in its totality, every photo ever taken represents a sample of this function. I will present recent methods we've developed to reconstruct the plenoptic function from sparse space-time samples of photos—inclu...

Perceiving Systems Talk Daniel Holden 10-08-2020 Functions, Machine Learning, and Game Development Game Development requires a vast array of tools, techniques, and expertise, ranging from game design, artistic content creation, to data management and low level engine programming. Yet all of these domains have one kind of task in common - the transformation of one kind of data into another. Meanwhile, advances in Machine Learning have resulted in a fundamental change in how we think about these kinds of data transformations - allowing for accurate and scalable function approximation, and the ability to train such approximations on virtually unlimited amounts of data. In this talk I will p... Abhinanda Ranjit Punnakkal

Perceiving Systems Talk Vittorio Ferrari 07-08-2020 Our Recent Research on 3D Deep Learning I will present three recent projects within the 3D Deep Learning research line from my team at Google Research: (1) a deep network for reconstructing the 3D shape of multiple objects appearing in a single RGB image (ECCV'20). (2) a new conditioning scheme for normalizing flow models. It enables several applications such as reconstructing an object's 3D point cloud from an image, or the converse problem of rendering an image given a 3D point cloud, both within the same modeling framework (CVPR'20); (3) a neural rendering framework that maps a voxelized object into a high quality image. It re... Yinghao Huang Arjun Chandrasekaran

Perceiving Systems Talk Antonio Torralba 28-07-2020 Learning from vision, touch and audition Babies learn with very little supervision, and, even when supervision is present, it comes in the form of an unknown spoken language that also needs to be learned. How can kids make sense of the world? In this work, I will show that an agent that has access to multimodal data (like vision, audition or touch) can use the correlation between images and sounds to discover objects in the world without supervision. I will show that ambient sounds can be used as a supervisory signal for learning to see and vice versa (the sound of crashing waves, the roar of fast-moving cars – sound conveys impor... Arjun Chandrasekaran

Perceiving Systems Talk Artsiom Sanakoyeu 22-07-2020 Learning Dense Correspondences for Animals with limited supervision and Improving Generalization for Deep Metric Learning Learning the embedding space, where semantically similar objects are located close together and dissimilar objects far apart, is a cornerstone of many computer vision applications. Existing approaches usually learn a single metric in the embedding space for all available data points,which may have a very complex non-uniform distribution with different notions of similarity between objects, e.g. appearance, shape, color or semantic meaning. We approach this problem by using the embedding space more efficiently by jointly splitting the embedding space and data into K smaller sub-problems. It ... Nikos Athanasiou

Perceiving Systems Talk Angela Dai 16-07-2020 Towards Commodity 3D Scanning for Content Creation In recent years, commodity 3D sensors have become widely available, spawning significant interest in both offline and real-time 3D reconstruction. While state-of-the-art reconstruction results from commodity RGB-D sensors are visually appealing, they are far from usable in practical computer graphics applications since they do not match the high quality of artist-modeled 3D graphics content. One of the biggest challenges in this context is that obtained 3D scans suffer from occlusions, thus resulting in incomplete 3D models. In this talk, I will present a data-driven approach towards genera... Yinghao Huang

Perceiving Systems Talk William T. Freeman 13-07-2020 Learning from videos played forwards, backwards, fast, and slow How can we tell that a video is playing backwards? People's motions look wrong when the video is played backwards--can we develop an algorithm to distinguish forward from backward video? Similarly, can we tell if a video is sped-up? We have developed algorithms to distinguish forwards from backwards video, and fast from slow. Training algorithms for these tasks provides a self-supervised task that facilitates human activity recognition. We'll show these results, and applications of these unsupervised video learning tasks, including a method to change the timing of people in videos. Yinghao Huang

Perceiving Systems Talk Matthias Nießner 10-07-2020 Learning Non-rigid Optimization Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus. One recent approach proposes self-supervision based on non-rigid reconstruction. Unfortunately, this method fails for important cases such as highly non-rigid deformations. We first address this problem of lack of data by introducing a novel semi-supervised strategy to obtain dense interframe correspondences from a sparse set of annotations. This way, we obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 2,537 d... Vassilis Choutas