The digitization of objects and scenes is of paramount importance for applications like collaborative working in VR or AR. It requires the photo-realistic synthesis of novel views, the composition of different objects in one view, as well as the editability of the scene (modifying material properties, illumination and geometry).

The research field has two subfields; the reconstruction of static objects and scenes, and the reconstruction and tracking of non-rigid objects. Especially, room scenes are mostly rigid, especially, furniture and structural elements. For these, there exists a variety of methods that ranges from classical multiview stereo, volumetric fusion-based approaches to neural rendering and reconstruction techniques []. But these methods still lack the controllability and details you would need for a photorealistic reproduction in AR/VR. Especially for room-scale scenes, completion of geometry and appearance of unobserved regions is challenging. Additionally, controlling and editing scenes (e.g., moving objects), requires semantic information and completion of each individual object.

Non-rigid objects such as cloth, pillows etc. are very challenging to track and reconstruct, especially, when there are topology changes (e.g., open and closed jacket). Similar to neural rendering, there are methods that learn components of classical non-rigid tracking pipelines, also known as neural non-rigid tracking []. The representation of the deformation is a key component of these approaches. Neural deformation graphs [] can be one representation. It is agnostic to the object class and can adapt to arbitrary objects. A core challenge is to improve the robustness w.r.t. topology changes, to improve run-time and to integrate color (for tracking and in the final reconstruction).