New sensing technologies and advanced processing algorithms are transforming computer-integrated surgery. While researchers are actively investigating neural perception and learned representations for vision-based surgical assistance, the majority of these methods build geometric estimates based on 2D white-light images. While being the clinical standard for decades, this type of imaging will evolve through miniaturization of hyperspectral and depth sensors. Furthermore, visual inspection in minimally invasive surgery is usually done with a single camera, preventing the availability of multi-viewpoint imaging data. To promote the generation of real-time, accurate, and robust spatiotemporal measurements of the intraoperative anatomy, this project expands in three main directions: i) investigation and prototyping of innovative 3D sensing technologies for surgery, ii) development of multi-viewpoint computational methods for kinematic fusion and scene reconstruction, iii) acquisition of highly realistic datasets with multiple visual and geometric measurement devices.

We advocate the introduction of one or more lidar (3D laser-based direct time-of-flight) in the abdominal cavity as additional imaging sensors []. To characterize the quality with which this technology could perform anatomical surface measurements during surgery, we conducted ex-vivo experiments comparing lidar to traditional learning-based stereo matching from endoscopic images. Our findings highlight the potential and limitations of lidar for surgical 3D perception and point toward new intraoperative methods that could complement time-of-flight with spectral imaging [][https://arxiv.org/abs/2401.10709].

We believe future computer-integrated surgery will benefit from an abundance of intra corporeal measurement sources and efficient processing of their outputs, with the dual goals of supporting surgeons and laying the foundation for surgical autonomy. To jointly achieve these goals, special attention should be given to intuitive user interfaces and effective visualization strategies [], as well as to building computational systems able to generate accurate quantitative estimates. We are currently developing algorithmic solutions to deploy multi-viewpoint systems in minimally invasive surgery [https://arxiv.org/abs/2408.04367].

Conducting research in fields like robotics and computer science applied to surgery - which is a strictly regulated, high-volume, and high-risk practice - comes with the challenge of developing technologies that can (and should) directly impact human life. In the era of learning systems, clinical translation of biomedical research, especially in the algorithmic space, is strongly dependent on the quality and availability of data for development and lab testing. To contribute in this direction, we are actively building hardware and software infrastructure for the collection of high-fidelity multi-sensor imaging datasets that will boost surgical research and deliver a significant impact to society.

This research project involves collaborations with Cesar Cadena (ETH Zürich), Marco Hutter (ETH Zürich), Sergey Prokudin (ETH Zürich), and Siyu Tang (ETH Zürich).