Force feedback is a crucial aspect of surgical procedures. Surgeons rely on their sense of touch to locate tumors, identify blood vessels, and safely manipulate tissue. However, in robot-assisted minimally invasive surgery, surgeons cannot directly feel the forces they apply to tissue through the robotic instruments. This lack of haptic feedback can lead to excessive force application, potentially causing tissue damage, or insufficient force, which can result in inefficient procedures. While direct force sensing could help, integrating sensors into surgical instruments requires addressing significant challenges of biocompatibility, sterilization, miniaturization, and robustness.

To overcome these limitations, we propose an indirect force-sensing method that combines monocular endoscopic video with inertial measurements units (IMUs) mounted on the instrument shafts. Our research objective is to demonstrate that visual tissue deformation cues combined with inertial information of the instruments can effectively estimate interaction forces during robotic surgery []. We developed this approach using a da Vinci Si robot, but designed it to be compatible with various robotic surgery platforms and traditional laparoscopic surgery.

We established a comprehensive experimental setup using four phantom tissue samples of varying stiffness and a force sensor for ground-truth data. The sensing system combines visual information acquired through the endoscope's left channel with inertial measurements from the IMUs mounted externally on the instrument shafts. We then collected 230 one-minute recordings from four operators performing palpation tasks perpendicularly to the tissue.

Based on extensive testing of different architectures, we developed a deep learning pipeline that uses DenseNet for visual processing combined with temporal modeling through (Bi)LSTM networks. Our system demonstrates strong performance in estimating normal forces in the palpations tasks, particularly on tissues of higher stiffness, while also showing the capability to estimate smaller shear forces. This indirect sensing approach has thus the potential to offer a practical solution for force estimation in robotic surgery while avoiding the complications of direct sensing methods.