Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Institute Homepage

Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Computational Modeling of Finger-Surface Contact

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Modeling Hand Deformations During Contact

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Immersive VR for Phantom Limb Pain

Visual and Haptic Perception of Real Surfaces

Haptipedia

Gait Propulsion Trainer

TouchTable: A Musical Interface with Haptic Feedback for DJs

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

How Should Robots Hug?

Hierarchical Structure for Learning from Demonstration

Fabrication of HuggieBot 2.0: A More Huggable Robot

Learning Haptic Adjectives from Tactile Data

Feeling With Your Eyes: Visual-Haptic Surface Interaction

S-BAN

General Tactile Sensor Model

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Sab training2 — We are investigating technical approaches for improving training in robotic surgery. From left to right: Vibration sensors are attached to the arms of a da Vinci surgical system, and a force sensor is placed below the task. Vibration and force signals are used for surgical skill evaluation. Trainees also feel the vibrations as vibrotactile feedback played by voice-coil actuators, while force feedback is rendered through tactile actuators. Visual augmented reality allows training with pre-recorded expert performances.

Minimally invasive surgery (MIS) allows surgeons to perform procedures through tiny incisions, thus reducing healing time compared to traditional open surgery. In robot-assisted MIS (RMIS), surgeons control the instruments from a console. Consequently, besides the advantages of using a robotic platform, RMIS has one main drawback: the lack of haptic feedback, which is critical in manipulation tasks.

We explored the effects of haptic feedback while training in RMIS. First, we investigated vibrotactile feedback using VerroTouch, a previously developed system that allows the surgeon to feel the instruments’ vibrations at the operator’s manipulators. The vibrotactile feedback did not increase or decrease the trainee's workload []; ongoing work is examining impacts on learning. We also explored force feedback and created bracelets that squeeze the operator’s wrists proportionally to the force applied by the surgical instruments. Participants applied significantly less force on the task materials when receiving the feedback [].

We identified two additional challenges when training in RMIS: the quantitative evaluation of surgical skills and the limits of existing simulators.

Currently, surgical skill assessment is mainly conducted through manual video evaluation, which is time-consuming and subject to bias. For MIS, we developed a motion-tracking system and machine-learning algorithm to evaluate trainee performance during suturing tasks. The automatic ratings closely matched human expert ratings []. For RMIS, we showed that surgical skill can be estimated through completion time, force applied to task materials, and vibrations generated at the surgical instruments [].

Finally, training with physical simulators requires experts to supervise the process. In RMIS, using a robotic console offers the possibility of evaluating surgical skills in real time and training in virtual and augmented reality. However, current virtual simulators result in lower skill transfer to actual surgeries. As such, we hypothesize that augmented reality simulators could combine the advantages of physical and virtual approaches. After an early prototype [], our final developed platform allows expert surgeons to record surgical procedures with multiple modalities and novice surgeons to replay and train with the multimodal recordings, including visual, auditory, and vibrotactile feedback [].

Members

Haptic Intelligence

Katherine J. Kuchenbecker

Director

Haptic Intelligence

Haliza MatHusin

Research Scientist

Haptic Intelligence, Perceiving Systems

Maria-Paola Forte

Postdoctoral Researcher

Haptic Intelligence

Fabian Krauthausen

Master Student

Publications

Haptic Intelligence Miscellaneous Bimanual Wrist-Squeezing Haptic Feedback Changes Speed-Force Tradeoff in Robotic Surgery Training Cao, E., Machaca, S., Bernard, T., Chi, A., Wolfinger, B., Patterson, Z., Adrales, G. L., Kuchenbecker, K. J. Short paper presented at the ACS Surgeons and Engineers: A Dialogue on Surgical Simulation meeting, Virtual, March 2021 (Published) URL BibTeX

Haptic Intelligence Miscellaneous Sleep, Stress, and Experience Supersede Vibrotactile Haptic Feedback as Contributors to Workload During Robotic Surgical Skill Acquisition Gomez, E. D., Mat Husin, H., Dumon, K. R., Williams, N. N., Kuchenbecker, K. J. Extended abstract presented as an ePoster at the Annual Meeting of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES), Cleveland, USA, August 2020 (Published)

Abstract ›

Introduction: How does the absence of haptic feedback in robotic surgery affect surgical skill acquisition? This study is a prospective single-blinded randomized controlled trial examining the effect of haptic feedback of instrument vibrations during simulation-based training on resident workload and performance during both simulated and live operating room Robotic-Assisted Sleeve Gastrectomy (RASG), which provides surgical trainees with significant robotic console experience. Methods: Twelve surgical residents (seven PGY-3, five PGY-7) were randomized to receive either haptic feedback or no haptic feedback during a proctored simulation session that took place before the first operative cases of a bariatric service rotation. Workload measures including pre- and post-procedure short-form State-Trait Anxiety Inventory (STAI) and NASA Task Load Index (TLX) were recorded in both the simulated and OR settings. Multivariable linear regression with backward selection was performed to examine potential associations between workload measures and factors including haptic feedback, PGY-level, case volume, robotic operative time, and hours of sleep. Results: Subjects performed a total of 60 simulated bariatric surgical procedures and 79 live patient RASGs. During the simulation session, PGY-7 status was associated with a 12.8% decrease in TLX score (p<0.001); one additional hour of sleep yielded a 4.43% decrease in TLX score (p=0.004); one additional point in pre-procedure STAI score yielded a 1.87% increase in TLX score (p=0.003); and a one percent increase in operative time yielded a 0.12% increase in TLX score (p<0.001). During live OR cases, one additional RASG case experience during the rotation was associated with a 1.1% decrease in TLX score (p=0.01); one additional point in pre-procedure STAI score was associated with a 2.64% increase in TLX score (p<0.001); and a one percent increase in robotic operative time was associated with a 0.17% increase in TLX score (p<0.001). Haptic feedback did not significantly affect workload in either setting. No factors had a significant association with pre- to post-procedural change in STAI score. Conclusion: Providing vibrotactile haptic feedback during training neither increased nor decreased resident workload during simulated or live robotic surgical cases, possibly because the utility of the feedback counterbalances the additional processing required. In contrast, PGY-level, baseline stress, operative time, sleep, and case experience all contribute to workload in robotic surgery; these factors can be potential targets of educational intervention. Finally, TLX may be a more robust workload measurement tool than STAI in the context of robotic surgery.

BibTeX

Haptic Intelligence Article Automatically Rating Trainee Skill at a Pediatric Laparoscopic Suturing Task Oquendo, Y. A., Riddle, E. W., Hiller, D., Blinman, T. A., Kuchenbecker, K. J. Surgical Endoscopy, 32(4):1840-1857, April 2018 (Published) DOI BibTeX

Haptic Intelligence Conference Paper A Wrist-Squeezing Force-Feedback System for Robotic Surgery Training Brown, J. D., Fernandez, J. N., Cohen, S. P., Kuchenbecker, K. J. In Proceedings of the IEEE World Haptics Conference (WHC), 107-112, Munich, Germany, June 2017 (Published)

Abstract ›

Over time, surgical trainees learn to compensate for the lack of haptic feedback in commercial robotic minimally invasive surgical systems. Incorporating touch cues into robotic surgery training could potentially shorten this learning process if the benefits of haptic feedback were sustained after it is removed. In this paper, we develop a wrist-squeezing haptic feedback system and evaluate whether it holds the potential to train novice da Vinci users to reduce the force they exert on a bimanual inanimate training task. Subjects were randomly divided into two groups according to a multiple baseline experimental design. Each of the ten participants moved a ring along a curved wire nine times while the haptic feedback was conditionally withheld, provided, and withheld again. The realtime tactile feedback of applied force magnitude significantly reduced the integral of the force produced by the da Vinci tools on the task materials, and this result remained even when the haptic feedback was removed. Overall, our findings suggest that wrist-squeezing force feedback can play an essential role in helping novice trainees learn to minimize the force they exert with a surgical robot.

DOI BibTeX