Publications

Despite the importance of sensitive skin for living creatures, most robots can feel contact on only a tiny fraction of their exterior, if at all. Furthermore, typical robot reactions to touch are limited to event-based acknowledgments, lacking perceptual richness, lifelike positive/negative responses, and temporal dynamics. We address these gaps by introducing a practical full-body tactile-perception system for social robots, turning a NAO robot into the Haptic Empathetic Robot Animal (HERA). The sixteen main regions of the robot's body are instrumented with soft resistive tactile sensors covered by a tailored koala suit. Windows of each time-varying sensor output are continually classified into five gestures at two intensities via a two-stage machine-learning model. On challenging testing data containing simultaneous contacts, touch detection achieves an F1 score of 0.773, and gesture recognition achieves 52.2% accuracy (5.2 times chance); considering the temporal, spatial, and semantic adjacency of the applied touches increases these metrics to 0.896 and 86.6%, respectively. In turn, each detected contact drives a real-time emotion model that represents the robot's affective state as a second-order dynamic system analogous to a mass-spring-damper. This model's parameters control the robot's disposition, stoicism, and calmness. We explain the connections between HERA's hardware and software subsystems and demonstrate their combined ability to create an affective robot that feels both touch and emotion.

Accessibility legislation mandates equal access to information for Deaf communities. While videos of human interpreters provide optimal accessibility, they are costly and impractical for frequently updated content. AI-driven signing avatars offer a promising alternative, but their development is limited by the lack of high-quality 3D motion-capture data at scale. Vision-based motion-capture methods are scalable but struggle with the rapid hand movements, self-occlusion, and self-touch that characterize sign language. To address these limitations, this dissertation develops two complementary solutions. SGNify improves hand pose estimation by incorporating universal linguistic rules that apply to all sign languages as computational priors. Proficient signers recognize the reconstructed signs as accurately as those in the original videos, but depth ambiguities along the camera axis can still produce incorrect reconstructions for signs involving self-touch. To overcome this remaining limitation, BioTUCH integrates electrical bioimpedance sensing between the wrists of the person being captured. Systematic measurements show that skin-to-skin contact produces distinctive bioimpedance reductions at high frequencies (240 kHz to 4.1 MHz), enabling reliable contact detection. BioTUCH uses the timing of these self-touch events to refine arm poses, producing physically plausible arm configurations and significantly reducing reconstruction error. Together, these contributions support the scalable collection of high-quality 3D sign language motion data, facilitating progress toward AI-driven signing avatars.

Approximating model predictive control (MPC) using imitation learning (IL) allows for fast control without solving expensive optimization problems online. However, methods that use neural networks in a simple L2-regression setup fail to approximate multi-modal (set-valued) solution distributions caused by local optima found by the numerical solver or non-convex constraints, such as obstacles, significantly limiting the applicability of approximate MPC in practice. We solve this issue by using diffusion models to accurately represent the complete solution distribution (i.e., all modes) at high control rates (more than 1000 Hz). This work shows that diffusion-based AMPC significantly outperforms L2-regression-based approximate MPC for multi-modal action distributions. In contrast to most earlier work on IL, we also focus on running the diffusion-based controller at a higher rate and in joint space instead of end-effector space. Additionally, we propose the use of gradient guidance during the denoising process to consistently pick the same mode in closed loop to prevent switching between solutions. We propose using the cost and constraint satisfaction of the original MPC problem during parallel sampling of solutions from the diffusion model to pick a better mode online. We evaluate our method on the fast and accurate control of a 7-DoF robot manipulator both in simulation and on hardware deployed at 250 Hz, achieving a speedup of more than 70 times compared to solving the MPC problem online and also outperforming the numerical optimization (used for training) in success ratio.

Distinguishing the feel of smooth silk from coarse cotton is a trivial everyday task for humans. When exploring such fabrics, fingertip skin senses both spatio-temporal force patterns and texture-induced vibrations that are integrated to form a haptic representation of the explored material. It is challenging to reproduce this rich, dynamic perceptual capability in robots because tactile sensors typically cannot achieve both high spatial resolution and high temporal sampling rate. In this work, we present a system that can sense both types of haptic information, and we investigate how each type influences robotic tactile perception of fabrics. Our robotic hand's middle finger and thumb each feature a soft tactile sensor: one is the open- source Minsight sensor that uses an internal camera to measure fingertip deformation and force at 50 Hz, and the other is our new sensor Minsound that captures vibrations through an internal MEMS microphone with a bandwidth from 50 Hz to 15 kHz. Inspired by the movements humans make to evaluate fabrics, our robot actively encloses and rubs folded fabric samples between its two sensitive fingers. Our results test the influence of each sensing modality on overall classification performance, showing high utility for the audio-based sensor. Our transformer-based method achieves a maximum fabric classification accuracy of 97% on a dataset of 20 common fabrics. Incorporating an external microphone away from Minsound increases our method's robustness in loud ambient noise conditions. To show that this audio-visual tactile sensing approach generalizes beyond the training data, we learn general representations of fabric stretchiness, thickness, and roughness.

Haptic feedback in today's VR systems is often limited to vibration delivered through handheld controllers, leaving a gap for compact devices that can convey spatial cues without occupying the hands. This thesis presents the design and evaluation of SuperCUTE, a wrist-worn pressure feedback device that uses four soft electrohydraulic actuators to elicit phantom tactile sensations around the wrist. The device was evaluated with n = 20 participants in a user study comprising two tasks. In Task 1 (circular GUI), single-actuator cues produced tightly clustered responses (median resultant length R = 0.92); about 70% of trials fell within ± 22.5° of the stimulated cardinal. Adjacent-actuator pairs yielded in-between percepts (about 70% of reports), and intensity imbalance shifted perceived location toward the stronger actuator; reported intensity was higher for strong than weak drives (mean 0.76 vs. 0.32). Across cues, Rayleigh tests indicated strong clustering of response angles (median R ≈ 0.82). In Task 2 (VR), hand trajectories during 5 s cues aligned with cue geometry; end-directions showed strong clustering (median R ≈ 0.78), and latency estimated from a 1 cm displacement threshold had a median of 1.25 s (IQR 0.61 s). Questionnaire responses indicated clear, comfortable, and usable cues. Overall, pressure pulses are a feasible approach for directional wrist cues in VR. We provide device documentation, datasets, and analysis code to support pressure-based wearable haptics.

Knee osteoarthritis (KOA) affects millions worldwide, with excessive joint loading linked to disease progression. Modifying the foot progression angle (FPA) while walking is one strategy to reduce knee adduction moments, a measure associated with medial knee joint loading. This study investigated whether two types of vibrotactile biofeedback during a 20-minute treadmill gait-retraining session helped healthy adults better learn and retain a 10°toe-in gait. Participants who received feedback showed greater improvements in FPA accuracy than those without feedback and also reported significantly higher mental effort. The type of feedback that scaled the duration of the vibration with the magnitude of the error led to better short-term retention than no feedback, and it was also preferred by almost all subjects over constant-duration cues. These findings suggest that despite the added cognitive demand, users value biofeedback, emphasizing the need to design gait-retraining tools that consider both learning effectiveness and user experience.

Although essential for an immersive experience in extended reality (XR), providing salient and versatile touch feedback remains a technical challenge. Existing solutions restrict hand movements with bulky rigid structures, require a tethered energy source to power actuators worn on the hand, or output vibrations that lack expressiveness. This study introduces a design strategy for compact, lightweight, untethered haptic feedback centering on a 30-µm-thick inflatable chamber that naturally conforms to the fingertip; to minimize fluidic losses and enable high bandwidth, a soft electrohydraulic pump mounted on the hand actuates the chamber via a mechanically transparent fluidic channel. A 15.2-mm-diameter prototypical actuation chamber achieves 8 N peak force, 3 N steady-state force, stroke up to 5 mm, and bandwidth from 0 to 500 Hz. In contrast to these salient fingertip cues, the entire hydraulic system has a weight less than 8 g and a thickness less than 2 mm. Additionally, this study presents a validation approach that uses a commercial fingertip sensor to confirm that the haptic feedback created by the device imitates the touch signals generated during typical hand interactions. Together, this design strategy and validation method can enable a broad spectrum of haptic activities in diverse XR applications, including medical training, online shopping, and social interactions.

Tube thoracostomy (chest tube insertion) is a surgical procedure that treats pneumothorax, a potentially life-threatening condition where air accumulates between the chest wall and the lungs. The literature reports high complication rates for this procedure, including accidental fatality due to poor manual depth control during tool insertion. We hypothesize that an instrumented needle-holder could help operators recognize pleural puncture and improve depth control, and we present a puncture-detection experiment that contributes toward this goal. An operator manually inserted a bevel-tip needle into ex vivo porcine ribs and through the parietal pleura via a sensorized percutaneous device that records position, force, and videos. We use this rich dataset of 63 insertions to thoroughly test four previously published data-driven puncture-detection (DDPD) algorithms against two new real-time algorithms: a custom recursive digital filter with coefficients optimized for our application, and a difference equation that compares standard deviations between adjacent sliding windows. Our algorithms achieve a precision (true positives over total identified punctures) of 23% and 22%, respectively, while the precision of existing DDPD algorithms ranges from 0% to 21%. Despite these performance improvements, our results show the limitations of DDPD algorithms and motivate new methods for detecting pleural membrane punctures in thoracostomy.

Chest trauma or disease progression can lead to tension pneumothorax, a condition where mounting pressurization of the pleural cavity (the space between the chest wall and the lungs) leads rapidly to cardiac arrest. In pre-hospital settings, tension pneumothorax is treated by venting the pleural cavity via a needle introduced through the chest wall. Very high failure rates (up to 94.1%) have been reported for pre-hospital needle decompression, however, and the procedure can result in the accidental puncture of critical thoracic tissues because it is performed blind. Instrumented needles could help operators more reliably identify when the tool has entered the target space. This paper investigates technical approaches to provide such support; we created an experimental system that acquires needle force and position signals, as well as the diffuse backscattered reflectance from white light carried to and collected from the needle's tip via two in-bore optical fibers. Data collection occurred while two experimenters inserted a bevel-tipped percutaneous needle into an ex vivo porcine rib section simulating human chest anatomy. Four data-driven puncture-detection (DDPD) algorithms from the literature, which are appropriate for use with the variable tool velocities produced by manual insertions, were applied to the resulting data set offline. Grid search was performed across key signal-processing parameters, high-pass filters (HPFs) were applied to examine their impact on puncture detection, and a first exploration of multimodal (ensemble) methods was performed. Combining high-pass filters with DDPD methods resulted in a 2.7-fold improvement (from 8.2% to 21.9%) in the maximum overall precision (MOP) produced by force signals. Applying this HPF + DDPD scheme to reflectance data streams yielded a peak MOP of 36.4%, and combining reflectance with force generated the best MOP overall (42.1%); these results represent 4.4-fold and 5.1-fold improvements, respectively, over the best MOP produced by the traditional application of DDPD algorithms to force signals alone. These results strongly support the utility of high-pass filters combined with both reflectance-only and multimodal reflectance-plus-force data-driven puncture-detection schemes for needle decompression applications.

Accurate 3D pose estimation of grasped objects is an important prerequisite for robots to perform assembly or in-hand manipulation tasks, but object occlusion by the robot's own hand greatly increases the difficulty of this perceptual task. Here, we propose that combining visual information with binary, low-resolution tactile contact measurements from across the interior surface of an articulated robotic hand can mitigate this issue. The visuo-tactile object-pose-estimation problem is formulated probabilistically in a factor graph. The pose of the object is optimized to align with the two kinds of measurements using a robust cost function to reduce the influence of outlier readings. The advantages of the proposed approach are first demonstrated in simulation: a custom 15-DOF robot hand with one binary tactile sensor per link grasps 17 YCB objects while observed by an RGB-D camera. This low-resolution in-hand tactile sensing significantly improves object-pose estimates under high occlusion and also high visual noise. We also show these benefits through grasping tests with a preliminary real version of our tactile hand, obtaining reasonable visuo-tactile estimates of object pose at approximately 12.9 Hz on average.

This study demonstrates that encasing a hyperelastic balloon in an inextensible sleeve greatly increases its burst pressure while not influencing its minimum pressure. This simple mechanical behavior can be used to produce an asymmetric inflation-deflation sequence for coupled balloons with different thicknesses so they could serve as a soft robot's rear and front anchors when driven from a single fluid supply.

Mammal whiskers have inspired many sensors that can help robots find obstacles, identify textures, or sense flow. Though they vary in geometry, past bio-inspired whisker sensors were primarily constructed from homogenous materials. Interestingly, animal whiskers tend to shift from a stiff root to a much softer point; this material stiffness gradient is hypothesized to provide functional benefits such as reduction of wear and amplification of contact sensations. We take inspiration from nature to fabricate bio-inspired gradient (BIG) whiskers via 3D printing, and we assess their performance compared to stiff, medium, and soft homogenous artificial whiskers with the same geometry. Tests with controlled quasi-static and dynamic perturbations allow us to measure the whisker point deflection and the reaction torque at the stationary whisker root, respectively. The dynamic results reveal that BIG whiskers uniquely encode contact location along their length through torque magnitude and frequency, features that are not seen in the homogenous whiskers. These exciting preliminary findings motivate further exploration of robotic whiskers and other sensing structures with bio-inspired stiffness gradients.

Recent efforts to connect builders to digital designs during construction have primarily focused on visual augmented reality, which requires accurate registration and specific lighting, and which could prevent a user from noticing safety hazards. Haptic interfaces, on the other hand, can convey physical design parameters through tangible local cues that don't distract from the surroundings. We propose two edge-changing haptic devices that use small inertial measurement units (IMUs) and linear actuators to guide users to perform construction tasks in real time: Drangle gives feedback for angling a drill relative to gravity, and Brangle assists with orienting bricks in the plane. We conducted a study with 18 participants to evaluate user performance and gather qualitative feedback. All users understood the edge-changing cues from both devices with minimal training. Drilling holes with Drangle was somewhat less accurate but much faster and easier than with a mechanical guide; 89% of participants preferred Drangle over the mechanical guide. Users generally understood Brangle's feedback but found its hand-size-specific grip, palmar contact, and attractive tactile cues less intuitive than Drangle's generalized form factor, fingertip contact, and repulsive cues. After summarizing design considerations, we propose application scenarios and speculate how such devices could improve construction workflows.

Humanoid social robots need to be able to move expressively. Traditional manipulation-focused teleoperation systems primarily control the end-effector's position and orientation, neglecting the extra degrees of freedom in human and robotic arms, which can lead to unnatural movements. This demonstration presents our Optimization-based Customizable Retargeting Algorithm (OCRA), designed for real-time motion mapping between dissimilar kinematic chains. OCRA functions well with widely varying robot-arm joint configurations. The presenter will use a commercial motion-capture suit to teleoperate the upper body of a NAO humanoid robot, demonstrating OCRA's ability to create intuitive, human-like movements in real time.

Background: New surgeons experience heavy workload during robot-assisted surgery partially because they must use vision to compensate for the lack of haptic feedback. We hypothesize that providing realistic haptic feedback during dry-lab simulation training may accelerate learning and reduce workload during subsequent surgery on patients. Methods: We conducted a single-blinded study with twelve general surgery residents (third and seventh post-graduate year, PGY) randomized into haptic and control groups. Participants performed five simulated bariatric surgeries on a custom inanimate simulator followed by live robot-assisted sleeve gastrectomies (RASGs) using da Vinci robots. The haptic group received naturalistic haptic feedback of instrument vibrations during their first four simulated procedures. Participants completed pre-/post-procedure STAI and post-procedure NASA-TLX questionnaires in both simulation and the operating room (OR). Results: Higher PGY level (simulation: p<0.001, OR p=0.004), shorter operative time (simulation: p<0.001, OR: p=0.003), and lower pre-procedure STAI (simulation: p=0.003, OR: p<0.001) were significantly associated with lower self-reported overall workload in both operative settings; PGY-7s reported about 10% lower workload than PGY-3s. The haptic group had significantly lower overall covariate-adjusted NASA-TLX during the fourth (p=0.03) and fifth (p=0.04) simulated procedures and across all OR procedures (p=0.047), though not for only the first three OR procedures. Haptic feedback reduced physical demand (simulation: p<0.001, OR: p=0.001) and increased perceived performance (simulation: p=0.031, OR: p<0.001) in both settings. Conclusion: Haptic feedback of instrument vibrations provided during robotic surgical simulation reduces trainee workload during both simulation and live OR cases. The implications of workload reduction and its potential effects on patient safety warrant further investigation.

Self-touch is crucial in human communication, psychology, and disease transmission, yet existing methods for detecting self-touch are often invasive or limited in scope. This study systematically investigates the feasibility of using non-invasive electrical bioimpedance for detecting discrete self-touch poses across individuals. While previous research has focused on classifying defined self-touch poses, our work explores how various poses cause bioimpedance changes, providing insights into the underlying physiological mechanisms. We thus created a dataset of 27 genuine self-touch poses, including skin-to-skin contact between the hands and face and skin-to-clothing contact between the hands and chest, alongside six adversarial mid-air gestures. We then measured the wrist-to-wrist bioimpedance of 30 adults (15 female, 15 male) across these poses, with each measurement preceded by a no-touch pose serving as a baseline. Statistical analysis of the measurements showed that skin-to-skin contacts cause significant changes in bioimpedance magnitude between 237.8 kHz and 4.1 MHz, while adversarial gestures do not; skin-to-clothing contacts cause less-significant changes due to the influence and variability of the clothing material. Furthermore, our analysis highlights the sensitivity of bioimpedance to the body parts involved, skin contact area, and individual's characteristics. Our contributions are two-fold: (1) we demonstrate that bioimpedance offers a practical, non-invasive solution for detecting self-touch poses involving skin-to-skin contact, (2) researchers can leverage insights from our study to determine whether a pose can be detected without extensive testing.

Coupled hyperelastic balloons that anchor alternately against a lumen wall provide an appealing locomotion method for soft robots, especially for pipe inspection and medical interventions. However, it is still challenging to use a single fluid channel to obtain a practical balloon actuation sequence, where the rear anchor is both the first to inflate and the first to deflate. The common solution delays the front balloon's reaction using fluid dynamics, producing a slow and/or bulky system. This study presents a new method that utilizes an inextensible sleeve along with geometry and mechanical properties to set the pressure-stretch curve of two silicone-rubber balloons so they could serve as the rear and front anchors when driven from a single fluid supply. Experimental measurements and numerical simulations compare the characteristic curves of thin and thick spherical balloons with identical diameters to that of a thin balloon inside a rigid encasing sleeve that delays its initial expansion. Pairing this encased thin balloon with a non-encased thick balloon yields the desired asymmetric actuation sequence. A physical demonstration of the behavior needed for self-propelling robots is achieved by placing such balloons within rigid tubes, connecting them to a shared supply, and sequentially adding and removing fluid.

How teams collaborate to perform complex tasks , from team sports to surgical procedures, has previously been investigated via multimodal sensing and analysis. Ultra-wideband (UWB) positioning systems are highly mobile and can be used to track collaborating team members even in cramped environments. However, the sampling rate of UWB systems is inversely proportional to the number of people tracked, and their accuracy is hindered by electromagnetic occlusion. To improve position and orientation estimation during team collaborative studies, we propose to fuse UWB positioning with a wearable inertial measurement unit (IMU) by applying an error-state extended Kalman filter (ES-EKF). This filter offers faster and more consistent estimation and remains functional even in the absence of UWB input. Single-human and multi-human sessions were recorded and filtered for evaluation against ground truth from optical motion capture. By integrating IMU readings, the ES-EKF increases the sampling rate from 0.5-20 Hz to 100 Hz. Even by correcting only planar position in the room, the ES-EKF yields improved results over UWB in four out of six DOF: lateral and longitudinal position and yaw and pitch orientation.

Grounded force-feedback (GFF) devices are a well-established and diverse category of haptic technology based on robotic arms. However, the number of designs and their specifications make it challenging to compare devices effectively. We address this challenge by presenting Haptify, a benchmarking system capable of evaluating GFF haptic devices in a thorough, fair, and non-invasive way. The user holds the instrumented device end-effector and moves it through a series of passive and active experiments. Haptify captures the interaction between the hand, device, and ground using a seven-camera optical motion-capture system, a custom 60-cm-square force plate, and a customized sensing end-effector. We propose six key metrics for evaluating GFF device performance: workspace shape, global free-space forces, global free-space vibrations, local dynamic forces and torques, frictionless surface rendering, and stiffness rendering. We then benchmark two commercial haptic devices using Haptify. The more expensive Touch X has a smaller workspace than the 3D Systems Touch, but it outputs smaller free-space forces and vibrations, smaller and more predictable dynamic forces and torques, and higher-quality renderings of a frictionless surface and high stiffness.