Institute Homepage
Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Haptipedia

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

We combine frontal camera images with wrist-to-wrist bioimpedance sensing to reconstruct sign-language motions.
In collaboration with the PS Department, we fuse computer vision [File Icon] with electrical bioimpedance measured between the wrists [File Icon] to reliably reconstruct sign-language movements in 3D.

Sign language is the primary means of communication for more than 70 million Deaf people worldwide. While video dictionaries are widely used for learning signs, they have limitations in terms of viewpoint and integration into AR/VR applications. Converting these videos into expressive 3D avatars could enhance learning and accessibility, but existing reconstruction methods struggle with the inherent challenges of sign language movements, such as rapid motion, self-occlusions, and complex finger articulation.

We propose a novel approach that combines complementary sensing modalities to achieve robust sign language capture. Our research objective is to test the hypothesis that fusing computer vision with electrical bioimpedance sensing between the wrists enables accurate reconstruction of signing, even in challenging real-world conditions.

Our first work, SGNify, introduces linguistic constraints derived from universal rules of sign languages to improve 3D reconstruction from monocular video [File Icon]. We evaluated our system through quantitative analysis using motion capture data as ground truth, as well as perceptual studies with fluent signers. Results showed that our approach significantly outperforms existing methods, producing more accurate reconstructions. However, the resulting avatars still suffer from depth ambiguities, especially during self-contact events.

Based on our findings that self-contact remains ambiguous in video alone, we then investigated whether electrical bioimpedance measured between the wrists could be used to detect self-contact [File Icon]. We conducted a systematic data collection with 30 participants, each performing 33 different poses that involved various combinations of body parts (hands touching each other, hands touching face, hands touching chest) and contact sizes (from single fingertip to full hand) and a no-contact baseline pose. We measured their bioimpedance while sweeping frequencies from 100 Hz to 5.1 MHz. Our results demonstrated that the bioimpedance magnitude at high frequencies can be used to reliably detect skin-to-skin contact across individuals with diverse physical characteristics. Ongoing work is evaluating the extent to which direct-sensed bioimpedance improves SGNify's performance.

Members

Haptic Intelligence, Perceiving Systems
Maria-Paola Forte
  • Postdoctoral Researcher
Perceiving Systems
Peter Kulits
  • Doctoral Researcher
Perceiving Systems
Chun-Hao Paul Huang
  • Guest Scientist
Perceiving Systems
Vassilis Choutas
Perceiving Systems
Dimitris Tzionas
  • Guest Scientist
Haptic Intelligence
Yasemin Vardar
Haptic Intelligence
Bernard Javot
  • Research Engineer
Perceiving Systems
Nikos Athanasiou
  • Guest Scientist
Haptic Intelligence
Giulia Ballardini
  • Research Scientist
Perceiving Systems
Michael Black
Emeritus / Acting Director
Haptic Intelligence
Katherine J. Kuchenbecker
Director

Publications

Haptic Intelligence Perceiving Systems Ph.D. Thesis An Interdisciplinary Approach to Human Pose Estimation: Application to Sign Language Forte, M. University of Tübingen, Tübingen, Germany, November 2025, Department of Computer Science (Published)
Accessibility legislation mandates equal access to information for Deaf communities. While videos of human interpreters provide optimal accessibility, they are costly and impractical for frequently updated content. AI-driven signing avatars offer a promising alternative, but their development is limited by the lack of high-quality 3D motion-capture data at scale. Vision-based motion-capture methods are scalable but struggle with the rapid hand movements, self-occlusion, and self-touch that characterize sign language. To address these limitations, this dissertation develops two complementary solutions. SGNify improves hand pose estimation by incorporating universal linguistic rules that apply to all sign languages as computational priors. Proficient signers recognize the reconstructed signs as accurately as those in the original videos, but depth ambiguities along the camera axis can still produce incorrect reconstructions for signs involving self-touch. To overcome this remaining limitation, BioTUCH integrates electrical bioimpedance sensing between the wrists of the person being captured. Systematic measurements show that skin-to-skin contact produces distinctive bioimpedance reductions at high frequencies (240 kHz to 4.1 MHz), enabling reliable contact detection. BioTUCH uses the timing of these self-touch events to refine arm poses, producing physically plausible arm configurations and significantly reducing reconstruction error. Together, these contributions support the scalable collection of high-quality 3D sign language motion data, facilitating progress toward AI-driven signing avatars.
BibTeX
Haptic Intelligence Perceiving Systems Conference Paper Contact-Aware Refinement of Human Pose Pseudo-Ground Truth via Bioimpedance Sensing Forte, M., Athanasiou, N., Ballardini, G., Bartels, J. U., Kuchenbecker, K. J., Black, M. J. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 5071-5080, Honolulu, USA, October 2025, Nikos Athanasiou and Giulia Ballardini contributed equally to this publication (Published) pdf URL BibTeX
Haptic Intelligence Perceiving Systems Article Wrist-to-Wrist Bioimpedance Can Reliably Detect Discrete Self-Touch Forte, M., Vardar, Y., Javot, B., Kuchenbecker, K. J. IEEE Transactions on Instrumentation and Measurement, 74(4006511):1-11, April 2025 (Published)
Self-touch is crucial in human communication, psychology, and disease transmission, yet existing methods for detecting self-touch are often invasive or limited in scope. This study systematically investigates the feasibility of using non-invasive electrical bioimpedance for detecting discrete self-touch poses across individuals. While previous research has focused on classifying defined self-touch poses, our work explores how various poses cause bioimpedance changes, providing insights into the underlying physiological mechanisms. We thus created a dataset of 27 genuine self-touch poses, including skin-to-skin contact between the hands and face and skin-to-clothing contact between the hands and chest, alongside six adversarial mid-air gestures. We then measured the wrist-to-wrist bioimpedance of 30 adults (15 female, 15 male) across these poses, with each measurement preceded by a no-touch pose serving as a baseline. Statistical analysis of the measurements showed that skin-to-skin contacts cause significant changes in bioimpedance magnitude between 237.8 kHz and 4.1 MHz, while adversarial gestures do not; skin-to-clothing contacts cause less-significant changes due to the influence and variability of the clothing material. Furthermore, our analysis highlights the sensitivity of bioimpedance to the body parts involved, skin contact area, and individual's characteristics. Our contributions are two-fold: (1) we demonstrate that bioimpedance offers a practical, non-invasive solution for detecting self-touch poses involving skin-to-skin contact, (2) researchers can leverage insights from our study to determine whether a pose can be detected without extensive testing.
DOI BibTeX
Haptic Intelligence Perceiving Systems Conference Paper Reconstructing Signing Avatars from Video Using Linguistic Priors Forte, M., Kulits, P., Huang, C., Choutas, V., Tzionas, D., Kuchenbecker, K. J., Black, M. J. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 12791-12801, Vancouver, Canada, CVPR, June 2023 (Published)
Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video; occlusion, noise, and motion blur make this task difficult. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose- and shape-estimation methods on SL videos. A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos. Code and data are available at sgnify.is.tue.mpg.de.
pdf arXiv project code DOI URL BibTeX