Institute Homepage
Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Haptipedia

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Efficient Large-Area Tactile Sensing for Robot Skin

We have pioneered large tactile sensors based on electrical resistance tomography (ERT). (A) The key concept of our sensing approach. (B) A sensor prototype made of conductive textiles. (C) Demonstrations of our system’s ability to sense multiple contact locations and force magnitude.

Being able to perceive physical contact is essential for intelligent robots to work well in cluttered everyday environments. Since physical contacts can occur at any location, a successful tactile sensing system should be able to cover all of a robot’s exposed surfaces. Most researchers have pursued the creation of large tactile skin by using many sensing elements that are each responsible for a region; however, deploying many sensing elements is not efficient considering manufacturability, cost-effectiveness, and durability.

This project aims to build a robot skin that combines a sensor design and a computational approach to achieve efficient large-area tactile sensing. The key principle is using a piezoresistive material with a small number of distributed point electrodes; current is injected between successive pairs of electrodes, and the resulting voltage distribution is measured at other electrodes. We used this approach with conductive textiles to create several flexible and stretchable tactile sensor prototype [File IconFile IconFile IconFile IconFile Icon]. They successfully estimate contact location, contact shape, and normal force magnitude over a broad region.

This project opened up two subprojects to enhance large-area tactile sensing performance. The first subproject was optimizing the current injection and voltage measurement by considering the temporal locality of contacts across a large area [File Icon]. This subproject achieved a tactile sensing framerate over 400 Hz, which is five times faster than the conventional method. The second subproject was enhancing the contact estimation performance using sim-to-real transfer learning [File Icon]. This subproject demonstrated that a multiphysics model of the sensor could substantially improve the contact information estimation performance when combined with deep neural networks. In the future, we plan to apply this tactile sensing approach on a real robot surface to demonstrate a whole-body robot skin.

Members

Haptic Intelligence
Hyosang Lee
Haptic Intelligence
Gokhan Serhat
  • Guest Scientist
Haptic Intelligence
Bernard Javot
  • Research Engineer
Haptic Intelligence
Katherine J. Kuchenbecker
Director
Autonomous Learning
Huanbo Sun
  • Doctoral Researcher
Empirical Inference, Autonomous Learning
Georg Martius
Senior Research Scientist

Publications

Haptic Intelligence Miscellaneous NearContact: Accurate Human Detection using Tomographic Proximity and Contact Sensing with Cross-Modal Attention Garrofé, G., Schoeffmann, C., Zangl, H., Kuchenbecker, K. J., Lee, H. Extended abstract (4 pages) presented at the International Workshop on Human-Friendly Robotics (HFR), Munich, Germany, September 2023 (Published) BibTeX
Autonomous Learning Haptic Intelligence Empirical Inference Article Predicting the Force Map of an ERT-Based Tactile Sensor Using Simulation and Deep Networks Lee, H., Sun, H., Park, H., Serhat, G., Javot, B., Martius, G., Kuchenbecker, K. J. IEEE Transactions on Automation Science and Engineering, 20(1):425-439, January 2023 (Published)
Electrical resistance tomography (ERT) can be used to create large-scale soft tactile sensors that are flexible and robust. Good performance requires a fast and accurate mapping from the sensor's sequential voltage measurements to the distribution of force across its surface. However, particularly with multiple contacts, this task is challenging for both previously developed approaches: physics-based modeling and end-to-end data-driven learning. Some promising results were recently achieved using sim-to-real transfer learning, but estimating multiple contact locations and accurate contact forces remains difficult because simulations tend to be less accurate with a high number of contact locations and/or high force. This paper introduces a modular hybrid method that combines simulation data synthesized from an electromechanical finite element model with real measurements collected from a new ERT-based tactile sensor. We use about 290,000 simulated and 90,000 real measurements to train two deep neural networks: the first (Transfer-Net) captures the inevitable gap between simulation and reality, and the second (Recon-Net) reconstructs contact forces from voltage measurements. The number of contacts, contact locations, force magnitudes, and contact diameters are evaluated for a manually collected multi-contact dataset of 150 measurements. Our modular pipeline's results outperform predictions by both a physics-based model and end-to-end learning.
DOI BibTeX
Haptic Intelligence Article Adaptive Optimal Measurement Algorithm for ERT-Based Large-Area Tactile Sensors Park, K., Lee, H., Kuchenbecker, K. J., Kim, J. IEEE/ASME Transactions on Mechatronics, 27(1):304-314, February 2022 (Published)
Electrical resistance tomography (ERT) is an inferential imaging technique that has shown promising results for enabling large-area tactile sensors constructed from a piezoresistive sheet. The performance of such sensors is improved by increasing the number of electrodes, but the number of measurements and the computational cost also increase. In this article, we propose a new measurement algorithm for ERT-based tactile sensors: it adaptively changes the measurement pattern to be optimal for the present external stimulus. Regions of normal pressure are first detected by a base measurement pattern that maximizes the distinguishability of local conductivity changes. When a new contact is detected, a set of local patterns is selectively recruited near the pressed region to acquire more detailed information. For fast and parallel execution, the proposed algorithm is implemented with a field-programmable gate array. It is validated through indentation experiments on an ERT-based sensor that has 32 electrodes. The optimized base pattern of 100 measurements enabled a frame rate five times faster than before. Transmitting only detected contact events reduced the idle data rate to 0.5\% of its original value. The pattern adapted to new contacts with a latency of only 80 μs and an accuracy of 99.5\%, enabling efficient, high-quality real-time reconstruction of complex multicontact conditions.
DOI BibTeX
Haptic Intelligence Article Piezoresistive Textile Layer and Distributed Electrode Structure for Soft Whole-Body Tactile Skin Lee, H., Park, K., Kim, J., Kuchenbecker, K. J. Smart Materials and Structures, 30(8):085036, July 2021, Hyosang Lee and Kyungseo Park contributed equally to this publication (Published)
Tactile sensors based on electrical resistance tomography (ERT) provide pressure sensing over a large area using only a few electrodes, which is a promising property for robotic tactile skin. Most ERT-based tactile sensors employ electrodes only on the sensor's edge to avoid undesirable artifacts caused by electrode contact. The distribution of these electrodes is critical, as electrode location largely determines the sensitive regions, but only a few studies have positioned electrodes in the sensor's central region to improve the sensitivity. Establishing the use of internal electrodes on a stretchable textile needs further investigation into piezoresistive structure fabrication, measurement strategy, and calibration. This article presents a comprehensive study of an ERT-based tactile sensor with distributed electrodes. We describe key fabrication details of a layered textile-based piezoresistive structure, an iterative method for choosing the current injection pathways that yields pairwise optimal patterns, and a calibration process to account for the spatially varying sensitivity of such sensors. We demonstrate two sample sensors with electrodes located only on the boundary or distributed across the surface, and we evaluate their performance via three methods widely used to test tactile sensing in biological systems: single-point localization, two-point discrimination, and contact force estimation.
DOI BibTeX
Haptic Intelligence Patent System and Method for Simultaneously Sensing Contact Force and Lateral Strain Lee, H., Kuchenbecker, K. J. (EP20000480.2), December 2020
A tactile sensing system having a sensor component which comprises a plurality of layers stacked along a normal axis Z and a detection unit electrically connected to the sensor component, wherein the sensor component comprises a first layer, designed as a piezoresistive layer, a third layer, designed as a conductive layer which is electrically connected to the detection unit, and a second layer, designed as a spacing layer between the first layer and the third layer, wherein the first layer comprises a plurality of electrodes In electrically connected to the detection unit, wherein at least one contact force along the normal axis Z on the sensor component is detectable by the detection unit due to a change of a current distribution between the first layer and the third layer, wherein at least one lateral strain on the sensor component is detectable by the detection unit due to a change of the resistance distribution change in the piezoresistive first layer.
BibTeX
Haptic Intelligence Miscellaneous Tactile Textiles: An Assortment of Fabric-Based Tactile Sensors for Contact Force and Contact Location Burns, R. B., Thomas, N., Lee, H., Faulkner, R., Kuchenbecker, K. J. Hands-on demonstration presented at EuroHaptics, Leiden, the Netherlands, September 2020, Rachael Bevill Burns, Neha Thomas, and Hyosang Lee contributed equally to this publication (Published)
Fabric-based tactile sensors are promising for the construction of robotic skin due to their soft and flexible nature. Conductive fabric layers can be used to form piezoresistive structures that are sensitive to contact force and/or contact location. This demonstration showcases three diverse fabric-based tactile sensors we have created. The first detects dynamic tactile events anywhere within a region on a robot’s body. The second design measures the precise location at which a single low-force contact is applied. The third sensor uses electrical resistance tomography to output both the force and location of multiple simultaneous contacts applied across a surface.
BibTeX
Haptic Intelligence Autonomous Learning Conference Paper Calibrating a Soft ERT-Based Tactile Sensor with a Multiphysics Model and Sim-to-real Transfer Learning Lee, H., Park, H., Serhat, G., Sun, H., Kuchenbecker, K. J. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 1632-1638, Paris, France, May 2020 (Published)
Tactile sensors based on electrical resistance tomography (ERT) have shown many advantages for implementing a soft and scalable whole-body robotic skin; however, calibration is challenging because pressure reconstruction is an ill-posed inverse problem. This paper introduces a method for calibrating soft ERT-based tactile sensors using sim-to-real transfer learning with a finite element multiphysics model. The model is composed of three simple models that together map contact pressure distributions to voltage measurements. We optimized the model parameters to reduce the gap between the simulation and reality. As a preliminary study, we discretized the sensing points into a 6 by 6 grid and synthesized single- and two-point contact datasets from the multiphysics model. We obtained another single-point dataset using the real sensor with the same contact location and force used in the simulation. Our new deep neural network architecture uses a de-noising network to capture the simulation-to-real gap and a reconstruction network to estimate contact force from voltage measurements. The proposed approach showed 82% hit rate for localization and 0.51 N of force estimation error performance in single-contact tests and 78.5% hit rate for localization and 5.0 N of force estimation error in two-point contact tests. We believe this new calibration method has the possibility to improve the sensing performance of ERT-based tactile sensors.
DOI BibTeX
Haptic Intelligence Conference Paper Internal Array Electrodes Improve the Spatial Resolution of Soft Tactile Sensors Based on Electrical Resistance Tomography Lee, H., Park, K., Kim, J., Kuchenbecker, K. J. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 5411-5417, Montreal, Canada, May 2019, Hyosang Lee and Kyungseo Park contributed equally to this publication (Published) DOI BibTeX
Haptic Intelligence Miscellaneous A Large-Scale Fabric-Based Tactile Sensor Using Electrical Resistance Tomography Lee, H., Park, K., Kim, J., Kuchenbecker, K. J. 107-109, Hands-on demonstration (3 pages) presented at AsiaHaptics, Incheon, South Korea, November 2018 (Published)
Large-scale tactile sensing is important for household robots and human-robot interaction because contacts can occur all over a robot’s body surface. This paper presents a new fabric-based tactile sensor that is straightforward to manufacture and can cover a large area. The tactile sensor is made of conductive and non-conductive fabric layers, and the electrodes are stitched with conductive thread, so the resulting device is flexible and stretchable. The sensor utilizes internal array electrodes and a reconstruction method called electrical resistance tomography (ERT) to achieve a high spatial resolution with a small number of electrodes. The developed sensor shows that only 16 electrodes can accurately estimate single and multiple contacts over a square that measures 20 cm by 20 cm.
DOI BibTeX
Haptic Intelligence Miscellaneous Soft Multi-Axis Boundary-Electrode Tactile Sensors for Whole-Body Robotic Skin Lee, H., Kim, J., Kuchenbecker, K. J. Workshop paper (2 pages) presented at the RSS Pioneers Workshop, Pittsburgh, USA, June 2018 (Published) BibTeX