Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Institute Homepage

Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Computational Modeling of Finger-Surface Contact

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Modeling Hand Deformations During Contact

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Immersive VR for Phantom Limb Pain

Visual and Haptic Perception of Real Surfaces

Haptipedia

Gait Propulsion Trainer

TouchTable: A Musical Interface with Haptic Feedback for DJs

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

How Should Robots Hug?

Hierarchical Structure for Learning from Demonstration

Fabrication of HuggieBot 2.0: A More Huggable Robot

Learning Haptic Adjectives from Tactile Data

Feeling With Your Eyes: Visual-Haptic Surface Interaction

S-BAN

General Tactile Sensor Model

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

20241029 haptic intelligence 90 — Mayumi Mohan wears an inertial motion-capture suit to control the arm and head movements of a NAO robot; the joint angle commands are calculated in real time by our algorithm, OCRA.

Aiming to enhance social exercise robots [

], this project investigates how to efficiently and effectively map human movements to robot movements for natural whole-arm teleoperation [

]. OCRA, our optimization-based customizable retargeting algorithm [

], requires minimal adjustments to function well for a wide variety of operators and robots; importantly, it considers the entire arm shape rather than only end-effector position and orientation.

Within the realm of HRI, prior studies often underutilize robots' physical motion capabilities, focusing instead on verbal or affective interactions. Teleoperation is a key tool for leveraging expert knowledge to develop and evaluate social robot behaviors. Motion capture-based teleoperation is considered one of the most intuitive methods for non-expert users to teach behaviors to robots. However, researchers in HRI come from varied backgrounds and work with diverse robotic systems. To address this, we introduce OCRA, an optimization-based, customizable retargeting algorithm for real-time teleoperation of any serial robotic arm [].

OCRA adapts to various robots and allows customization via a novel arm-skeleton error term. By considering entire arm shape and hand orientation, the algorithm enables expressive, natural motion for social-physical interactions such as high-fives or fist bumps []. To evaluate the effectiveness of OCRA, we conducted a user study where participants compared OCRA-generated motions with human demonstrations. Results confirmed that OCRA produces human-like motion. The algorithm utilizes full-body motion capture to map movements onto robots such as Baxter and NAO [].

OCRA is operator-agnostic, requiring no additional setup between users once configured for a robot. This feature enables non-roboticists, such as clinicians, to personalize robot exercises without advanced robotics expertise, facilitating seamless integration into rehabilitation settings. To assess clinician interest in utilizing such a system with their clients, we gathered expert perspectives through semi-structured interviews on teleoperated social exercise robots []. Experts broadly agreed that teleoperation improves perceptions of social robots, alleviating concerns about their role in therapy.

Future work will evaluate OCRA's usability across platforms and explore autonomous learning from teleoperated behaviors. Our vision is to enable social robots to act as exercise partners or coaches in rehabilitation settings []. Teleoperated behaviors can be recorded, autonomously learned, and repeated, paving the way for intuitive, personalized robot-assisted therapy.

Members

Haptic Intelligence

Mayumi Mohan

Research Scientist

Haptic Intelligence

Rachael Lorsa

Research Scientist

Haptic Intelligence

Moaaz Hudhud Mughrabi

Doctoral Researcher

Haptic Intelligence

Sophia Ling

Intern

Haptic Intelligence

Haliza MatHusin

Research Scientist

Haptic Intelligence

Katherine J. Kuchenbecker

Director

Publications

Haptic Intelligence Conference Paper My Robot, My Motion: Expressive Real-Time Teleoperation Mohan, M., Kuchenbecker, K. J. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 1797-1799, Hands-on demonstration presented at the ACM/IEEE International Conference on Human-Robot Interaction (HRI), Melbourne, Australia, April 2025 (Published)

Abstract ›

Humanoid social robots need to be able to move expressively. Traditional manipulation-focused teleoperation systems primarily control the end-effector's position and orientation, neglecting the extra degrees of freedom in human and robotic arms, which can lead to unnatural movements. This demonstration presents our Optimization-based Customizable Retargeting Algorithm (OCRA), designed for real-time motion mapping between dissimilar kinematic chains. OCRA functions well with widely varying robot-arm joint configurations. The presenter will use a commercial motion-capture suit to teleoperate the upper body of a NAO humanoid robot, demonstrating OCRA's ability to create intuitive, human-like movements in real time.

DOI URL BibTeX

Haptic Intelligence Miscellaneous Demonstration: OCRA - A Kinematic Retargeting Algorithm for Expressive Whole-Arm Teleoperation Mohan, M., Kuchenbecker, K. J. Hands-on demonstration presented at the Conference on Robot Learning (CoRL), Munich, Germany, November 2024 (Published)

Abstract ›

Traditional teleoperation systems focus on controlling the pose of the end-effector (task space), often neglecting the additional degrees of freedom present in human and many robotic arms. This demonstration presents the Optimization-based Customizable Retargeting Algorithm (OCRA), which was designed to map motions from one serial kinematic chain to another in real time. OCRA is versatile, accommodating any robot joint counts and segment lengths, and it can retarget motions from human arms to kinematically different serial robot arms with revolute joints both expressively and efficiently. One of OCRA's key features is its customizability, allowing the user to adjust the emphasis between hand orientation error and the configuration error of the arm's central line, which we call the arm skeleton. To evaluate the perceptual quality of the motions generated by OCRA, we conducted a video-watching study with 70 participants; the results indicated that the algorithm produces robot motions that closely resemble human movements, with a median rating of 78/100, particularly when the arm skeleton error weight and hand orientation error are balanced. In this demonstration, the presenter will wear an Xsens MVN Link and teleoperate the arms of a NAO child-size humanoid robot to highlight OCRA's ability to create intuitive and human-like whole-arm motions.

BibTeX

Haptic Intelligence Conference Paper Expert Perception of Teleoperated Social Exercise Robots Mohan, M., Mat Husin, H., Kuchenbecker, K. J. In Companion of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 769-773, Boulder, USA, March 2024, Late-Breaking Report (LBR) (5 pages) presented at the IEEE/ACM International Conference on Human-Robot Interaction (HRI) (Published)

Abstract ›

Social robots could help address the growing issue of physical inactivity by inspiring users to engage in interactive exercise. Nevertheless, the practical implementation of social exercise robots poses substantial challenges, particularly in terms of personalizing their activities to individuals. We propose that motion-capture-based teleoperation could serve as a viable solution to address these needs by enabling experts to record custom motions that could later be played back without their real-time involvement. To gather feedback about this idea, we conducted semi-structured interviews with eight exercise-therapy professionals. Our findings indicate that experts' attitudes toward social exercise robots become more positive when considering the prospect of teleoperation to record and customize robot behaviors.

DOI BibTeX

Haptic Intelligence Ph.D. Thesis Gesture-Based Nonverbal Interaction for Exercise Robots Mohan, M. University of Tübingen, Tübingen, Germany, October 2023, Department of Computer Science (Published)

Abstract ›

When teaching or coaching, humans augment their words with carefully timed hand gestures, head and body movements, and facial expressions to provide feedback to their students. Robots, however, rarely utilize these nuanced cues. A minimally supervised social robot equipped with these abilities could support people in exercising, physical therapy, and learning new activities. This thesis examines how the intuitive power of human gestures can be harnessed to enhance human-robot interaction. To address this question, this research explores gesture-based interactions to expand the capabilities of a socially assistive robotic exercise coach, investigating the perspectives of both novice users and exercise-therapy experts. This thesis begins by concentrating on the user's engagement with the robot, analyzing the feasibility of minimally supervised gesture-based interactions. This exploration seeks to establish a framework in which robots can interact with users in a more intuitive and responsive manner. The investigation then shifts its focus toward the professionals who are integral to the success of these innovative technologies: the exercise-therapy experts. Roboticists face the challenge of translating the knowledge of these experts into robotic interactions. We address this challenge by developing a teleoperation algorithm that can enable exercise therapists to create customized gesture-based interactions for a robot. Thus, this thesis lays the groundwork for dynamic gesture-based interactions in minimally supervised environments, with implications for not only exercise-coach robots but also broader applications in human-robot interaction.

BibTeX

Haptic Intelligence Miscellaneous OCRA: An Optimization-Based Customizable Retargeting Algorithm for Teleoperation Mohan, M., Kuchenbecker, K. J. Workshop paper (3 pages) presented at the ICRA Workshop Toward Robot Avatars, London, UK, May 2023 (Published)

Abstract ›

This paper presents a real-time optimization-based algorithm for mapping motion between two kinematically dissimilar serial linkages, such as a human arm and a robot arm. OCRA can be customized based on the target task to weight end-effector orientation versus the configuration of the central line of the arm, which we call the skeleton. A video-watching study (N=70) demonstrated that when this algorithm considers both the hand orientation and the arm skeleton, it creates robot arm motions that users perceive to be highly similar to those of the human operator, indicating OCRA would be suitable for telerobotics and telepresence through avatars.

URL BibTeX

Haptic Intelligence Miscellaneous Evaluation of a Teleoperated Robotic Exercise Coach Mohan, M., Mat Husin, H., Kuchenbecker, K. J. Workshop paper (4 pages) presented at the HRI Workshop on Workshop YOUR study design! Participatory critique and refinement of participants’ studies, Virtual, March 2021 (Published) BibTeX

Haptic Intelligence Miscellaneous A Design Tool for Therapeutic Social-Physical Human-Robot Interactions Mohan, M., Kuchenbecker, K. J. 727-729, Workshop paper (3 pages) presented at the HRI Pioneers Workshop, Daegu, South Korea, March 2019 (Published)

Abstract ›

We live in an aging society; social-physical human-robot interaction has the potential to keep our elderly adults healthy by motivating them to exercise. After summarizing prior work, this paper proposes a tool that can be used to design exercise and therapy interactions to be performed by an upper-body humanoid robot. The interaction design tool comprises a teleoperation system that transmits the operator’s arm motions, head motions and facial expression along with an interface to monitor and assess the motion of the user interacting with the robot. We plan to use this platform to create dynamic and intuitive exercise interactions.

DOI BibTeX