Learning Upper-Limb Exercises from Demonstrations

Institute Homepage

Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Computational Modeling of Finger-Surface Contact

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Modeling Hand Deformations During Contact

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Immersive VR for Phantom Limb Pain

Visual and Haptic Perception of Real Surfaces

Haptipedia

Gait Propulsion Trainer

TouchTable: A Musical Interface with Haptic Feedback for DJs

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

How Should Robots Hug?

Hierarchical Structure for Learning from Demonstration

Fabrication of HuggieBot 2.0: A More Huggable Robot

Learning Haptic Adjectives from Tactile Data

Feeling With Your Eyes: Visual-Haptic Surface Interaction

S-BAN

General Tactile Sensor Model

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Learning Upper-Limb Exercises from Demonstrations

Final — (a) A user sitting in front of the PR2 robot. They each use one hand to hold an object that was specifically designed for upper-limb exercises for patients with stroke. Example trajectories from the (b) 1D, (c) 2D, and (d) pick-and-place exercises tested in the study.

Compared to conventional upper-limb therapies, robotic devices can offer multiple advantages in physical training and rehabilitation. Automation can reduce the workload of rehabilitation professionals and augment their ability to provide care to patients by facilitating intensive and repeatable exercise. Additionally, robots can provide objective assessments of a patient’s progress using onboard sensors.

In the scope of this research, we developed a learning-from-demonstration (LfD) technique that enables a general-purpose humanoid robot to lead a user through object-mediated upper-limb exercises []. Built upon our prior research [], our approach requires only tens of seconds of training data from a therapist teleoperating the robot to do the task with the user.

We model the robot behavior as a regression problem: during training, the joint distribution of robot position, velocity, and effort (force output at the end-effector) are modeled by a Gaussian mixture model (GMM), and during testing, desired robot effort is regressed from the current state (position and velocity) from the GMM. Compared to the conventional approach of learning time-based trajectories, our state-based strategy produces customized robot behavior and eliminates the need to tune gains to adapt to the user’s motor ability.

This approach was evaluated through a user study involving one occupational therapist and six people with stroke []. The therapist trained a Willow Garage PR2 on three example tasks for each client: i) periodic 1D motions, ii) periodic 2D motions, and iii) episodic pick and place operations. Both the person with stroke and the therapist then repeatedly performed the tasks alone with the robot and blindly compared the state- and time-based controllers learned from the training data.

Our results show that working models were reliably learned to enable the robot to do the exercise with the user. Furthermore, our state-based approach enabled users to be more actively involved, allowed larger excursion, and generated power outputs more similar to the therapist demonstrations. Finally, the therapist found our strategy more agreeable than the traditional time-based approach. More detailed descriptions of this project’s algorithms and results can be found in the Ph.D. thesis of Siyao Hu [].

Members

Haptic Intelligence

Siyao Hu

Haptic Intelligence

Katherine J. Kuchenbecker

Director

Publications

Haptic Intelligence Article Robotics for Occupational Therapy: Learning Upper-Limb Exercises From Demonstrations Hu, S., Mendonca, R., Johnson, M. J., Kuchenbecker, K. J. IEEE Robotics and Automation Letters, 6(4):7781-7788, October 2021 (Published)

Abstract ›

We describe a learning-from-demonstration technique that enables a general-purpose humanoid robot to lead a user through object-mediated upper-limb exercises. It needs only tens of seconds of training data from a therapist teleoperating the robot to do the task with the user. We model the robot behavior as a regression problem, inferring the desired robot effort using the end-effector's state (position and velocity). Compared to the conventional approach of learning time-based trajectories, our strategy produces customized robot behavior and eliminates the need to tune gains to adapt to the user's motor ability. In our study, one occupational therapist and six people with stroke trained a Willow Garage PR2 on three example tasks (periodic 1D and 2D motions plus episodic pick-and-place). They then repeatedly did the tasks with the robot and blindly compared the state- and time-based controllers learned from the training data. Our results show that working models were reliably obtained to allow the robot to do the exercise with the user; that our state-based approach enabled users to be more actively involved, allowed larger excursion, and generated power outputs more similar to the therapist demonstrations; and that the therapist found our strategy more agreeable than the traditional time-based approach.

DOI BibTeX

Haptic Intelligence Article Hierarchical Task-Parameterized Learning from Demonstration for Collaborative Object Movement Hu, S., Kuchenbecker, K. J. Applied Bionics and Biomechanics, 2019(9765383):1-25, December 2019 (Published)

Abstract ›

Learning from demonstration (LfD) enables a robot to emulate natural human movement instead of merely executing preprogrammed behaviors. This article presents a hierarchical LfD structure of task-parameterized models for object movement tasks, which are ubiquitous in everyday life and could benefit from robotic support. Our approach uses the task-parameterized Gaussian mixture model (TP-GMM) algorithm to encode sets of demonstrations in separate models that each correspond to a different task situation. The robot then maximizes its expected performance in a new situation by either selecting a good existing model or requesting new demonstrations. Compared to a standard implementation that encodes all demonstrations together for all test situations, the proposed approach offers four advantages. First, a simply defined distance function can be used to estimate test performance by calculating the similarity between a test situation and the existing models. Second, the proposed approach can improve generalization, e.g., better satisfying the demonstrated task constraints and speeding up task execution. Third, because the hierarchical structure encodes each demonstrated situation individually, a wider range of task situations can be modeled in the same framework without deteriorating performance. Last, adding or removing demonstrations incurs low computational load, and thus, the robot’s skill library can be built incrementally. We first instantiate the proposed approach in a simulated task to validate these advantages. We then show that the advantages transfer to real hardware for a task where naive participants collaborated with a Willow Garage PR2 robot to move a handheld object. For most tested scenarios, our hierarchical method achieved significantly better task performance and subjective ratings than both a passive model with only gravity compensation and a single TP-GMM encoding all demonstrations.

DOI BibTeX