Institute Homepage
Institute Homepage Sign In

Back

Research Overview

Natural Embodied Touch

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Fingertip Sliding

Material Intelligence of Animal Whiskers

Artificial Touch Sensing and Processing

Giving Touch to Soft Robot Fingertips Using Vision, Audio and Machine Learning

Capturing and Recognizing Multimodal Surface Interactions

Multimodal Puncture Detection for Urgent Percutaneous Therapies

Haptic Actuation

Quantifying the Quality of Haptic Interfaces

Shape-Changing Haptic Interfaces

Generating Clear Vibrotactile Cues with Magnets Embedded in a Soft Finger Sheath

Salient Full-Fingertip Haptic Feedback Enabled by Wearable Electrohydraulic Actuation

Cutaneous Electrohydraulic (CUTE) Wearable Devices for Pleasant Broad-Bandwidth Haptic Cues

Modeling Finger-Touchscreen Contact during Electrovibration

Perception of Ultrasonic Friction Pulses

CAPT Motor: A Two-Phase Ironless Motor Structure

Immersive Teleoperation

4D Intraoperative Surgical Perception: Anatomical Shape Reconstruction from Multiple Viewpoints

Visual-Inertial Force Estimation in Robotic Surgery

Enhancing Robotic Surgical Training

AiroTouch: Naturalistic Vibrotactile Feedback for Large-Scale Telerobotic Assembly

Optimization-Based Whole-Arm Teleoperation for Natural Human-Robot Interaction

Human-Robot Interaction

Enhancing HRI through Responsive Multimodal Feedback

Haptic Empathetic Robot Animal (HERA)

How does HuggieBot compare to a human hugging partner?

HuggieBot: Evolution of an Interactive Hugging Robot with Visual and Haptic Perception

Human Movement

Reconstructing Sign-Language Movements from Images and Bioimpedance Measurements

Advancing Gait-Retraining Techniques

Multimodal Sensing Reveals the Dynamics of Time-Constrained Teamwork

Previous HI Projects

Finger-Surface Contact Mechanics in Diverse Moisture Conditions

Perceptual Integration of Contact Force Components During Tactile Stimulation

Dynamic Models and Wearable Tactile Devices for the Fingertips

Novel Designs and Rendering Algorithms for Fingertip Haptic Devices

Dimensional Reduction from 3D to 1D for Realistic Vibration Rendering

Prendo: Analyzing Human Grasping Strategies for Visually Occluded Objects

Learning Upper-Limb Exercises from Demonstrations

Minimally Invasive Surgical Training with Multimodal Feedback and Automatic Skill Evaluation

Efficient Large-Area Tactile Sensing for Robot Skin

Haptic Feedback and Autonomous Reflexes for Upper-limb Prostheses

Gait Retraining

Intraoperative AR Assistance for Robot-Assisted Minimally Invasive Surgery

Haptipedia

Exercise Games with Baxter

Intuitive Social-Physical Robots for Exercise

Insight: a Haptic Sensor Powered by Vision and Machine Learning

Haptic Intelligence Members Publications

Enhancing HRI through Responsive Multimodal Feedback

The robot in one of our studies sought to communicate three movement tasks: the depicted task required the user to walk to the robot and make contact with both of its end effectors. The robot provided half of the users with real-time formative feedback, which greatly enhanced their task performance and comprehension.
This research project uses our Robot Interaction Studio [File Icon] to explore how social-physical robots should behave to create engaging movement-based interactions with users. Our studies have shown that providing multi-modal feedback (gestures, facial expressions, sounds) based on real-time user motion enables robots to promote physical activity [File Icon] and enhance task comprehension [File Icon].

We investigate how multi-modal feedback-gestures, facial expressions, and sounds can promote physical activity and enhance task comprehension by exploring how social-physical robots can create engaging, movement-based interactions.

To study such interactions, we developed eight exercise games for the Rethink Robotics Baxter Research Robot that examine how people respond to social-physical interactions with a humanoid robot. These games were developed with the input and guidance of experts in game design, therapy, and rehabilitation [File Icon], and validated via extensive pilot testing [File IconFile Icon]. Results from our game evaluation study with 20 younger and 20 older adults show that bimanual humanoid robots could motivate users to stay healthy via social-physical exercise [File IconFile IconFile Icon]. However, we found that these robots must have customizable behaviors and end-user monitoring capabilities to be viable in real-world scenarios.

To address the above needs, we developed the Robot Interaction Studio, a platform for enabling minimally supervised human-robot interaction [File Icon]. This system combines Captury Live, a real-time markerless motion-capture system, with a ROS-compatible robot to estimate user actions in real time and provide corrective feedback [File Icon]. We evaluated this platform via a user study where Baxter sequentially presented the user with three gesture-based cues in randomized order [File Icon]. The study results showed that participants explored the interaction workspace, mimicked Baxter, and interacted with Baxter's hands even without instructions.

Given the importance of gestures in motor learning, we examined how nonverbal robot feedback influences human behavior [File Icon]. Inspired by educational research, we tested formative feedback (real-time corrections) and summative feedback (post-task scores) across three tasks: positioning, mimicking arm poses, and contacting the robot's hands. We found that both formative and summative feedback improved user task performance. However, only formative feedback enhanced task comprehension.

These findings highlight how real-time, multi-modal interactions enable robots to encourage physical activity and improve learning, making them valuable tools for exercise and rehabilitation.

This research project involves collaborations with Naomi T. Fitter (Oregon State University) and Michelle J. Johnson (University of Pennsylvania).

Members

Haptic Intelligence
Mayumi Mohan
  • Research Scientist
Haptic Intelligence
Cara Nunez
  • Visitor
Haptic Intelligence
Katherine J. Kuchenbecker
Director

Publications

Haptic Intelligence Article Closing the Loop in Minimally Supervised Human-Robot Interaction: Formative and Summative Feedback Mohan, M., Nunez, C. M., Kuchenbecker, K. J. Scientific Reports, 14(1):10564, May 2024 (Published)
Human instructors fluidly communicate with hand gestures, head and body movements, and facial expressions, but robots rarely leverage these complementary cues. A minimally supervised social robot with such skills could help people exercise and learn new activities. Thus, we investigated how nonverbal feedback from a humanoid robot affects human behavior. Inspired by the education literature, we evaluated formative feedback (real-time corrections) and summative feedback (post-task scores) for three distinct tasks: positioning in the room, mimicking the robot's arm pose, and contacting the robot's hands. Twenty-eight adults completed seventy-five 30-second-long trials with no explicit instructions or experimenter help. Motion-capture data analysis shows that both formative and summative feedback from the robot significantly aided user performance. Additionally, formative feedback improved task understanding. These results show the power of nonverbal cues based on human movement and the utility of viewing feedback through formative and summative lenses.
DOI BibTeX
Haptic Intelligence Article How Should Robots Exercise with People? Robot-Mediated Exergames Win with Music, Social Analogues, and Gameplay Clarity Fitter, N. T., Mohan, M., Preston, R. C., Johnson, M. J., Kuchenbecker, K. J. Frontiers in Robotics and AI, 10(1155837):1-18, January 2024 (Published)
The modern worldwide trend toward sedentary behavior comes with significant health risks. An accompanying wave of health technologies has tried to encourage physical activity, but these approaches often yield limited use and retention. Due to their unique ability to serve as both a health-promoting technology and a social peer, we propose robots as a game-changing solution for encouraging physical activity. This article analyzes the eight exergames we previously created for the Rethink Baxter Research Robot in terms of four key components that are grounded in the video-game literature: repetition, pattern matching, music, and social design. We use these four game facets to assess gameplay data from 40 adult users who each experienced the games in balanced random order. In agreement with prior research, our results show that relevant musical cultural references, recognizable social analogues, and gameplay clarity are good strategies for taking an otherwise highly repetitive physical activity and making it engaging and popular among users. Others who study socially assistive robots and rehabilitation robotics can benefit from this work by considering the presented design attributes to generate future hypotheses and by using our eight open-source games to pursue follow-up work on social-physical exercise with robots.
DOI BibTeX
Haptic Intelligence Ph.D. Thesis Gesture-Based Nonverbal Interaction for Exercise Robots Mohan, M. University of Tübingen, Tübingen, Germany, October 2023, Department of Computer Science (Published)
When teaching or coaching, humans augment their words with carefully timed hand gestures, head and body movements, and facial expressions to provide feedback to their students. Robots, however, rarely utilize these nuanced cues. A minimally supervised social robot equipped with these abilities could support people in exercising, physical therapy, and learning new activities. This thesis examines how the intuitive power of human gestures can be harnessed to enhance human-robot interaction. To address this question, this research explores gesture-based interactions to expand the capabilities of a socially assistive robotic exercise coach, investigating the perspectives of both novice users and exercise-therapy experts. This thesis begins by concentrating on the user's engagement with the robot, analyzing the feasibility of minimally supervised gesture-based interactions. This exploration seeks to establish a framework in which robots can interact with users in a more intuitive and responsive manner. The investigation then shifts its focus toward the professionals who are integral to the success of these innovative technologies: the exercise-therapy experts. Roboticists face the challenge of translating the knowledge of these experts into robotic interactions. We address this challenge by developing a teleoperation algorithm that can enable exercise therapists to create customized gesture-based interactions for a robot. Thus, this thesis lays the groundwork for dynamic gesture-based interactions in minimally supervised environments, with implications for not only exercise-coach robots but also broader applications in human-robot interaction.
BibTeX
Haptic Intelligence Conference Paper Robot Interaction Studio: A Platform for Unsupervised HRI Mohan, M., Nunez, C. M., Kuchenbecker, K. J. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 3330-3336, Xi’an, China, May 2021 (Published)
Robots hold great potential for supporting exercise and physical therapy, but such systems are often cumbersome to set up and require expert supervision. We aim to solve these concerns by combining Captury Live, a real-time markerless motion-capture system, with a Rethink Robotics Baxter Research Robot to create the Robot Interaction Studio. We evaluated this platform for unsupervised human-robot interaction (HRI) through a 75-minute-long user study with seven adults who were given minimal instructions and no feedback about their actions. The robot used sounds, facial expressions, facial colors, head motions, and arm motions to sequentially present three categories of cues in randomized order while constantly rotating its face screen to look at the user. Analysis of the captured user motions shows that the cue type significantly affected the distance subjects traveled and the amount of time they spent within the robot’s reachable workspace, in alignment with the design of the cues. Heat map visualizations of the recorded user hand positions confirm that users tended to mimic the robot’s arm poses. Despite some initial frustration, taking part in this study did not significantly change user opinions of the robot. We reflect on the advantages of the proposed approach to unsupervised HRI as well as the limitations and possible future extensions of our system.
DOI BibTeX
Haptic Intelligence Miscellaneous How Does Real-Time Feedback Affect Communicative Actions in Social-Physical Human-Robot Interaction? Mohan, M., Nunez, C. M., Kuchenbecker, K. J. Workshop paper (2 pages) presented at the ROMAN Workshop on Quality of Interaction in Socially Assistive Robots (QISAR), Virtual, August 2020 (Published)
Social robots are becoming more common, especially to motivate older adults to exercise and stay healthy. To increase the effectiveness of such robots, researchers need to develop autonomous interactions that are understandable to the user without help from a human operator. Motivated by this requirement, we have programmed a Baxter robot to play an exercise game via multi-modal non-verbal communication. When the user is confused or makes a mistake, the robot can optionally provide corrective feedback based on real-time measurements of user actions. We hypothesize feedback will improve both the user's physical performance and the user's opinion of the robot's social skills during a planned experiment.
BibTeX
Haptic Intelligence Article Exercising with Baxter: Preliminary Support for Assistive Social-Physical Human-Robot Interaction Fitter, N. T., Mohan, M., Kuchenbecker, K. J., Johnson, M. J. Journal of NeuroEngineering and Rehabilitation, 17:1-22, February 2020 (Published)
Background: The worldwide population of older adults will soon exceed the capacity of assisted living facilities. Accordingly, we aim to understand whether appropriately designed robots could help older adults stay active at home. Methods: Building on related literature as well as guidance from experts in game design, rehabilitation, and physical and occupational therapy, we developed eight human-robot exercise games for the Baxter Research Robot, six of which involve physical human-robot contact. After extensive iteration, these games were tested in an exploratory user study including 20 younger adult and 20 older adult users. Results: Only socially and physically interactive games fell in the highest ranges for pleasantness, enjoyment, engagement, cognitive challenge, and energy level. Our games successfully spanned three different physical, cognitive, and temporal challenge levels. User trust and confidence in Baxter increased significantly between pre- and post-study assessments. Older adults experienced higher exercise, energy, and engagement levels than younger adults, and women rated the robot more highly than men on several survey questions. Conclusions: The results indicate that social-physical exercise with a robot is more pleasant, enjoyable, engaging, cognitively challenging, and energetic than similar interactions that lack physical touch. In addition to this main finding, researchers working in similar areas can build on our design practices, our open-source resources, and the age-group and gender differences that we found.
DOI BibTeX
Haptic Intelligence Miscellaneous Exercising with Baxter: Design and Evaluation of Assistive Social-Physical Human-Robot Interaction Fitter, N. T., Mohan, M., Kuchenbecker, K. J., Johnson, M. J. Workshop paper (6 pages) presented at the HRI Workshop on Personal Robots for Exercising and Coaching, Chicago, USA, March 2018 (Published)
The worldwide population of older adults is steadily increasing and will soon exceed the capacity of assisted living facilities. Accordingly, we aim to understand whether appropriately designed robots could help older adults stay active and engaged while living at home. We developed eight human-robot exercise games for the Baxter Research Robot with the guidance of experts in game design, therapy, and rehabilitation. After extensive iteration, these games were employed in a user study that tested their viability with 20 younger and 20 older adult users. All participants were willing to enter Baxter’s workspace and physically interact with the robot. User trust and confidence in Baxter increased significantly between pre- and post-experiment assessments, and one individual from the target user population supplied us with abundant positive feedback about her experience. The preliminary results presented in this paper indicate potential for the use of two-armed human-scale robots for social-physical exercise interaction.
URL BibTeX
Haptic Intelligence Miscellaneous Physically Interactive Exercise Games with a Baxter Robot Fitter, N. T., Kuchenbecker, K. J. Hands-on demonstration presented at the IEEE World Haptics Conference (WHC), Munich, Germany, June 2017 (Published) BibTeX
Haptic Intelligence Miscellaneous Hand-Clapping Games with a Baxter Robot Fitter, N. T., Kuchenbecker, K. J. Hands-on demonstration presented at the ACM/IEEE International Conference on Human-Robot Interaction (HRI), Vienna, Austria, March 2017 (Published)
Robots that work alongside humans might be more effective if they could forge a strong social bond with their human partners. Hand-clapping games and other forms of rhythmic social-physical interaction may foster human-robot teamwork, but the design of such interactions has scarcely been explored. At the HRI 2017 conference, we will showcase several such interactions taken from our recent work with the Rethink Robotics Baxter Research Robot, including tempo-matching, Simon says, and Pat-a-cake-like games. We believe conference attendees will be both entertained and intrigued by this novel demonstration of social-physical HRI.
BibTeX
Haptic Intelligence Miscellaneous Designing Human-Robot Exercise Games for Baxter Fitter, N. T., Hawkes, D. T., Johnson, M. J., Kuchenbecker, K. J. Daejeon, South Korea, October 2016, Late-breaking results report presented at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (Published) BibTeX