Starting from theoretically sound robotic control structures for task representation and execution, we replace analytic modules by more flexible learned ones pubLink{PetersKMNK2012}. To this end, we tackle problems such as accurate but compliant execution, learning of elementary behaviors, hierarchical composition of behaviors, and parsing complex demonstrations into elementary behaviors.
Accurate execution of movements ideally requires only low-gain controls, such that the robot can accomplish tasks without harming humans. Following a trajectory with little feedback requires accurate prediction of needed torques, which often cannot be achieved using classical methods. However, learning such models is hard: the joint-space can never be fully explored and the learning algorithm has to cope with a data stream in real time. We have developed learning methods for tasks represented in joint-space pubLink{6505} or task-space pubLink{NguyenTuongP2012}.
\emph{Executing} a task is important, but often the task itself needs to be learned. We focused on learning elementary tasks or movement primitives, which are parameterized nonlinear differential equations with desired attractor properties. We mimic how children learn new motor tasks, using imitation to initialize these movement primitives, and reinforcement learning to subsequently improve performance. We have learned ta
For more complex tasks, hierarchical solutions compose behaviors based on a large number of elementary ones. As the `drosophila' of complex behavior, we chose the task of returning table tennis balls over the net. This requires all the methods described in the previous paragraphs, as well as forms of reinforcement learning discussed below. We created a compiler that segments movements of a human teacher into elementary movements pubLink{6742,6743}. These then train the single primitives discussed above pubLink{6803,6802}. Novel behaviors, modulated by the opponent's incoming ball, are composed by mixing motor primitives pubLink{6745}. The learning system then generalizes between the primitives. This created successful returns of 88\% of balls. Further improvement is limited by the robot's hardware and reaction time.
Human players infer the direction of incoming balls from the opponent's movement, and so can prepare their stroke before the opponent even hits the ball. Inspired by this, we developed learning methods anticipating the aim of human opponent. They achieved a prediction accuracy of $35cm$, already $320ms$ before the ball is hit pubLink{WangDBVSP2012}, significantly increasing available reaction time.
Figure 0.1: The robot table tennis setup pubLink{6745}.
Recently, we developed a method maintaining multiple conflicting strategies in its motor policy pubLink{DanielNP2012_2}. This generalizes previous work on mixtures of motor primitives pubLink{6745} and puts it on stronger theoretical footing. It was tested in a game of tetherball (Figure 0.2), and may become useful in table tennis as it allows more ambiguous gameplay.
Figure 0.2: The Tetherball task pubLink{DanielNP2012_2}, which won the IROS Best Cognitive Robotics Paper Award while being finalist for both the best paper and the best student paper award.
Motor skill learning can also be applied to grasping and manipulation. We have developed methods pubLink{6436,6636} learning how to grasp novel objects (a preliminary version pubLink{6436} won the ICINCO Best Paper Award) and generalize such basic manipulations to different objects pubLink{KroemerUOP2012}. We also studied how to recognize surfaces from tactile frequency patterns obtained by sliding a finger over surfaces pubLink{7054}. The robot autonomously discovered the most relevant dimensions of the tactile data by training jointly on vision and tactile data from various textured surfaces. Subsequently, when presented with only tactile stimuli, it was able to quickly and reliably recover surface type.
The direct perception of actions allows a robot to predict the afforded actions of observed novel objects. In addition
to learning which actions are afforded, the robot must also learn to adapt its actions according to the object being manipulated. In this paper, we present a non-parametric approach to representing the affordance-bearing subparts of objects. This representation forms the basis of a kernel function for computing the similarity between different subparts. Using this kernel function, the robot can learn the required mappings to perform direct action perception. The proposed approach was successfully implemented on a real robot, which could then quickly learn to generalize
grasping and pouring actions to novel objects.
Abstract—Task-space control of redundant robot systems
based on analytical models is known to be susceptive to modeling errors. Here, data driven model learning methods may present an interesting alternative approach. However, learning models for task-space tracking control from sampled data is an illposed problem. In particular, the same input data point can yield many different output values, which can form a non-convex solution space. Because the problem is ill-posed, models cannot be learned from such data using common regression methods.
While learning of task-space control mappings is globally illposed, it has been shown in recent work that it is locally a well-defined problem. In this paper, we use this insight to formulate a local, kernel-based learning approach for online model learning for task-space tracking control. We propose a parametrization for the local model which makes an application in task-space tracking control of redundant robots possible. The model parametrization further allows us to apply the kerneltrick and, therefore, enables a formulation within the kernel learning framework. For evaluations, we show the ability of the method for online model learning for task-space tracking control
of redundant robots.
Inference of human intention may be an essential step towards understanding human actions [21] and is hence
important for realizing efficient human-robot interaction. In this paper, we propose the Intention-Driven Dynamics Model (IDDM), a latent variable model for inferring unknown human intentions. We train the model based on observed human behaviors/actions and we introduce an approximate inference algorithm to efficiently infer the human’s intention from an ongoing action.
We verify the feasibility of the IDDM in two scenarios, i.e., target inference in robot table tennis and action recognition for interactive humanoid robots. In both tasks, the IDDM achieves substantial improvements over state-of-the-art regression and classification.
Many motor skills in humanoid robotics can be learned using parametrized motor primitives. While successful applications to date have been achieved with imitation learning, most of the interesting motor learning problems are high-dimensional reinforcement learning problems. These problems are often beyond the reach of current reinforcement learning methods. In this paper, we study parametrized policy search methods and apply these to benchmark problems of motor primitive learning in robotics. We show that many well-known parametrized policy search methods can be derived from a general, common framework. This framework yields both policy gradient methods and expectation-maximization (EM) inspired algorithms. We introduce a novel EM-inspired algorithm for policy learning that is particularly well-suited for dynamical system motor primitives. We compare this algorithm, both in simulation and on a real robot, to several well-known parametrized policy search methods such as episodic REINFORCE, ‘Vanilla’ Policy Gradients with optimal baselines, episodic Natural Actor Critic, and episodic Reward-Weighted Regression. We show that the proposed method out-performs them on an empirical benchmark of learning dynamical system motor primitives both in simulation and on a real robot. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task on a real Barrett WAM™ robot arm.
Dynamic tactile sensing is a fundamental ability to recognize materials and objects. However, while humans are born with partially developed dynamic tactile sensing and quickly master this skill, today's robots remain in their infancy. The development of such a sense requires not only better sensors but the right algorithms to deal with these sensors' data as well. For example, when classifying a material based on touch, the data are noisy, high-dimensional, and contain irrelevant signals as well as essential ones. Few classification methods from machine learning can deal with such problems. In this paper, we propose an efficient approach to infer suitable lower dimensional representations of the tactile data. In order to classify materials based on only the sense of touch, these representations are autonomously discovered using visual information of the surfaces during training. However, accurately pairing vision and tactile samples in real-robot applications is a difficult problem. The proposed approach, therefore, works with weak pairings between the modalities. Experiments show that the resulting approach is very robust and yields significantly higher classification performance based on only dynamic tactile sensing.
Table tennis is a sufficiently complex motor task
for studying complete skill learning systems. It consists of several
elementary motions and requires fast movements, accurate
control, and online adaptation. To represent the elementary
movements needed for robot table tennis, we rely on dynamic
systems motor primitives (DMP). While such DMPs have been
successfully used for learning a variety of simple motor tasks,
they only represent single elementary actions. In order to select
and generalize among different striking movements, we present
a new approach, called Mixture of Motor Primitives that uses
a gating network to activate appropriate motor primitives. The
resulting policy enables us to select among the appropriate
motor primitives as well as to generalize between them. In
order to obtain a fully learned robot table tennis setup, we
also address the problem of predicting the necessary context
information, i.e., the hitting point in time and space where
we want to hit the ball. We show that the resulting setup
was capable of playing rudimentary table tennis using an
anthropomorphic robot arm.
Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasps location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controllers upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshapin
g the hand depending on the objects geometry. The system was evaluated both in simulation and on a real robot.
Grasping is one of the most important abilities needed for future service robots. Given the task of picking up
an object from betweem clutter, traditional robotics approaches would determine a suitable grasping point and
then use a movement planner to reach the goal. The planner would require precise and accurate information
about the environment and long computation times, both of which may not always be available. Therefore,
methods for executing grasps are required, which perform well with information gathered from only standard
stereo vision, and make only a few necessary assumptions about the task environment. We propose techniques
that reactively modify the robots learned motor primitives based on information derived from Early Cognitive
Vision descriptors. The proposed techniques employ non-parametric potential fields centered on the Early
Cognitive Vision descriptors to allow for curving hand trajectories around objects, and finger motions that
adapt to the objects local geometry. The methods were tested on a real robot and found to allow for easier
imitation learning of human movements and give a considerable improvement to the robots performance in
grasping tasks.
In this article, we present both novel learning algorithms and experiments using the dynamical system MPs. As such, we describe this MP representation in a way that it is straightforward to reproduce. We review an appropriate imitation learning method, i.e., locally weighted regression, and show how this method can be used both for initializing RL tasks as well as for modifying the start-up phase in a rhythmic task. We also show our current best-suited RL algorithm for this framework, i.e., PoWER. We present two complex motor tasks, i.e., ball-in-a-cup and ball paddling, learned on a real, physical Barrett WAM, using the methods presented in this article. Of particular interest is the ball-paddling application, as it requires a combination of both rhythmic and discrete dynamical systems MPs during the start-up phase to achieve a particular task.
Online model learning in real-time is required
by many applications such as in robot tracking
control. It poses a difficult problem, as
fast and incremental online regression with
large data sets is the essential component
which cannot be achieved by straightforward
usage of off-the-shelf machine learning methods
(such as Gaussian process regression or
support vector regression). In this paper,
we propose a framework for online, incremental
sparsification with a fixed budget designed
for large scale real-time model learning.
The proposed approach combines a
sparsification method based on an independence
measure with a large scale database.
In combination with an incremental learning
approach such as sequential support vector
regression, we obtain a regression method
which is applicable in real-time online learning.
It exhibits competitive learning accuracy
when compared with standard regression
techniques. Implementation on a real
robot emphasizes the applicability of the proposed
approach in real-time online model
learning for real world systems.
Empirical InferenceConference PaperMovement extraction by detecting dynamics switches and repetitionsChiappa, S., Peters, J.
In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, Advances in Neural Information Processing Systems 23, 388-396, (Editors: Lafferty, J. , C. K.I. Williams, J. Shawe-Taylor, R. S. Zemel, A. Culotta), Curran, Red Hook, NY, USA, Twenty-Fourth Annual Conference on Neural Information Processing Systems (NIPS 2010), 2010
Many time-series such as human movement data consist of a sequence of basic actions, e.g., forehands and backhands in tennis. Automatically extracting and characterizing such actions is an important problem for a variety of different applications. In this paper, we present a probabilistic segmentation approach in which an observed time-series is modeled as a concatenation of segments corresponding
to different basic actions. Each segment is generated through a noisy transformation of one of a few hidden trajectories representing different types of movement,
with possible time re-scaling. We analyze three different approximation methods for dealing with model intractability, and demonstrate how the proposed approach
can successfully segment table tennis movements recorded using a robot arm as haptic input device.
Empirical InferenceConference PaperSwitched Latent Force Models for Movement Segmentation
Alvarez, M., Peters, J., Schölkopf, B., Lawrence, N.
In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, Advances in neural information processing systems 23, 55-63, (Editors: J Lafferty and CKI Williams and J Shawe-Taylor and RS Zemel and A Culotta), Curran, Red Hook, NY, USA, 24th Annual Conference on Neural Information Processing Systems (NIPS 2010), 2010
Latent force models encode the interaction between multiple related dynamical systems in the form of a kernel or covariance function. Each variable to be modeled is represented as the output of a differential equation and each differential equation is driven by a weighted sum of latent functions with uncertainty given by a Gaussian process prior. In this paper we consider employing the latent force model framework for the problem of determining robot motor primitives. To deal with discontinuities in the dynamical systems or the latent driving force we introduce
an extension of the basic latent force model, that switches between different latent functions and potentially different dynamical systems. This creates a versatile
representation for robot movements that can capture discrete changes and non-linearities in the dynamics. We give illustrative examples on both synthetic data and for striking movements recorded using a BarrettWAM robot as haptic input device. Our inspiration is robot motor primitives, but we expect our model to have wide application for dynamical systems including models for human motion capture data and systems biology.
