Research

The goal of this project is to bring efficient learning-based control methods to real robots. A promising direction to learning from a few trials is to separate the task. First, train a model of the interactions with the environment and then use model-based reinforcement learning or model-based planning to control the robot. In a first work, we have explored the use of our equation learning framework for the forward model and obtained unparalleled performance on a cart-pendulum system. However, we found that it does not scale yet to relevant robotic systems.

A widely used method in model-based reinforcement learning is the Cross-Entropy Method (CEM), which is a zero-order optimization scheme to compute good action sequences. We have improved this method by replacing the uncorrelated sampling of actions with a temporally-correlated sampling (colored noise). As a result, we were able to reduce the computational cost of CEM by a factor of 3-22 while yielding a performance increase of up to 10 times in a variety of challenging robotic tasks.

Even with our improvements, it is still challenging to run such a model-based planning method in real-time on a robot with a high update frequency. Thus we set out to extract neural network policies from the data generated by a model-based planning algorithm. A challenging task, as it turns out, since naive policy learning methods fail to learn from such a stochastic planning algorithm. We have proposed an adaptive guided policy search method that is able to distill strong policies for challenging simulated robotics tasks.

The next step in our aim to run these algorithms on a real robot is to make them risk-aware. We extend the learned dynamics models with the ability to estimate their prediction uncertainty. In fact, the models distinguish between uncertainty due to lack of data and inherent unpredictability (noise). We demonstrate on several continuous control tasks how to obtain active learning and risk-averse planning to avoid dangerous situations. This is an important step towards safe reinforcement learning on real hardware.

In several research projects, we investigate data-driven approaches to the control of robotic systems. By combining optimization and reinforcement learning, intrinsic motivation, and hierarchical planning, we are tackling various challenges arising in robotic systems.

Similar to how children perform curious free play, we want to develop reinforcement learning agents that can efficiently and autonomously explore their environment without relying on manually designed tasks and extrinsic rewards.

The goal of intrinsically-motivated reinforcement learning (RL) is for agents to explore environments efficiently, similar to the curious play of children. We study intrinsic motivation to maximize future task capabilities. Initially, we focus on learning progress and surprise as motivators. We also explore the causal influence of a robot as a driver for exploration. We investigate intrinsic reward signals, inductive biases, and learning frameworks to optimize exploration. Recently, we consider predicted information gain as an internal drive to improve exploration efficiency and to learn capable world models. We find that structured world models allow for relational inductive biases that are suitable for tasks with many objects. Combined with our planning method, we obtain a task-agnostic free-play behavior that explores what can be done in the environment. Using the online-planning method, a robotic arm can then perform tasks such as block stacking without further training, so we achieved zero-shot task generalization.

Inspired by child development, we postulate that the search for structure partially guides exploration. We propose Regularity as Intrinsic Reward (RaIR) in model-based RL. RaIR is a measure of regularity of a scene and is computed using Shannon Entropy of a suitable scene representation. We find that RaIR and epistemic uncertainty promote autonomous construction of structures during free play which also leads to improved zero-shot task performance in construction tasks.

We further investigate how meaningful behaviors can emerge. Foundation models are a great source of high-level knowledge. We propose SEmaNtically Sensible ExploratIon (SENSEI), a framework to equip model-based RL agents with intrinsic motivation from vision language models for semantically meaningful behavior. We observe more directed exploration towards “reasonable” behavior during free-play that translates into better world-models and improved downstream performance.

Learning diverse and useful skills is one of the major challenges in robotics. Imitation learning from state-only demonstrations is a promising approach to capture preferences and to easily specify tasks. We first study how to obtain agile behavior from only partial demonstrations. We develop a Wasserstein discrimination method to compute the reward. This paper was one of the three best paper finalists at CoRL 2022.

To obtain diverse behaviors online, we developed an algorithm that learns versatile and controllable skills based on mutual information and information gain as diversity objective. Similar to our earlier work, it combines a GAIL-like imitation reward with task-specific rewards. The subsequent work enhances diversity using a physics-inspired Van der Waals (VdW) force objective and encodes utilities as multiple constraints.

We then studied the offline setting, i.e., when only data from previous interactions are available and no further environmental interaction is possible. We propose the first principled offline method for maximizing skill diversity under imitation constraints. The method requires access to an explorative dataset of the environment and a few demonstrations of solving a task. Here, we focus on the challenging and practically relevant setting where the demonstrations are provided only in terms of visited states, i.e., without actions. To generate diversity, based on an information-theoretic objective, the method trains a skill discriminator. The follow-up work eliminates the need to learn a skill discriminator by providing an offline estimator of the VdW distance. We also use Functional Reward Encoding to better handle non-stationary rewards, allowing for stable, efficient training and zero-shot recall of all skills encountered during training.

Biological systems rely on muscles for actuation. Do muscles offer advantages over electric motors when viewed through the lens of learning and closed-loop control of behavior? We explored this question and found that muscles can enhance learning speed and robustness. This research highlighted the significant potential of leveraging muscle actuator morphology for natural and robust motion, so far only validated in simulation.

How can these findings be applied to real robots? Currently, physical muscles are not mature or easy to integrate into robotic systems. In this study, we simulated muscle actuator properties in real time using affordable modern electric motors, allowing us to emulate a simplified muscle model on a real robot controlled by a learned policy. Our results suggest that artificial muscles could serve as highly advantageous actuators for future generations of robust legged robots. We continue to investigate which muscle properties are most critical for improving learning speed and behavioral robustness.

Understanding muscle control is critical not only for robotics, but also for advancing our understanding of human motor control. This has promising implications for the development of assistive devices and improved therapies for patients. Learning to control systems with the large number of degrees of freedom present in biological systems has been based primarily on imitation learning, which involves mimicking recorded biological data.

Developing exploration methods and reinforcement learning (RL) algorithms capable of scaling to simulated systems with many muscles was a major challenge we addressed. Our method was first applied to an ostrich and a simple humanoid model. The effort paid off, as we can now achieve human-like walking in several high-dimensional muscle-driven models using only generic reward terms. This paves the way for the analysis of muscle patterns in patients with deficits and for the joint optimization of assistive devices such as exoskeletons.

Deep learning is the tool for our research to obtain learned representations, fit functions such as policies or value functions and learn internal models. Along the way of using deep learning techniques for our core focus of autonomous learning we frequently need to develop new methods. Quite often, we stumble upon unsolved or puzzling problems in the techniques themselves, some of which we are solving, see our projects below. In general, we are particularly interested in techniques for representation learning, internal model learning and continual learning.

Deep learning has shown remarkable success in solving problems from high-dimensional raw input, such as image recognition or speech-to-text translation. However, deep learning is not particularly good when it comes to problems with a combinatorial nature, such as solving a puzzle, finding the shortest path or matching two sets of items. For each of these problems, computer scientists have developed highly optimized algorithms with correspondingly fast implementations in the field of combinatorial optimization. These methods require a perfect input representation and cannot be used on raw inputs as such.

In this work, we bring deep learning and combinatorial optimization together. Ideally, we would like to have the best of both worlds: having rich feature representations through deep neural networks and efficient algorithm implementations that enable combinatorial generalization. We have developed a method with which such algorithms can indeed be used as building blocks in deep neural networks. The original implementations do not need to be known or modified. Instead, it is enough to invoke them one time during the gradient computation on a modified input to obtain an informative gradient.

Equipped with our new method, called Blackbox differentiation, we tackled two real-world problems in computer vision: the keypoint correspondence problem and optimizing rank-based functions. The keypoint correspondence problem is about finding matching pairs of points in two or more images. Our architecture, embedding a strong graph-matching solver, is able to outperform the state of the art in this domain on several benchmarks. Another remarkable success of our method is that it allows computing gradients for ranking-based functions – commonly used in, e.g., object-retrieval. Our paper was nominated for the best-paper award at CVPR.

Enhancing reinforcement learning agents with planning algorithms as part of their policy networks is another exciting application.

Recently, we conceptually enhanced our method with the ability to learn the type of combinatorial algorithm that should be solved. This is achieved by learning the constraints of an integer linear program.

The rapid evolution of robotic technologies informs practical benefits in various physical application areas. In complex, changing, and especially human-involved scenarios, a robot must be well-equipped to perceive the interactions between its own body and other things. Due to the visual occlusion and the small scale of the deformations during interactions, robots need touch-sensitive skin in addition to well-developed vision feedback. In this research field, we explore creating durable and robust haptic sensors to meet various application requirements and design machine-learning algorithms to enhance the data processing flow.

In various collaborations, we use machine learning methods in other scientific domains to advance our understanding and gain new insights. One of our own machine learning methods, called the equation learner, is particularly suitable for applications in the sciences, as it attempts to find very concise analytical equations for data.

Projects that we have worked on in the past.