Special Seminar - Kishor Jothimurugan, "Specification-Guided Reinforcement Learning"

Abstract:
Recent advances in Reinforcement Learning (RL) have enabled data-driven controller design for autonomous systems such as robotic arms and self-driving cars. Applying RL to such a system typically involves encoding the objective using a reward function (mapping transitions of the system to real values) and then training a neural network controller (from simulations of the system) to maximize the expected reward. However, many challenges arise when one tries to train controllers to perform complex long-horizon tasks---e.g., navigating a car along a complex track with multiple turns. Firstly, it can be quite challenging to manually define well-shaped reward functions for such tasks. It is much more natural to use a high-level specification language such as Linear Temporal Logic (LTL) to specify these tasks. Secondly, existing algorithms for learning controllers from logical specifications do not scale well to complex tasks due to a number of reasons including the use of sparse rewards and lack of compositionality.

In this talk, I'll present my work on RL from temporal specifications that attempts to tackle these challenges. First, I'll present a compositional RL algorithm that can be used to train policies to perform complex tasks by leveraging the structure in the given specification. Then, I'll discuss a theoretical hardness result regarding learning from temporal specifications. Finally, I'll show that compositional approaches are also useful in verifying safety of neural network controllers in closed-loop systems.

Bio:
Kishor Jothimurugan is a final-year PhD student at the University of Pennsylvania, advised by Prof. Rajeev Alur. His research focuses on incorporating formal reasoning in ML tools and techniques to enable building of reliable, interpretable, and intelligent systems. In particular, he has worked on reinforcement learning (RL) from temporal specifications, compositional RL and verification of neural network controllers. He received his Bachelor's with Honors in 2017 from Chennai Mathematical Institute.