On the Global Convergence and Approximation Benefits of Policy Gradient Methods, Daniel Russo; iDS2 Seminar Series
From Oluwasanmi Koyejo
views
comments
From Oluwasanmi Koyejo
Abstract: Policy gradient methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, due to the multi-period nature of the objective, they face non-convex optimization problems. The first part of the talk aims to clarify when policy gradient methods can be expected to converge to a global optimum despite this non-convexity. We show they can get stuck in suboptimal stationary points in extremely simple problems but identify structural properties – shared by several canonical control problems – that guarantee the policy gradient objective function has no suboptimal stationary points despite being non-convex. In the second part of the talk, I’ll zoom in on the special case of state aggregated policies, which induce unavoidable approximation errors as they cannot represent an optimal policy. In this case, I'll show policy gradient provably converges to better policies than popular approximate policy iteration and value iteration methods.