References & Citations
Mathematics > Optimization and Control
Title: Variational Dynamic Programming for Stochastic Optimal Control
(Submitted on 23 Apr 2024 (v1), last revised 26 Apr 2024 (this version, v2))
Abstract: We consider the problem of stochastic optimal control where the state-feedback control policies take the form of a probability distribution, and where a penalty on the entropy is added. By viewing the cost function as a Kullback-Leibler (KL) divergence between two Markov chains, we bring the tools from variational inference to bear on our optimal control problem. This allows for deriving a dynamic programming principle, where the value function is defined as a KL divergence again. We then resort to Gaussian distributions to approximate the control policies, and apply the theory to control affine nonlinear systems with quadratic costs. This results in closed-form recursive updates, which generalize LQR control and the backward Riccati equation. We illustrate this novel method on the simple problem of stabilizing an inverted pendulum.
Submission history
From: Marc Lambert [view email][v1] Tue, 23 Apr 2024 07:37:41 GMT (224kb,D)
[v2] Fri, 26 Apr 2024 06:47:28 GMT (224kb,D)
Link back to: arXiv, form interface, contact.