Current browse context:
cs
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Deep Reinforcement Learning: A Convex Optimization Approach
(Submitted on 29 Feb 2024 (v1), last revised 28 Mar 2024 (this version, v4))
Abstract: In this paper, we consider reinforcement learning of nonlinear systems with continuous state and action spaces. We present an episodic learning algorithm, where we for each episode use convex optimization to find a two-layer neural network approximation of the optimal $Q$-function. The convex optimization approach guarantees that the weights calculated at each episode are optimal, with respect to the given sampled states and actions of the current episode. For stable nonlinear systems, we show that the algorithm converges and that the converging parameters of the trained neural network can be made arbitrarily close to the optimal neural network parameters. In particular, if the regularization parameter is $\rho$ and the time horizon is $T$, then the parameters of the trained neural network converge to $w$, where the distance between $w$ from the optimal parameters $w^\star$ is bounded by $\mathcal{O}(\rho T^{-1})$. That is, when the number of episodes goes to infinity, there exists a constant $C$ such that \[\|w-w^\star\| \le C\cdot\frac{\rho}{T}.\] In particular, our algorithm converges arbitrarily close to the optimal neural network parameters as the time horizon increases or as the regularization parameter decreases.
Submission history
From: Ather Gattami [view email][v1] Thu, 29 Feb 2024 14:41:31 GMT (38kb)
[v2] Thu, 7 Mar 2024 07:09:11 GMT (38kb)
[v3] Wed, 20 Mar 2024 16:34:27 GMT (36kb)
[v4] Thu, 28 Mar 2024 16:59:03 GMT (37kb)
Link back to: arXiv, form interface, contact.