Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Real-Time Recurrent Reinforcement Learning
(Submitted on 8 Nov 2023 (v1), last revised 28 Mar 2024 (this version, v2))
Abstract: In this paper we propose real-time recurrent reinforcement learning (RTRRL), a biologically plausible approach to solving discrete and continuous control tasks in partially-observable markov decision processes (POMDPs). RTRRL consists of three parts: (1) a Meta-RL RNN architecture, implementing on its own an actor-critic algorithm; (2) an outer reinforcement learning algorithm, exploiting temporal difference learning and dutch eligibility traces to train the Meta-RL network; and (3) random-feedback local-online (RFLO) learning, an online automatic differentiation algorithm for computing the gradients with respect to parameters of the network.Our experimental results show that by replacing the optimization algorithm in RTRRL with the biologically implausible back propagation through time (BPTT), or real-time recurrent learning (RTRL), one does not improve returns, while matching the computational complexity for BPTT, and even increasing complexity for RTRL. RTRRL thus serves as a model of learning in biological neural networks, mimicking reward pathways in the basal ganglia.
Submission history
From: Julian Lemmel [view email][v1] Wed, 8 Nov 2023 16:56:16 GMT (3934kb,D)
[v2] Thu, 28 Mar 2024 10:30:57 GMT (4065kb,D)
Link back to: arXiv, form interface, contact.