References & Citations
Computer Science > Machine Learning
Title: Remembering Transformer for Continual Learning
(Submitted on 11 Apr 2024 (this version), latest version 16 May 2024 (v3))
Abstract: Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task knowledge interferes with previously learned knowledge. We propose Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS), to tackle this issue. Remembering Transformer employs a mixture-of-adapters and a generative model-based routing mechanism to alleviate CF by dynamically routing task data to relevant adapters. Our approach demonstrated a new SOTA performance in various vision continual learning tasks and great parameter efficiency.
Submission history
From: Yuwei Sun [view email][v1] Thu, 11 Apr 2024 07:22:14 GMT (4834kb,D)
[v2] Tue, 23 Apr 2024 08:02:23 GMT (5074kb,D)
[v3] Thu, 16 May 2024 00:12:11 GMT (1627kb,D)
Link back to: arXiv, form interface, contact.