References & Citations
Computer Science > Artificial Intelligence
Title: Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?
(Submitted on 29 Oct 2023 (v1), last revised 26 Apr 2024 (this version, v2))
Abstract: Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.
Submission history
From: Kyung Geun Kim [view email][v1] Sun, 29 Oct 2023 08:00:13 GMT (420kb,D)
[v2] Fri, 26 Apr 2024 08:11:21 GMT (634kb,D)
Link back to: arXiv, form interface, contact.