References & Citations
Computer Science > Computation and Language
Title: Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU
(Submitted on 11 Oct 2020 (v1), last revised 28 Mar 2024 (this version, v2))
Abstract: While humans process language incrementally, the best language encoders currently used in NLP do not. Both bidirectional LSTMs and Transformers assume that the sequence that is to be encoded is available in full, to be processed either forwards and backwards (BiLSTMs) or as a whole (Transformers). We investigate how they behave under incremental interfaces, when partial output must be provided based on partial input seen up to a certain time step, which may happen in interactive systems. We test five models on various NLU datasets and compare their performance using three incremental evaluation metrics. The results support the possibility of using bidirectional encoders in incremental mode while retaining most of their non-incremental quality. The "omni-directional" BERT model, which achieves better non-incremental performance, is impacted more by the incremental access. This can be alleviated by adapting the training regime (truncated training), or the testing procedure, by delaying the output until some right context is available or by incorporating hypothetical right contexts generated by a language model like GPT-2.
Submission history
From: Brielen Madureira [view email][v1] Sun, 11 Oct 2020 19:51:21 GMT (5548kb,D)
[v2] Thu, 28 Mar 2024 11:26:58 GMT (3805kb,D)
Link back to: arXiv, form interface, contact.