Current browse context:
eess.AS
Change to browse by:
References & Citations
Computer Science > Multimedia
Title: Robust Active Speaker Detection in Noisy Environments
(Submitted on 27 Mar 2024 (v1), last revised 30 Mar 2024 (this version, v2))
Abstract: This paper addresses the issue of active speaker detection (ASD) in noisy environments and formulates a robust active speaker detection (rASD) problem. Existing ASD approaches leverage both audio and visual modalities, but non-speech sounds in the surrounding environment can negatively impact performance. To overcome this, we propose a novel framework that utilizes audio-visual speech separation as guidance to learn noise-free audio features. These features are then utilized in an ASD model, and both tasks are jointly optimized in an end-to-end framework. Our proposed framework mitigates residual noise and audio quality reduction issues that can occur in a naive cascaded two-stage framework that directly uses separated speech for ASD, and enables the two tasks to be optimized simultaneously. To further enhance the robustness of the audio features and handle inherent speech noises, we propose a dynamic weighted loss approach to train the speech separator. We also collected a real-world noise audio dataset to facilitate investigations. Experiments demonstrate that non-speech audio noises significantly impact ASD models, and our proposed approach improves ASD performance in noisy environments. The framework is general and can be applied to different ASD approaches to improve their robustness. Our code, models, and data will be released.
Submission history
From: Siva Sai Nagender Vasireddy [view email][v1] Wed, 27 Mar 2024 20:52:30 GMT (3561kb,D)
[v2] Sat, 30 Mar 2024 14:00:27 GMT (3565kb,D)
Link back to: arXiv, form interface, contact.