References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: SFusion: Self-attention based N-to-One Multimodal Fusion Block
(Submitted on 26 Aug 2022 (v1), last revised 4 Jul 2023 (this version, v2))
Abstract: People perceive the world with different senses, such as sight, hearing, smell, and touch. Processing and fusing information from multiple modalities enables Artificial Intelligence to understand the world around us more easily. However, when there are missing modalities, the number of available modalities is different in diverse situations, which leads to an N-to-One fusion problem. To solve this problem, we propose a self-attention based fusion block called SFusion. Different from preset formulations or convolution based methods, the proposed block automatically learns to fuse available modalities without synthesizing or zero-padding missing ones. Specifically, the feature representations extracted from upstream processing model are projected as tokens and fed into self-attention module to generate latent multimodal correlations. Then, a modal attention mechanism is introduced to build a shared representation, which can be applied by the downstream decision model. The proposed SFusion can be easily integrated into existing multimodal analysis networks. In this work, we apply SFusion to different backbone networks for human activity recognition and brain tumor segmentation tasks. Extensive experimental results show that the SFusion block achieves better performance than the competing fusion strategies. Our code is available at this https URL
Submission history
From: Zecheng Liu [view email][v1] Fri, 26 Aug 2022 16:42:14 GMT (4429kb,D)
[v2] Tue, 4 Jul 2023 14:50:31 GMT (3299kb,D)
Link back to: arXiv, form interface, contact.