References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Segmenting the motion components of a video: A long-term unsupervised model
(Submitted on 2 Oct 2023 (v1), last revised 17 Apr 2024 (this version, v3))
Abstract: Human beings have the ability to continuously analyze a video and immediately extract the motion components. We want to adopt this paradigm to provide a coherent and stable motion segmentation over the video sequence. In this perspective, we propose a novel long-term spatio-temporal model operating in a totally unsupervised way. It takes as input the volume of consecutive optical flow (OF) fields, and delivers a volume of segments of coherent motion over the video. More specifically, we have designed a transformer-based network, where we leverage a mathematically well-founded framework, the Evidence Lower Bound (ELBO), to derive the loss function. The loss function combines a flow reconstruction term involving spatio-temporal parametric motion models combining, in a novel way, polynomial (quadratic) motion models for the spatial dimensions and B-splines for the time dimension of the video sequence, and a regularization term enforcing temporal consistency on the segments. We report experiments on four VOS benchmarks, demonstrating competitive quantitative results, while performing motion segmentation on a whole sequence in one go. We also highlight through visual results the key contributions on temporal consistency brought by our method.
Submission history
From: Etienne Meunier [view email][v1] Mon, 2 Oct 2023 09:33:54 GMT (39996kb,D)
[v2] Sun, 28 Jan 2024 01:15:50 GMT (39984kb,D)
[v3] Wed, 17 Apr 2024 17:44:24 GMT (32395kb,D)
Link back to: arXiv, form interface, contact.