Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Reasoning over the Behaviour of Objects in Video-Clips for Adverb-Type Recognition
(Submitted on 9 Jul 2023 (v1), last revised 27 Mar 2024 (this version, v3))
Abstract: In this work, following the intuition that adverbs describing scene-sequences are best identified by reasoning over high-level concepts of object-behavior, we propose the design of a new framework that reasons over object-behaviours extracted from raw-video-clips to recognize the clip's corresponding adverb-types. Importantly, while previous works for general scene adverb-recognition assume knowledge of the clips underlying action-types, our method is directly applicable in the more general problem setting where the action-type of a video-clip is unknown. Specifically, we propose a novel pipeline that extracts human-interpretable object-behaviour-facts from raw video clips and propose novel symbolic and transformer based reasoning methods that operate over these extracted facts to identify adverb-types. Experiment results demonstrate that our proposed methods perform favourably against the previous state-of-the-art. Additionally, to support efforts in symbolic video-processing, we release two new datasets of object-behaviour-facts extracted from raw video clips - the MSR-VTT-ASP and ActivityNet-ASP datasets.
Submission history
From: Amrit Diggavi Seshadri [view email][v1] Sun, 9 Jul 2023 09:04:26 GMT (3190kb,D)
[v2] Wed, 12 Jul 2023 10:57:00 GMT (3190kb,D)
[v3] Wed, 27 Mar 2024 18:17:46 GMT (2399kb,D)
Link back to: arXiv, form interface, contact.