References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation
(Submitted on 6 Dec 2023 (v1), last revised 28 Mar 2024 (this version, v3))
Abstract: Advances in 3D generation have facilitated sequential 3D model generation (a.k.a 4D generation), yet its application for animatable objects with large motion remains scarce. Our work proposes AnimatableDreamer, a text-to-4D generation framework capable of generating diverse categories of non-rigid objects on skeletons extracted from a monocular video. At its core, AnimatableDreamer is equipped with our novel optimization design dubbed Canonical Score Distillation (CSD), which lifts 2D diffusion for temporal consistent 4D generation. CSD, designed from a score gradient perspective, generates a canonical model with warp-robustness across different articulations. Notably, it also enhances the authenticity of bones and skinning by integrating inductive priors from a diffusion model. Furthermore, with multi-view distillation, CSD infers invisible regions, thereby improving the fidelity of monocular non-rigid reconstruction. Extensive experiments demonstrate the capability of our method in generating high-flexibility text-guided 3D models from the monocular video, while also showing improved reconstruction performance over existing non-rigid reconstruction methods.
Submission history
From: Xinzhou Wang [view email][v1] Wed, 6 Dec 2023 14:13:54 GMT (23273kb,D)
[v2] Wed, 20 Dec 2023 07:52:24 GMT (23273kb,D)
[v3] Thu, 28 Mar 2024 09:40:08 GMT (29743kb,D)
Link back to: arXiv, form interface, contact.