WildSmoke: Ready-to-Use Dynamic 3D Smoke Assets from a Single Video in the Wild

Yuqiu Liu , Jialin Song , Manolis Savva , Wuyang Chen
School of Computing Science, Simon Fraser University
arXiv Code coming soon Checkpoints Smoke Sequence
pipeline

In this work, we propose WildSmoke to reconstruct an editable, physics-consistent 4D smoke asset from a single in-the-wild video, enabling novel view synthesis, future prediction and downstream simulation.

Abstract

We propose a pipeline to extract and reconstruct dynamic 3D smoke assets from a single video in the wild, and further integrate interactive simulation for smoke design and editing. Recent developments in 3D vision have enabled significant progress in reconstructing and rendering fluid dynamics, enabling realistic and temporally consistent view synthesis. However, current fluid reconstructions highly depend on carefully controlled clean lab environments, whereas real-world videos captured in the wild are largely underexplored. We pinpoint three key challenges of reconstructing smoke in real-world videos and design targeted techniques, including smoke extraction with background removal, initializations of smoke particles with pose estimation, and inferring multi-view videos. We not only outperform previous reconstruction or generation methods with high-quality smoke reconstructions (+2.22 average PSNR on wild videos), but also enable diverse and realistic editing of fluid dynamics by simulating our smoke assets.

Reconstructed Wild Smoke

Simulation

BibTeX


@InProceedings{WildSmoke,
    author    = {Yuqiu, Liu and Jialin, Song and Manolis, Savva and Wuyang, Chen},
    title     = {WildSmoke: 4D Reconstruction of Dynamic Smoke from a Single Video},
    booktitle = {...},
    month     = {...},
    year      = {...},
    pages     = {...}
}