Diffusion-Based Extreme High-speed Scenes Reconstruction with the Complementary Vision Sensor

Recording and reconstructing high-speed scenes poses a significant challenge. The high bandwidth of high-speed cameras makes continuous recording unsustainable, while the frame interpolation methods using traditional RGB cameras (typically 30 fps) introduce artifacts and are affected by motion blur. Leveraging sensors inspired by the human visual system, such as event cameras, provides high-speed parse temporal variation or spatial variation data to alleviate the ill-conditioned problem of high-speed reconstructing with traditional RGB cameras. However, existing methods still suffer from RGB blur, temporal aliasing, and loss of event information. To overcome the above challenges, we leverage a novel dual-pathway complementary vision sensor, which outputs high-speed, sparse spatio-temporal differential frames between two RGB frames as reconstruction conditions. Further, we propose a cascaded bi-directional recurrent diffusion model (CBRDM) that can achieve accurate, sharp, color-rich video frames reconstruction results. Our method improves the LPIPS metric by 37.6% over state-of-the-art RGB interpolation algorithms and achieves superior performance in real-world comparisons with event cameras. Our code and dataset will be publicly available.