BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting

arXiv 2024

Lingzhe Zhao1* Peng Wang1,2* Peidong Liu1
1Westlake University 2Zhejiang University
* denotes equal contribution.
arXiv Code

Given a set of severe motion blurred images, BAD-Gaussians jointly learns the scene representation with 3D Gaussians and recovers the camera motion trajectories within exposure time. It achieves state-of-the-art performance on both synthetic and real datasets in deblurring and novel-view synthesis tasks, with faster training, realtime rendering and lower GPU memory consumption.

Abstract

While neural rendering has demonstrated impressive capabilities in 3D scene reconstruction and novel view synthesis, it heavily relies on high-quality sharp images and accurate camera poses. Numerous approaches have been proposed to train Neural Radiance Fields (NeRF) with motion-blurred images, commonly encountered in real-world scenarios such as low-light or long-exposure conditions. However, the implicit representation of NeRF struggles to accurately recover intricate details from severely motion-blurred images and cannot achieve real-time rendering. In contrast, recent advancements in 3D Gaussian Splatting achieve high-quality 3D scene reconstruction and real-time rendering by explicitly optimizing point clouds as Gaussian spheres.

In this paper, we introduce a novel approach, named BAD-Gaussians (Bundle Adjusted Deblur Gaussian Splatting), which leverages explicit Gaussian representation and handles severe motion-blurred images with inaccurate camera poses to achieve high-quality scene reconstruction. Our method models the physical image formation process of motion-blurred images and jointly learns the parameters of Gaussians while recovering camera motion trajectories during exposure time.

In our experiments, we demonstrate that BAD-Gaussians not only achieves superior rendering quality compared to previous state-of-the-art deblur neural rendering methods on both synthetic and real datasets but also enables real-time rendering capabilities.

Keywords

3D Gaussian Splatting, Deblurring, Bundle Adjustment, Differentiable Rendering.

Pipeline

overview

Motion Blur Image Formation Model

The mathematical modeling of motion blur process involves integrating over a set of virtual sharp images, as shown by the green line in the figure.

Camera Motion Trajectory Modeling

We model the camera motion during the exposure time with a coutinuous trajectory in SE(3) space, as shown by the orange line in the figure. The joint optimization of Gaussians and camera trajectories is achieved by minimizing the photometric loss between synthesized and actual blurry images.

Camera Motion Trajectory Modeling

overview

Cubic B-Spline Formulation

Compared to a linear SE(3) interpolation, a more complex camera trajectory within exposure time can be controlled by four control knots in SE(3) space, denoted as T0, T1, T2 and T3. This cubic B-spline formulation is differentiable and can be optimized jointly with the 3D Gaussians. The camera motion trajectory is then recovered by the optimized control knots. This formulation is more flexible and can better capture the complex camera motion during exposure time on real-world datasets.

Results

The experimental results demonstarte that our method can effectively deblur images, render novel view images and recover the camera motion trajectories accurately within exposure time.

Novel View Synthesis

To showcase the effectiveness of our BAD-Gaussians, we provide videos illustrating the capability of BAD-Gaussians to recover high-quality latent sharp video from blurry images of Deblur-NeRF's synthetic and real datasets, as well as from our challenging real-captured demo dataset. In the videos below, on the left are our rendered novel view images and on the right are the input blurry images.

Notably, in Deblur-NeRF's real word datasets, thanks to the fast training speed and low GPU memory requirements of our method, we are able to train the real scenes at the full resolution of 2400×1600 to achieve maximum reconstruction quality.

BibTeX

@article{zhao2024badgaussians,
      title={{BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting}},
      author={Zhao, Lingzhe and Wang, Peng and Liu, Peidong},
      year={2024},
      eprint={2403.11831},
      archivePrefix={arXiv},
  }