Overview of our occlusion-free 4D Gaussian generation. Our major contributions lie in the two occlusion removal modules (a, b). The outcome is a 3D video without occlusions (e.g., the head in this figure) and free viewpoint control.
Video recording of open surgery is in great demand for education and research purposes but is challenging due to the busy and dynamic environment. The state-of-the-art system uses multi-view cameras installed in shadowless lamps (McSL) and implements an automatic camera switching algorithm to avoid disturbances. However, this algorithm leads to missing pixels and distorted projection due to mathematical image warping and does not always provide the best perspective. We propose using 4D Gaussian Splatting (4DGS) to create editable 3D videos and remove Gaussians occluding surgical fields from a perspective. We enable occlusion-free 3D videos by addressing two occlusion removal approaches via (1) occlusion masking and (2) density-based Gaussian filtering. We create a real-surgery dataset and demonstrate that our method outperforms the state-of-the-art auto view-switching approach.
We created a new surgery dataset with occlusion and ground-truth pairs. Surgeries include seven scenes: polysyndactyly, external nasal deformity, intramuscular lipoma of the gluteus maximus, Frontal bone fracture, preauricular sinus, scalp scar revision and hair transplantation, and keloid on the anterior chest. We extract (a) occlusion regions and overlay them to (b) frames without occlusions in the same scene to create (c) virtual input frames. (b) is the ground truth of (c).
Qualitative comparisons between the state-of-the-art view-switching method Switch [11] and our approaches on two surgical cases. The input videos contain synthetic occlusions, such as the surgeon's head, superimposed on unobstructed surgical scenes. While Switch partially restores visibility, it introduces warped views and leaves missing regions. In contrast, Ours:Mask reduces major occlusions through segmentation-based masking, though boundaries can remain unclear, and Ours:Dist removes occluder clusters but may inadvertently eliminate relevant details. Our combined method, Ours:Mask+Dist, yields the cleanest and most stable results, closely resembling the ground truth.
Videos used in the expert review. Five medical doctors were asked to provide comments on the strengths and weaknesses of each method after watching the corresponding videos. The experts noted that Switch provided consistent viewpoints but often suffered from blurs, warped regions, and temporary loss of points of interest. In contrast, Ours enabled intuitive free-viewpoint changes and reduced occlusions, though some lack of clarity and occasional missing instruments were reported. Among our methods, Mask+Dist was preferred for its cleaner and more stable visual results, with one surgeon highlighting its high educational value in reproducing videos from the surgeon’s perspective.
@inproceedings{kato2025occlusion,
title = {Occlusion-free 4D Gaussians for Open Surgery Videos Using Multi-Camera Shadowless Lamps},
author = {Kato, Yuna and Mori, Shohei and Saito, Hideo and Takatsume, Yoshifumi and Kajita, Hiroki and Isogawa, Mariko},
booktitle = {International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)},
year = {2025},
}