4D Gaussian Splats

Reconstructing a moving scene as per-frame 3D Gaussian splats works, but it stores the same geometry over and over and the result is enormous. I wrote a differentiable renderer in CUDA that trains splats with custom attributes. Each one carries a lifetime and a polynomial motion path, so a single splat can cover a stretch of time instead of one frame. Motion is learned rather than re-fit, and the trained models come out about 5× smaller than the per-frame equivalent. I also wrote a WebGPU version of the renderer for in-browser viewing: 60fps for >2 million gaussians.

Highlights

• Differentiable CUDA rasterizer with custom per-splat attributes and hand-written backward passes, exposed to PyTorch as a training op.
• 5× smaller trained models than per-frame 3DGS at comparable quality. Leverages time-constrained splats with polynomial motion.
• Real-time WebGPU renderer for 60fps rendering in-browser for over 2 million Gaussians.