Gaussian splatting is a method of synthesizing scenes captured from a set of photos or videos. In simple terms, 3D gaussian splats allow you to create a 3D scene from 2D photos or video frames, resulting in a renderable 3D representation of the object from any angle.
3D Gaussians are a generalization of 1D Gaussians; they are ellipsoids in 3D space with a center, scale, rotation, and softened edges. They are basically a radiance field without the slow neural rendering part and allow for 3D editing. Their advantages include faster training (with equal quality), quicker rendering, and easier understanding and post-processing.
How does it work?
Capture: Photograph the scene from various different viewpoints (angles).
Reconstruct: Use structure for motion (COLMAP, RealityScan) to capture the sparse point cloud from posed imagery. This point cloud serves as a foundation for a 3d Gaussian splat centered at each of these points.
Optimize: Use gradient descent to optimize Gaussian parameters such as position, size, and orientation to match the Gaussian images to the original images.
Adaptive Density Control: It’s difficult to have an even distribution of Gaussians to model the complexities of reality. This is solved by adjusting the density of the Gaussians according to the required detail in the scenery. For example - there are more Gaussians for the trees, while the skies will have fewer.
Represent Color: The view-dependent lighting is modeled by spherical harmonics. First, you optimize the base color and then gradually add frequency bands.
Finally, project the 3D Gaussians into the image plane and render.
Applications
Car insurance claims would be faster and more accurate with a reduced possibility of faking car accidents -
https://x.com/michalgula/status/1729806941461913923