The Hybrid Relightable 3D Gaussian Rendering System is a real-time novel view synthesis tool that leverages advanced machine learning techniques, 3D geometry processing, and optimized rendering methods to create high-quality, interactive 3D scenes. The system is designed to work with image-based data (such as video or image sequences) and polygonal models, seamlessly integrating them into a unified rendering pipeline.

The process begins by capturing a video input and extracting frames to splice into a sequence of images, which are then processed using Structure from Motion (SfM) techniques. This process generates point clouds—collections of 3D points representing the scene's structure. From these point clouds, the system creates Gaussian splats, simplified representations of the scene’s geometry, with each point in the cloud representing a Gaussian distribution in 3D space.

Next, the generated Gaussian points are fed into a PyTorch-based optimizer. This optimizer utilizes machine learning techniques to refine the positions and properties of the points. Additionally, it creates extra attributes crucial for accurate rendering and details. This optimization ensures the final 3D scene is visually realistic and computationally efficient.

The output from the optimizer and its model is then passed through a parser that prepares the data for rendering. This includes creating a Bounding Volume Hierarchy (BVH), a hierarchical structure that accelerates the ray tracing process by organizing the 3D objects in the scene for more efficient intersection tests. For scenes requiring more detailed geometry, users can integrate polygonal meshes (triangle meshes), which are also processed into their own BVH structures, enabling them to coexist and be rendered alongside the Gaussian splats.

The core of the system’s rendering capability lies in its ray tracer. This component takes the BVH data (whether Gaussian or triangle mesh-based). It combines it with the Unity camera to calculate how light interacts within the scene, ultimately producing realistic images. At path intersection points, each hit is inserted into a sorted list of intersections. Post insertion, we loop through the sorted hits list, keeping track of the paths' transmittance as it passes through translucent materials. Once the path’s transmittance value reaches a minimum value, we cull the remaining list and consider that the point of intersection. The ray tracing process accounts for various optical effects, such as shadows, reflections, and global illumination, ensuring the final output is both physically accurate and visually compelling.

Creating realistic 3D models of real-world objects using 3D modeling software presents significant challenges, including accurately replicating materials and topology. This process often requires tens to hundreds of hours of work from highly skilled graphic designers. Our system will enable users to generate realistic 3D Gaussian models directly from a video, seamlessly integrated into the Unity Game Engine.

The technique of generating images of a scene from any perspective using only a video or a series of photographs is known as novel view synthesis. Previously, this was done using Neural Radiance Fields (NeRFs). This method has costly training and rendering times, can produce subpar results, and cannot be combined with traditional polygon meshes.

Our solution is to create a novel view synthesis system using a new technique called 3D Gaussian Splatting (3DGS). This solution is significantly faster and creates higher-quality renders than NeRFs, but one lingering issue remains. Standard 3DGS models are not compatible with industry-standard polygon meshes.

To solve these issues, we built off the work of Relightable Gaussians and created a ray-traced rendering system that seamlessly integrates triangle and Gaussian models. This system will allow users to change the scene's lighting in real-time, take advantage of global illumination features such as reflections and shadows, and add traditional polygon meshes into the scene.

This system will be beneficial in various circumstances, including game development and 3D art, as part of sales to allow customers to see an accurate and high-quality depiction of a product, and as an improvement for current NeRFs or photogrammetry systems.

This project, the Hybrid Relightable 3D Gaussian Rendering, is a tool designed to create high-quality 3D models, using machine learning, from videos of everyday objects. The user can then use it in 3D modeling spaces like games. We wanted our project to be able to be used by anyone without specialized equipment, and we wanted it to be available to everyone to download our program. The goal was for the user to just take a video of an object with their phone and send it to their Unity project. After running our program, it should have a high-quality 3D Gaussian model of the object.

Our program was to split our project into two parts: the machine learning part and the graphics path tracer part. The machine learning part was broken down into Structure-For-Motion and the Optimizer. The Structure-For-Motion code took the video of an object and transformed it into Gaussian points in the form of a ply file that was then passed into the Optimizer code. The Optimizer code took the cloud of Gaussian points and, using a PyTorch ML model, extracted data from the points. This data was then passed to the Unity path tracer code in the form of a .ply file.

With this information, the Unity side was able to create a 3D Gaussian object to raytrace. Our design meets the standard need for high-quality 3D Gaussian models. The requirements for anyone running our program would be that they have Unity installed and an NVIDIA graphics card to run the ML. With these two requirements, it doesn’t reach everyone, but we believe that anyone who would like to use our program would most likely meet the requirements already.

The next steps of our projects would be to increase performance, get it to run faster and produce higher quality 3D models, and add more features to the scope.