Senior Design Team 40 • sdmay25-40 SD Site

Project Overview

Our project is a novel view synthesis system that uses 3D Gaussian splats with a raytracer that allows for real time dynamic lighting and the incorporation of traditional polygon meshes.
Use a Machine Learning model to extract material properties from videos to allow for dynamic lighting.

Problem Statement

The process of creating 3D renders or models from a video or a series of photos of a real world location is called novel view synthesis. Previously this was done using Neural Radiance Fields or photogrammetry but these methods are slow, can produce low quality models and renders, and can’t be combined with traditional polygon meshes.

Our solution is to create a system which performs novel view synthesis using a new technique called 3D Gaussian Splatting which works by transforming the images into a cloud of 3D Gaussians and then rendering it. This solution is significantly faster and creates higher quality renders and models but still has two lingering issues. Standard gaussian renderers are not relightable meaning that the scene created with them cannot have different lighting than what the original images or video had and it still cannot be combined with traditional polygon meshes.

To solve these issues we plan to implement a system which uses a machine learning model to extract the lighting data from a 3D Gaussian model and then render that using a raytracer. This will allow a user to change the lighting of the scene in real time, take advantage of global illumination features such as reflections and shadows and add traditional polygon meshes into the scene which the raytracer can also render.

This system will be beneficial in a variety of circumstances including in game development and 3D art, as part of sales to allow customers to see an accurate and high quality depiction of a product, and as an improvement for anything currently using NeRFs or photogrammetry.

Team Members

Jackson Vanderheyden

Graphics Scope Manager

The skills, expertise, and unique perspectives Jackson Vanderheyden brings to the team are: computer graphics industry experience, multiple Unity projects, & previous experience implementing ray tracing.

Brian Xicon

Machine Learning Scope Manager

The skills, expertise, and unique perspectives Brian Xicon brings to the team are: C, C+ +,HTML, CSS, JS, Python, PyTorch experience, Machine Learning experience, and Computer Vision experience.

Luke Broglio

Schedule Manager

The skills, expertise, and unique perspectives Luke Broglio brings to the team are: Experience wth C, C++, python, graphics programming, Unity, HTML/CSS, Javascript, writing a raytracer and experience with working in agile/scrum development environments.

Ethan Gasner

Documentation Manager

The skills, expertise, and unique perspectives Ethan Gasner brings to the team are: C, C++, python, Ai Experience, Machine learning experience, Unity Experience, unique Cyber security perspective, Javascript (+html/css). additionally, a cooperative and easy going attitude.

Kyle Kohl

Communication Manager

The skills, expertise, and unique perspectives Kyle Kohl brings to the team are: C, C++, Java, Python, a little bit of HTML and CSS. He has the definite advantage of being an extrovert that loves to encourage others. Lots of experience in the communication role.


Weekly Reports

Report 1 || (09/13-09/19)
Report 2 || (09/20-09/26)
Report 3 || (09/27-10/03)
Report 4 || (10/04-10-10)
Report 5 || (10/11-10/17)
Report 6 || (10/18-10/24)
Report 7 || (10/25-10/31)
Report 8 || (11/01-11/07)
Report 9 || (11/08-11/14)
Report 10 || (11/15-11/21)

Design Process Slides

Product Research
Problem and Users
User needs and Requirements
Project Planning
Detailed Design
Design Check-In
Prototyping
Ethics and Professional Responsibility

Design Documents

Design Document

Context and Techniques Document