Could we develop a game engine that uses a stable diffusion model to render user-desired segmentation masks? Then, can we condition an off-the-shelf near real-time diffusion model on these masks?
- The user will input an image segmentation mask.
- We will develop a game engine with a near real-time diffusion model to create imagery based on the user-specified segmentation masks.
- The users specify game elements using parameters such as height, width, and shape, and our model works to turn these specifications into game elements in real-time.
Basically, imagine you are playing a video game, and you don’t like the map. You can create your own by dragging game elements, like trees, cars, and buildings across the screen and have the game update simultaneously so you can see what you made!
Minimum Viable Product: A diffusion model that takes in a segmentation map and generates a corresponding game scene using predefined game elements (i.e., trees, cars, buildings, roads).
Project Leads
