3D Scene Generation

LayoutVLM can generate 3D layouts from text instructions. It improves how well layouts match the intended design and works effectively in crowded spaces.

After NeRFs and Gaussian Splatting we got Triangle Splatting. A new method that can render real-time radiance fields at over 2,400 FPS with a 1280x720 resolution. It combines triangle representations with differentiable rendering for better visual quality and faster results than Gaussian splatting methods.

SceneFactor generates 3D scenes from text using an intermediate 3D semantic map. This map can be edited to add, remove, resize, and replace objects, allowing for easy regeneration of the final 3D scene.

RenderFormer can render images from triangle mesh representations with full global illumination effects.

LT3SD can generate large-scale 3D scenes using a method that captures both basic shapes and fine details. It allows for flexible output sizes and produces high-quality scenes, even completing missing parts of a scene.

ReStyle3D can transfer the look of a style image to real-world scenes from different angles. It keeps the structure and details intact, making it great for interior design and virtual staging.

GPS-Gaussian+ can render high-resolution 3D scenes from 2 or more input images in real-time.

PhysFlow can simulate dynamic interactions in complex scenes. It identifies material types through image queries and enhances realism using video diffusion and a Material Point Method for detailed 4D representations.

LVSM can generate high-quality 3D views of objects and scenes from a few input images.

VideoScene can generate 3D scenes from sparse video views in one step.

GeometryCrafter can recover detailed 3D point maps from open-world videos.

MVGenMaster can generate up to 100 new views from a single image using a multi-view diffusion model.

So far it has been tough to imagine the benefits of AI agents. Most of what we’ve seen from that domain has been focused on NPC simulations or solving text-based goals. 3D-GPT is a new framework that utilizes LLMs for instruction-driven 3D modeling by breaking down 3D modeling tasks into manageable segments to procedurally generate 3D scenes. I recently started to dig into Blender and I pray this gets open sourced at one point.

FreeTimeGS can reconstruct dynamic 3D scenes in real-time using Gaussian primitives that can appear at different times and places.

Google DeepMind has been researching 4DiM, a cascaded diffusion model for 4D novel view synthesis. It can generate 3D scenes with temporal dynamics from a single image and a set of camera poses and timestamps.

LayerPano3D can generate immersive 3D scenes from a single text prompt by breaking a 2D panorama into depth layers.

OmniPhysGS can generate realistic 3D dynamic scenes by modeling objects with Constitutive 3D Gaussians.

Wonderland can generate high-quality 3D scenes from a single image using a camera-guided video diffusion model. It allows for easy navigation and exploration of 3D spaces, performing better than other methods, especially with images it hasn’t seen before.

DAS3R can decompose scenes and rebuild static backgrounds from videos.

L4GM is a 4D Large Reconstruction Model that can turn a single-view video into an animated 3D object.