AI Toolbox

AudioLM can generate high-quality audio by treating it like a language task. It produces coherent speech and piano music continuations while keeping the speaker’s voice and style consistent, even for new speakers.

Splatter Image can reconstruct a 4D video from a single image at 38 frames per second and render them at 588 frames per second.

ARF: Artistic Radiance Fields can transfer the style of a 2D image to a 3D scene by stylizing radiance fields. It captures style details while ensuring that different views of the scene look consistent, resulting in high-quality 3D content that closely matches the original style image.

MCVD can generate videos and predict future and past frames using a masked conditional score-based diffusion model. It achieves high quality and diversity in generated frames, excelling in various video synthesis tasks.

Adobe is entering the image-to-3D game. LRM can create high-fidelity 3D object meshes from a single image in just 5 seconds. The model is trained on massive multi-view data containing around 1 million objects. The results are pretty impressive and the method is able to generalize well to real-world pictures and images from generative models.

Even though Gaussian Splats have seen a lot of love, NeRFs haven’t been abandoned. This week we got three different NeRF editing papers. The first two are about inpainting. InseRF and GO-NeRF are both methods to insert 3D objects into NeRF scenes.

[Temporal Residual Jacobians] can transfer motion from one 3D mesh to another without needing rigging or shape keyframes. It uses two neural networks to predict changes, allowing for realistic motion transfer across different body shapes.

UnZipLoRA can break down an image into its subject and style. This makes it possible to create variations and apply styles to new subjects.

SDEdit can generate and edit photo-realistic images using user-guided inputs like hand-drawn strokes or text prompts. It outperforms GAN-based methods, achieving high scores in realism and overall satisfaction without needing specific training.

[Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries] can retrieve high-quality sound effects from a single video frame without needing text metadata. It uses a combination of large language models and contrastive learning to match sound effects to video better than existing methods.

GFPGAN can restore realistic facial details from low-quality images using a pretrained face GAN. It works well on both synthetic and real-world images, allowing for quick restoration with just one pass, unlike older methods.