AI Art Weekly #65

Hello there, my fellow dreamers, and welcome to issue #65 of AI Art Weekly! πŸ‘‹ and a Happy New Year πŸŽ‰πŸŽ‰πŸŽ‰

I hope you all had some well deserved down-time and are ready to kick off 2024 πŸ’₯

Apparently even ML researchers are doing holiday breaks, so this issue is a bit shorter than usual, but I still managed to find some cool stuff for you to check out.

Let’s dive in:

  • MoonShot is a new video generation model that can condition on both image and text inputs
  • VideoDrafter creates content-consistent multi-scene videos
  • SIGNeRF can edit NeRF scenes in a controllable manner
  • Spacetime Gaussian Feature Splatting renders dynamic 8k scenes at 60fps
  • DreamGaussian4D generates animated 3D meshes from a single image
  • MACH creates lifelike 3D avatars from text descriptions
  • En3D generates 3D animatable humans
  • 3D-Fauna turns image of quadruped animals into 3D objects
  • Personalized Restoration can restore images and preserve identities
  • Auffusion is a new text-to-audio model
  • and some gems!

Cover Challenge 🎨

Theme: do whatever the f*ck you wanna do
150 submissions by 88 artists
AI Art Weekly Cover Art Challenge do whatever the f*ck you wanna do submission by dustinhollywood
πŸ† 1st: @dustinhollywood
AI Art Weekly Cover Art Challenge do whatever the f*ck you wanna do submission by BonsaiFox1
πŸ₯ˆ 2nd: @BonsaiFox1
AI Art Weekly Cover Art Challenge do whatever the f*ck you wanna do submission by simonjaquemet
πŸ₯‰ 3rd: @simonjaquemet
AI Art Weekly Cover Art Challenge do whatever the f*ck you wanna do submission by Cryptotocollege
πŸ₯‰ 3rd: @Cryptotocollege

News & Papers

MoonShot: Towards Controllable Video Generation and Editing with Multimodal Conditions

MoonShot is a new video generation model that can condition on both image and text inputs. The model is also able to integrate with pre-trained image ControlNet modules for geometry visual conditions, making it possible to generate videos with specific visual appearances and structures.

Moonshot video generated with an input image and the prompt wear a hat on the beach

VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM

VideoDrafter is a framework for content-consistent multi-scene video generation. The model is able to convert a text prompt into a multi-scene script, generate reference images for each scene, and finally output a video clip for each scene.

This entire video was generated with the prompt A person with red clothes is preparing dessert in the kitchen.

SIGNeRF: Scene Integrated Generation for Neural Radiance Fields

SIGNeRF is a new approach for fast and controllable NeRF scene editing and scene-integrated object generation. The method is able to generate new objects into an existing NeRF scene or edit existing objects within the scene in a controllable manner by either proxy object placement or shape selection.

SIGNeRF editing with shape selection

Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

Spacetime Gaussian Feature Splatting is a novel dynamic scene representation that is able to capture static, dynamic, as well as transient content within a scene and can render them at 8K resolution and 60 FPS on an RTX 4090.

Spacetime Gaussian Feature Splatting in action

DreamGaussian4D: Generative 4D Gaussian Splatting

DreamGaussian4D can generate animated 3D meshes from a single image. The method is able to generate diverse motions for the same static model and do that in 4.5 minutes instead of several hours compared to other methods.

DreamGaussian4D generated 3D meshes

Make-A-Character: High Quality Text-to-3D Character Generation within Minutes

MACH can create lifelike 3D avatars from text descriptions. The system is able to generate fully-realized 3D characters with detailed facial features, hair, and clothing within 2 minutes. The characters are highly-realistic and animatable, making them ready for immediate use in various scenarios.

MACH generated 3D character with the prompt A girl with deep red lips, blue eyes, and a purple bob haircut.

En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data

En3D is yet another method for generating 3D humans either from text or 2D images. This one is trained on millions of synthetic 2D images and is capable of producing visually realistic 3D humans that can be seamlessly rigged and animated.

An En3D generated 3D human dancing on a table

3D-Fauna: Learning the 3D Fauna of the Web

3D-Fauna is able to turn a single image of a quadruped animal into an articulated, textured 3D mesh in a feed-forward manner, ready for animation and rendering.

3D-Fauna generated 3D animals

Personalized Restoration via Dual-Pivot Tuning

Personalized Restoration is a method that can restore degraded images of faces while retaining the identity of the person using reference images. The method is able to edit the restored image using text prompts, enabling modifications like changing the color of the eyes or making the person smile.

Personalized Restoration example with George Clooney

Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation

Auffusion is a Text-to-Audio system that is able to generate audio from natural language prompts. The model is able to control various aspects of the audio, such as acoustic environment, material, pitch, and temporal order. It can also generate audio based on labels or be combined with an LLM model to generate descriptive audio prompts.

Also interesting

  • TF-T2V: A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
  • Image Sculpting: Precise Object Editing with 3D Geometry Control
  • ZeroShape: Regression-based Zero-shot Shape Reconstruction

β€œPondering existence” by me

And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:

  • Sharing it πŸ™β€οΈ
  • Following me on Twitter: @dreamingtulpa
  • Buying me a coffee (I could seriously use it, putting these issues together takes me 8-12 hours every Friday πŸ˜…)
  • Buying a physical art print to hang onto your wall

Reply to this email if you have any feedback or ideas for this newsletter.

Thanks for reading and talk to you next week!

– dreamingtulpa

by @dreamingtulpa