AI Art Weekly #86
Hello there, my fellow dreamers, and welcome to issue #86 of AI Art Weekly! 👋
What an insane week it has been! People say AI research is slowing down and we’re reaching a capabilities plateau, but I beg to differ. This week I’ve gone through another 180+ papers and projects from the world of computer vision and AI art, lets jump right into it!
In this weeks issue:
- Highlights: Luma AI: Dream Machine, Stable Diffusion 3 Medium, Midjourney Personalization
- 3D: M-LRM, Human 3Diffusion, WonderWorld, LE3D, AvatarPopUp, IllumiNeRF, GGHead, StableMaterials
- Image: Eye-for-an-eye, Image Neural Field Diffusion Models, Neural Gaffer, Ctrl-X, FontStudio, Layered Image Vectorization, LLamaGen, MimicBrush, CFG++, AsyncDiff, EMMA
- Video: HOI-Swap, T2S-GPT
- Audio: Action2Sound
- and more!
Want me to keep up with AI for you? Well, that requires a lot of coffee. If you like what I do, please consider buying me a cup so I can stay awake and keep doing what I do 🙏
Cover Challenge 🎨
For the next cover I’m looking for weird submissions! Reward is again $50 and a rare role in our Discord community which lets you vote in the finals. Rulebook can be found here and images can be submitted here.
News & Papers
Highlights
Luma AI: Dream Machine
Luma AI had the AI art community buzzing this week with their new video genration model called Dream Machine. Compared to Sora, it’s the most advanced video generation model that you can access today. It is able to generate 5 second clips with 120 frames in 120 seconds. I’ve compiled a list of some cool creations from the community over on X.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/dream-machine.gif)
Dream Machine example by @NathanBoey
Stable Diffusion 3 Medium
Stability AI finally released Stable Diffusion 3 weights, well, kind of. They released a new model called Stable Diffusion 3 Medium which is a smaller version of the original model they showcased a few weeks back. The new model also supports overall quality improvements in photorealism, prompt understanding and can generate text in images. Although human anatomy is still an issue apparently.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/sd3.jpg)
SD3 Medium examples
Midjourney Personalization
Midjourney released a new personalization feature that completely changes the way MJ interprets your prompts. For it to work, you have to rank at least 200 images and then add the --p
flag to the end of your prompts. I’m already having a ton of fun with it.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/midjourney.jpg)
Midjourney output with my personalization code --personalize 9zk3pun
3D
M-LRM: Multi-view Large Reconstruction Model
M-LRM is yet another model that can reconstruct high-quality 3D shapes from either a single or multiple images.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/m-lrm.gif)
M-LRM examples
Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models
Human 3Diffusion can reconstruct realistic avatars from a single RGB image, achieving high-fidelity in both geometry and appearance.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/human-3diffusion.gif)
Human 3Diffusion example
WonderWorld: Interactive 3D Scene Generation from a Single Image
WonderWorld can generate interactive 3D scenes from a single image and a text prompt in less than 10 seconds on a single A6000 GPU.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/wonderworld.gif)
WonderWorld example
Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis
LE3D can turn noisy RAW images into a Gaussian Splat and perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/le3d.gif)
LE3D example
Instant 3D Human Avatar Generation using Image Diffusion Models
GoogleMind’s AvatarPopUp can generate high-quality rigged 3D human avatars from a single image or text prompt in as few as 2 seconds.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/avatarpopup.gif)
Animated AvatarPopUp example
IllumiNeRF: 3D Relighting without Inverse Rendering
Also by Google, IllumiNeRF can relight images. The methods uses an image diffusion model conditioned on lighting and then reconstructs a NeRF with these relit images, from which it can render novel views under the target lighting.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/illuminerf.gif)
IllumiNeRF example
GGHead: Fast and Generalizable 3D Gaussian Heads
GGHead can generate and render 3D heads at 1K resolution in real-time.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/gghead.gif)
GGHead example
StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learning
StableMaterials an generate high-resolution tileable PBR materials from text prompts or input images in just 4 diffusion steps.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/stable-materials.jpg)
StableMaterials examples
Image
Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models
Eye-for-an-eye makes it possible for diffusion models to transfer the appearance of objects from a reference image to a target image.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/eye-for-an-eye.jpg)
Eye-for-an-eye example
Image Neural Field Diffusion Models
Image Neural Field Diffusion Models can be used to train diffusion models on image neural fields, which can be rendered at any resolution. This makes it possible to train diffusion models using mixed-resolution image datasets.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/infd.jpg)
CLIF render example at a 2048×2048 resolution
Neural Gaffer: Relighting Any Object via Diffusion
Neural Gaffer can relight any object in an image under any novel environmental lighting condition by simply conditioning an image generator on a target environment map.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/neural-gaffer.gif)
Neural Gaffer examples
Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance
Ctrl-X enables structure and appearance control for text-to-image and text-to-video models that with any image as input! This makes it possible to generate images and videos with the structure of one image and the appearance of another.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/ctrl-x.jpg)
Ctrl-X examples
FontStudio: Shape-Adaptive Diffusion Model for Coherent and Consistent Font Effect Generation
FontStudio can generate text effects for multilingual fonts. The model is able to interpret the given shape of a font and strategically plan pixel distributions within the irregular canvas.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/font-studio.jpg)
FontStudio examples
Layered Image Vectorization via Semantic Simplification
Layered Vectorization can turn images into layered vectors that represent the original image from coarse to fine detail levels.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/vector.jpg)
Layered Vectorization examples
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
LLamaGen is a new family of image generation models that are based on the same approach as LLMs. The largest model has 3.1B parameters and is able to generate 256x256 images.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/llama.jpg)
LLamaGen examples
Zero-shot Image Editing with Reference Imitation
MimicBrush can edit an image region of interest by drawing inspiration from a reference image by capturing the semantic correspondence between separate images in a self-supervised manner.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/mimic-brush.gif)
MimicBrush examples
CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
CFG++ fixes CFG’s issues with lower guidance scales, improving text-to-image quality and invertibility.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/cfgpp.jpg)
CFG++ comparison
AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
AsyncDiff brings parallelism to diffusion model which results in a significant boost in inference latency while minimally impacting the generative quality.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/asyncdiff.jpg)
2.8x Faster on SDXL with 4 devices. Top: 50 step original (13.81s). Bottom: 50 step AsyncDiff (4.98s)
EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Prompts
EMMA is a new image generation model that can generate images from text prompts and additional modalities such as reference images or portraits. It especially shines at preserving individual identities.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/emma.jpg)
EMMA examples
Video
HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
HOI-Swap can swap objects in videos with a focus on those interacted with by hands, given one user-provided reference object image.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/hoiswap.gif)
HOI-Swap examples
T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text
T2S-GPT can generate sign language videos from text and is able to control the speed of the signing.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/t2s.gif)
T2S-GPT example
Audio
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
Action2Sound can generate realistic action sounds for human interactions in videos. The model is able to disentangle foreground action sounds from the ambient background sounds and can even generate ambient sounds for silent videos.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/action2sound.jpg)
Action2Sound illustration
Also interesting
- Visual Words: Understanding Visual Concepts Across Models
- CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Models
- Weights2Weights: Interpreting the Weight Space of Customized Diffusion Models
- CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion
- AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction
- MCM: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation
My piece “Stop Breathing and Contemplate Your Insignificance” as part of “Polyptych 1” for the Eternal Peace collection is being exhibited in Basel, Switzerland this weekend 🧡
@spiritform made an AI short using Luma’s Dream Machine and Stable Audio for sound effects using only text.
@juliewdesign_ also created an AI short by using personalized Midjourney v6 images as input.
![](https://fly.storage.tigris.dev/aiartweekly/assets/issues/86/footer.jpg)
“up/above IV” by me.
And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:
- Sharing it 🙏❤️
- Following me on Twitter: @dreamingtulpa
- Buying me a coffee (I could seriously use it, putting these issues together takes me 8-12 hours every Friday 😅)
- Buying a physical art print to hang onto your wall
Reply to this email if you have any feedback or ideas for this newsletter.
Thanks for reading and talk to you next week!
– dreamingtulpa