AI Art Weekly #23

Hello there my fellow dreamers and welcome to issue #23 of AI Art Weekly! 👋

I hope you’re all well and kicking. Writing these newsletters is like being in a time machine as fast as this field is moving.

This weeks highlights are:

  • Composer (ControlNet 2.0) sneakpeak
  • Human Motion Diffusion just got so much better
  • Interview with Olena speakingtomato 🍅
  • OpenPosexBlender plugin
  • MultiDiffusion Spatial Control demo

Cover Challenge 🎨

The prompt challenge for this weeks cover went well. I put a ton of time into improving the algorithm this week to generate the AI collage canvases I presented last week. We got a wooping 87 people to submit a prompt which resulted in canvases with 97-107 tiles at a resolution of 5120x7168 pixels which took around one hour to render each. Out of the three canvases, the community chose their favourite to be this weeks cover. You can see the full glory of the piece here.

A big thank you to everybody who made this experiment possible 🙏. I’m gonna experiment some more with this in the upcoming weeks and we’ll certainly do another one in the near future. So keep an eye out for that.

For the next challenge, we’re back to traditional image prompting. The next challenge is all about “fashion”. Time to become a fashion designer and design some clothes. The reward is again $50. Rulebook can be found here and images can be submitted here.

I’m looking forward to all of your stunning submissions 🙏


Reflection: News & Gems

Composer

One of my most anticipated upcoming projects is Composer. Similar to ControlNet, Composer gives more controllability to the image generation process by providing depth maps, sketches, segmentation masks and even color palettes for guidance, making it easy to recolor existing images for instance. Three standout features that I’m looking forward are 1) the ability to explore interpolation frames between one or two images by providing a fixed color palette or other an guidance method, 2) style and pose transfer for images and 3) region-specific image editing, making it possible to mask a specific area of an image and recolor it with a palette or image embedding. Can’t wait.

Image interpolations example when fixing sketch, depth, segmentation map and/or palette

Human Motion Diffusion as a Generative Prior

I’ve shared some Motion Diffusion research before, but this one takes the cake. priorMDM is a novel human motion diffusion model that is able to generate up to 10-minute long animations of prompted intervals including accurate transitions between them. But not only that, the model is the first of its kind that is able to generate motion for two people interacting with each other by fine-tuning the model with as little as 14 motion examples. The most exciting part is that the model is introducing end-effector control. Similar to what ControlNet did to image control, end-effectors enable the control of the general trajectory or the hand positioning while the rest of the human motion is generated in relation to that input. Imagine being able to combine this with SinMDM from two weeks ago to control the style of the motion and we have unlocked the ability to generate a limitless amount of human motion in any way we want. Making it for example possible to procedurally generate a character in a 3D world based on its surroundings and the actions they’re taking.

Trajectory + Left Hand Control (DiffusionBlending) example

Yet another text-to-video diffusion model

This (unnamed) text-to-video model was published on ModelScope and it seems to be able to generate 16 frame videos which can be interpolated and upscaled to up to 64 frames with a resolution of 256x256. The examples on the model page look amazing.

Video diffusion examples

I hear your true colors: Image Guided Audio Generation

Now imagine you want to add audio to the generated videos above, easy peasy (when the code gets released). Just take the frames from the video and put it into Im2Wav to generate a semantically relevant 4 second audio clip – and voilà – no need for a sound engineer anymore. It’s worth to checkout the examples on the project page.

im2wav architecture

All-In-One-Deflicker

Now one of the biggest issues with frame-to-frame diffusion so far has been the insufferable flickering we all had to look at during the past few months. Enter All-In-One-Deflicker, a general postprocessing framework that can remove different types of flicker from various videos, including videos from video capturing, processing, and generation. I can’t wait to see how this will affect diffusion based animations generated with Deforum or ControlNet 🤞.

All-In-On-Deflicker example


Imagination: Interview & Inspiration

Two tomatoes were sitting in a fridge. One tomato says to the other: “Ain’t it cold?” And the other replies: “HOOOOOLY SHIT, A SPEAKING TOMATO!!!” 🤪

There are only a few people in the world that can have a 10/10 joke like this as an introduction and Todays AI Art Weekly interview guest is on of them. Please welcome Olena speakingtomato 🍅! Olena is a member of TEZgirls and the MAIF collective, and her work has been exhibited in London, Miami, Milan, Paris, and soon, LA. I’m delighted to have her on the newsletter. Let’s dive in!

[AI Art Weekly] Olena, what’s your background and how did you get into AI art?

As an artist who focused on camera portraits for 15 years, I was drawn to the world of AI art after a period of immobility due to surgery. During this time, I had the opportunity to explore AI technology, which fascinated me and inspired me to try my hand at creating art using these cutting-edge tools. Since then, I have been creating figurative art using a mix of modernism, surrealism, and symbolism, with a focus on creating meaningful details that capture the essence of my life and dreams. The process of transitioning from traditional photography to AI art has been both challenging and rewarding. While I miss the tactile experience of working with a camera, I have found that the endless possibilities of AI art have opened up new avenues for creative expression. I’m excited to see where this journey takes me next and to continue exploring the fascinating intersection of art and technology.

Camera work by @speakingtomato

[AI Art Weekly] Do you have a specific project you’re currently working on? What is it?

I’m currently working on my main collection on Foundation, which is a series of AI-generated artworks that explore themes of identity, memory, and the passage of time.

[AI Art Weekly] What drives you to create?

What drives me to create is the ability to use artificial intelligence to create unique works of art. I became so fascinated with AI technology that I shifted the focus of my artistic endeavors. I’m driven to create art that resonates emotionally with viewers and pushes the boundaries of what is possible with AI art.

“🐖 Swineherd 🐖” by @speakingtomato

[AI Art Weekly] What does your workflow look like?

My workflow typically commences with an idea that manifests either during the day or in a dream. Occasionally, I have a fully-formed image in mind, while other times, I provide the AI with a sketch of the words swirling in my head and collaborate with it.

To bring the final product to life, I input my initial concept into the AI model, which generates an initial image. Subsequently, I use a suite of software, including digital painting in Adobe Photoshop, to refine the image. I work with a few different types of AI, which allows me to polish the artwork and imbue it with meaningful details that capture the essence of my vision.

“🦌 Lost 🦌” by @speakingtomato

[AI Art Weekly] What is your favourite prompt when creating art?

My favorite prompt for creating art is to explore themes of identity, memory, and the passage of time. These concepts enable me to delve deeply into the human experience and create work that is both deeply personal and universally relatable.

However, I should note that my prompt may vary from work to work, as I often draw inspiration from a variety of sources, including personal experiences, cultural and historical events, and other works of art.

“🏡 Welcome home 🏡” by @speakingtomato

[AI Art Weekly] How do you imagine AI (art) will be impacting society in the near future?

As an AI artist, I wholeheartedly believe in the power and potential of AI-generated art. It’s incredible to think that I am contributing to the development of this emerging field and shaping the future of art history. AI art will enable new forms of creativity and expression that were previously impossible or difficult to achieve. With the potential to push boundaries and create new aesthetic experiences, AI-generated art will captivate and engage audiences. AI art will undoubtedly play an increasingly important role in the art world and society at large.

[AI Art Weekly] Who is your favourite artist?

I am inspired by a wide range of artists, including traditional artists and AI artists. It’s hard for me to pick just one favorite, but here are a few artists whose work I greatly admire: Nikolena Petolas, Casey Weldon, Daria Petrilli, and Dolce Paganne.

“🩸🐲Time Is A Dragon🐲🩸” by @dolce_paganne

[AI Art Weekly] Anything else you would like to share?

Just my catchphrase: My images tell stories. Thank you for the opportunity to share your work with you.


Creation: Tools & Tutorials

These are some of the most interesting resources I’ve come across this week.

AI collage made out of bacon prompts inspired by the #baconwave crew (@CoffeeVectors, @KyrickYoung, @amli_art, @weird_momma_x) and me

And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:

Reply to this email if you have any feedback or ideas for this newsletter.

Thanks for reading and talk to you next week!

– dreamingtulpa

by @dreamingtulpa