AI Art Weekly #12

Hello there my fellow dreamers, welcome to issue #12 of AI Art Weekly 👋

If you enjoy my weekly ramblings, please consider buying me a coffee so I can stay awake and keep the good stuff coming 😉

Here are some of this weeks highlights:

  • Stable Diffusion v2.1. Audio-Driven Co-Speech Gesture Video Generation. “Snapchat” Filters on Steroid. Paint by Example.
  • Interview with _MemoryMod_.
  • Super simple (almost no setup required) Dreambooth and Stable Diffusion web interfaces on HuggingFace

Let’s jump in.

Cover Challenge 🎨

This weeks “ dreamscape ” challenge got 55 submissions from 31 artists and the community decided on the final winner. Congratulations to @_DrFetus_ for making the final round 🥳 And as always a big thank you to everyone who contributed!

Today’s the first day of snowfall in Switzerland. So what better theme for the next cover challenge than “ snowfall ”. Think tranquility, calmness, solitude. Prize is another $50. Rulebook can be found here and images can be submitted here.

I’m looking forward to all of your submissions 🙏

Reflection: News & Gems

Stable Diffusion v2.1

Just two weeks after releasing Stable Diffusion v2.0, Stability AI releases v2.1. The new model has been trained with a more diverse and wide-ranging dataset and a less aggressive filter for adult content. This results in improved image quality for architecture, interior design, wildlife, and landscape scenes, as well as better anatomy and hands and more diverse art styles. The model also supports non-standard resolutions for wider aspect ratios which is pretty neat. v2 also supports a new prompting style and Stability released a Prompt Book to teach you how to create different types of images using v2.

Prompt: A valley in the Alps at sunset, epic vista, beautiful landscape, 4k, 8k / Stable Diffusion v2.1-768 CLIP off

More ChatGPT examples

This week, my personal bubble was bursting with excitement as I saw the incredible ways people were using ChatGPT. From generating captivating stories to creating thought-provoking art, ChatGPT has proven itself to be a valuable tool for anyone looking to tap into their creative potential. Here are my top five favourite examples of how creatives have been using ChatGPT this week:

With just a brief example you can teach #GPTchat how to generate #AIart prompt templates for any topic (and yes, you can get it to dream up the topics too!) and then create matching prompts
December 3, 2022 18:12
This would've seemed straight out of a sci-fi movie just a year ago. #chatgpt #midjourney
December 1, 2022 18:45
i trained an ai chatbot on my childhood journal entries - so that i could engage in real-time dialogue with my "inner child" some reflections below:
November 27, 2022 23:12
I asked @OpenAI's ChatGPT to write some rap lyrics about being a graphic designer in the style of Eminem. I then asked it to create a prompt for @midjourney to create an image of Eminem toiling away as a graphic designer. #midjourney #ChatGPT
December 6, 2022 10:31
December 8, 2022 20:43

Overall, ChatGPT has proven to be an invaluable tool for anyone looking to unleash their creativity and push the boundaries of their artistic abilities. Whether you’re a writer, artist, musician, designer, or filmmaker, ChatGPT has something to offer you. Give it a try and see how it can inspire and enhance your creative process!

Score Jacobian Chaining

You might have come across some spinning hamburger or action figure this week (if not, take a look below). These were created with a newly open-sourced model called Score Jacobian Chaining (SJC) that utilizes pretrained 2D diffusion models to generate 3D renderings. That means you’re not bound to one specific model like Stable Diffusion, but it also means you can use custom Dreambooth models to generate customized results. There is a HuggingFace demo and a Google Colab notebook in case you want to try it out.

Score Jacobian Chaining examples

ANGIE: Audio-Driven Co-Speech Gesture Video Generation

With all these amazing new papers coming out every week, it’s hard to not build up a resistance against inventions that would have blown my mind 3 months ago. ANGIE is a novel framework which is able to generate high-fidelity co-speech gesture video sequences from audio inputs and a single image. How is this relevant for art you might think? Well, imagine creating a three-quarter-portrait of a character in MidJourney, generating some voice lines with GPT, voice it and then watch your character come to life by speaking to you using matching gestures based on what they’re saying. That’s pretty mind blowing to me. We can only hope that they release their code 🥹

Consecutive frames from different ANGIE examples. Check out the linked project page above for video examples.

Stitch it in Time: Snapchat Filters on Steroids

Now imagine you’ve created your gesturing speaking AI character from above and you want to fine-tune its facial appearance. Stitch it in Time makes it possible to change the age, expression and gender of a person in a video. Suddenly your character doesn’t have a soulless expression anymore but is smiling and looks 20 years younger or older.

“Stitch it in Time” example of Emma Watson as angry and young

ObjectStitch and Paint by Example

Say Goodbye to Photoshop™️. ObjectStitch lets you composite objects into an image scene by transforming the objects viewpoint, geometry, color and shadow all without manual labeling.

Paint by Example does something similar by allowing you to draw a mask onto the source image and feed it a reference image which then gets painted into the masked area, taking the surrounding context into account. There is HuggingFace demo in case you want to try it out.

ObjectStitch examples

Imagination: Interview & Inspiration

Today we talk to AI artist _MemoryMod_. Mod caught my eye a few weeks ago with their beautiful Twitter profile picture and stunning Cyberpunk art. After I found out how those images got created, I knew I wanted to publish an interview with Mod. Enjoy!

[AI Art Weekly] Hey Mod, what’s your background and how did you get into AI art?

I am a CGI artist with a background in classical painting and sculpture. I have done everything: from 3D renderings for products, building VR environments, to digital sculpture for Marvel movies and Netflix shows. Currently I manage projects for an artist doing large-scale sculpture installations around the world. I got into AI art by dabbling with early google Colab documents and then got hooked when the local install of Stable Diffusion was released. 

“MemoryMod Genesis 033” by MemoryMod

[AI Art Weekly] Do you have a specific project you’re currently working on? What is it?

I am currently working on my Genesis series which I describe as an AI fever dream envisioning the merge of humanity and machines. The idea is inspired from the movies and books I grew up with such as Akira, Ghost in the Shell, and Snowcrash. The fact that the images are created by AI perfectly plays into the aesthetic of the world building I’m exploring. 

In addition, I have a few projects on the back burner that have gained recognition via Claire Silver’s second AI art contest. One piece won the 3rd place, and another was a finalist.

“Stemma” by MemoryMod

[AI Art Weekly] What does your workflow look like?

My current workflow consists in using Stable Diffusion 1.4, which focuses on custom model training. Experimentation and iteration are obvious keys here. I sometimes end up training models on 5-7 different image sets and then experiment with checkpoint, merging the results to other trained models.

Usually, I have a very basic prompt, and use it during the training phase so that I can see how the model is changing from version to version. Then I use the x/y plot in Automatic1111’s webui to iterate and attempt to find outputs that fit the aesthetic I am going for. Once I find a model and prompt that is doing what I want (or something unexpected that I like) I then iterate further using the x/y prompts with smaller more specific changes. 

[AI Art Weekly] What is your favourite prompt when creating art?

Cyberpunk, haha! This was a prompt I used for my Genesis series.

I’ve found prompting on trained models to be different than prompting on the vanilla model. Image training in Dreambooth seems to evolve the entire model’s response to prompts and, as a result, my prompts are very simple. For instance — “portrait of a girl as cyberpunk“ — and then I let the training take the reins. Sometimes, the prompts are weird broken sentences that get interesting results.

“MemoryMod Genesis 017” by MemoryMod

[AI Art Weekly] How do you imagine AI will be impacting society in the near future?

Artists throughout history have employed others to create parts of or their entire artwork for them. During the Renaissance apprentices would paint parts of the masters’ paintings in order to assist in commissions. In more recent decades, some of the biggest artists in the world don’t even touch a brush or define a curve of a sculpture with their own hands. Instead, some have become more akin to art directors having tens to hundreds of other people create the work for them. In some ways, this can be seen as an advanced form of prompting! AI art, I think, can be seen as a form of this — with the peculiarity that this particular form is accessible to the general population. AI plays the role of the hired hand executing the art itself. The impact will be widespread, especially because the technology continues to advance. It is easy to foresee AI art changing a few art fields in the near future (concept art, for instance) and changing the way people generally make art and employ references. 

[AI Art Weekly] Who is your favourite artist?

This past year I’ve been fascinated with the paintings of Sainer Etam, especially after seeing his show in Rouen, France over the summer. There are so many interesting aspects to his work, from composition and color, to the blending of abstract elements into representational, that are so well executed and inspiring.  

“BOMBERMAN” mural in Weil am Rhein, Germany by Sainer Etam

[AI Art Weekly] Anything else you would like to share?

Needless to say,  I’m looking forward to the future of AI art and technology. It is an exciting time to be exploring and creating with all these tools and there are a lot of people doing amazing things, and more to come I’m sure.

Creation: Tools & Tutorials

These are some of the most interesting resources I’ve come across this week.

“AI plays the role of the hired hand executing the art itself, cyberpunk girl action scene, 3d crocheted wool style, by sainer etam” by me

And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:

Reply to this email if you have any feedback or ideas for this newsletter.

Thanks for reading and talk to you next week!

– dreamingtulpa

by @dreamingtulpa