AI Art Weekly #20
Hello there my fellow dreamers and welcome to Issue #20 of AI Art Weekly! 👋
The newsletter has grown by over 200 subscribers since last week and we are now more than 1100 strong 🔥. Thank you all for your new and continued support. It’s the fuel that keeps me invested in growing and building this community. If you have a few seconds, I would love to hear where you all came from, as I have no idea what’s causing the sudden surge in subscribers this week. With that out of the way, let’s dive in. This week’s highlights are:
- Gen-1 – RunwayMLs new video-to-video model which is able to produce constant and flickerless styled transfers
- TEXTure – a novel method to generate textures for 3D meshes
- PEZ – a new way of turning images into optimzied text prompts
- Interview with “AI gardener” and designer Linus Ekenstam
- X-Decoder – which lets you edit subjects in existing images
Sponsored – This weeks issue and challenge is sponsored by @YoupiCamera. They recently released a personalized AI art sharing app for iOS called MetaReal – a mix of Lensa A.I. and MidJourney. Join the “ruins” cover art challenge to get one month premium membership for free.
Cover Challenge 🎨
The challenge for this weeks cover was “fungus” and we received a wooping 132 submissions from 80 artists 🤯. The community decided on the final winner:
Congratulations to @spiritform for winning the cover art with such a beautiful and frankly funny artwork 🍄👑. Which reminds me of this joke:
Q: Why is the mushroom always getting invited to the party?
A: Because he’s a fun-guy!
Sorry 🤣. Anyway, a big thank you to everyone who contributed! It’s fun and humbling to see a challenge get this many submissions!
The next challenge is about exploring structures from the past that have remained over years and centuries: “ruins”. Medieval, fantastical, overgrown, alien and abandoned buildings are the theme for this week! The reward is another $50. Rulebook can be found here and images can be submitted here.
I’m looking forward to all of your submissions 🙏
If you want to support the newsletter, this weeks cover is available for collection on objkt as a limited edition of 10 for 3ꜩ a piece. Thank you for your support 🙏😘
Reflection: News & Gems
GEN-1 Video-To-Video
After working on Latent Diffusion in 2021 and Stable Diffusion in 2022, RunwayML introduced Gen-1 this week. Gen-1 is a video-to-video model and system that takes an input video and lets you apply a new style given a simple text prompt. Now, the concept isn’t new, but the key difference is in how Gen-1 is able to retain the consistency of the original input video across the entire video in comparison to current methods like img2img or InstructPix2Pix. But apart from only text guided editing, Gen-1 is also able to stylize a video given an input image, turn mockup videos into animated renders, isolate objects and modify only those, convert untextured 3D renders into animated sequences and even fine-tuning the model based on your own training images. Access can be requested via this Google Form, let them know I sent you (my Runway email is pulleasy@gmail.com) 😘
Gen-1 Render mode: Turn untextured renders into realistic outputs by applying an input image or prompt.
Pix2Pix Zero: Zero-shot Image-to-Image Translation
Gen-1 wasn’t the only research that was published regarding image manipulation. Pix2Pix-Zero is a novel zero-shot method that requires neither fine-tuning nor text input to edit images on the fly while preserving the input structure of the given image. How it fares in comparison to Microsoft’s X-Decoder remains to be seen as soon as the code becomes available. A link to the X-Decoder demo can be found below in the resource section.
pix2pix-zero demonstration
TEXTure: Text-Guided Texturing of 3D Shapes
Looking to texture your 3D shapes? TEXTure is a novel method which takes an input mesh alongside a conditioning text prompt and paints the mesh with high-quality textures. This is by far the most advanced generative 3D texture method I’ve seen so far. Dabbling with 3D is pretty high on my todo list and luckily there is code and a HuggingFace space available for this one.
TEXTure examples
Neural Congealing: Aligning Images to a Joint Semantic Atlas
Neural Congealing is a zero-shot self-supervised framework for detecting and jointly aligning semantically-common content across a given set of images. This makes it possbile to extract an average representation of a subject out of a set of multiple images, edit the average, and then propagate the changes back to the original images, making it possible for example to efficiently edit multiple frames of a video.
Neural Congealing example
Hard Prompts Made Easy (PEZ)
This week, @ywen99 and team presented PEZ (short for Hard Prompts Made Easy). PEZ is a new method that is able to turn images and longer prompts into shorter optimized “harder” text prompts, making it easier to generate, discover, and mix and match image concepts without prior knowledge on how to prompt a diffusion model. There is a demo on HuggingFace and the code is available on GitHub.
PEZ example
@paultrillo put out a stunning animation transforming some plain ol’ video footage into a cloudy dream combining tools like @NVIDIAStudio #instantnerf, #stablediffusion, @runwayml and #aftereffects. All on a @dell laptop. Pretty impressive.
@angrypenguinPNG is building an 360 image generator with #stablediffusion. Try the prompt: medieval castle ruins in sunset forest, golden trees, in the style of elden ring, octane render, wide angle.
@dreamwieber shared his process on how he creates Stable Diffusion animations using ChatGPT and Midjourney.
With last weeks audio models arrived the ability to turn images into music. @DrJimFan shared how with the combinations of image captioners, LLM and text-to-audio we can generate inifinite atmopsheric background music for images.
Imagination: Interview & Inspiration
In this week’s issue of AI Art Weekly, we talk to AI gardener, designer and entrepreneur Linus Ekenstam. Linus caught my and many others attention with his explorations of various subjects through Midjourney. If you’re stuck with your prompts or just want some inspiration, checkout his Twitter feed or Substack for prompt tutorials. Let’s jump in.
[AI Art Weekly] What’s your background and how did you get into AI art?
My background involves starting companies and working as a product designer. I established my first company back in 2009 and have been involved on the Internet since 2004/2005. I was ahead of the curve on many of the major trends since then. I have a rich history of work in the industry, including serving as an early employee and senior designer at Typeform, helped to start up Thingtesting as the head of product and design, and more recently serving as the principal designer at Flodesk.
More recently, I have been exploring the world of generative AI, starting with GPT-2 and then GPT-3. My interest was piqued by image generation with DALLE2, and then my time playing with Midjourney led to a deep dive into the world of generative AI. It’s easy to spend hours creating in MJ, and I could say that I followed the internet’s “breadcrumbs” to find the next big thing.
“Macro shot of a human eye, blue iris like a galaxy, window into the soul –ar 3:2 –v 4 –q 2 –upbeta –s 250” by Linus
[AI Art Weekly] Do you have a specific project you’re currently working on? What is it?
Since 2020, I’ve been dedicating my time to developing a mental wellness app for iOS called Sensive.xyz. It’s a scientifically-grounded mood tracker and journaling app that has helped a few thousand of people so far. However, it’s currently taking a back seat as I’ve started a new project with my partner and friend in October 2022. Our latest venture is a generative story-making tool called bedtimestory.ai.
We’re creating a narrative platform that enables users to create and share stories. It’s been a thrilling experience so far and the platform has already gained around 10’000 members, with about 20’000 stories created since its launch less than two months ago.
Our focus is on building out the basic features and improving the user experience. Our vision for the platform is grand and we feel that the world is our oyster. Currently, we’re working on enhancing text and image capabilities, adding more controls to the generation aspect, and developing a comprehensive editor. In the future, we’re also looking to add personalised characters and the ability to turn your kids into cartoon characters, which we believe has the potential to be very powerful.
Next, we plan to support more types of distribution channels, such as audio and video. But what we’re hoping the most for is a Midjourney API, so we can start integrating images made with Midjourney.
[AI Art Weekly] What does your workflow look like?
When it comes to my Twitter account and the work I’m doing there and in my newsletter, it stems from play. I experiment a lot with these tools, and I’m mindful of my approach when I explore and play. Sometimes, I find interesting learnings or outcomes that I try to capture and share. I’m humbled that it resonates with people. I never expected any of this to work or be interesting, but it happened and I’ve been building on that ever since.
[AI Art Weekly] What is your favourite prompt when creating art?
It’s difficult to say; I’ve been doing a lot of interior shots. For those, this prompt is great because you can adjust parameters as desired:
“Home design, Interior Design in the style of a geometric, polymorph, Cutest vibe, colorful, magical, whimsical, surreal, fantasy, detailed, complex, polished, 8k, Octane Render, Unreal Engine, Realistic, Abstract –v 4 –q 2 –ar 2:3” by Linus
Another really nice concept I’ve been exploring is the art of Knolling, about which I’ve made entire thread about.
“crystals, rose quarts, onyx, carnelian, jasper, prasiolite, agate, Knolling on white tray, white Background –q 2” by Linus
Then there is this funny series where I put batman in odd situations:
“batman in a domestic setting, making pasta –ar 3:2 –q 2 –v 4 – uplight” by Linus
[AI Art Weekly] How do you imagine AI (art) will be impacting society in the near future?
It’s difficult to predict the future of generative art and AI, but one thing is for certain: the genie is out of the bottle and it’s here to stay. In fact, the term “generative art” may not even last long, as AI becomes just another tool in the artist’s toolbox. The pace of change will be so rapid and intense that it will be challenging to keep up with it all. We’re already seeing this with new breakthroughs happening almost weekly.
As we look ahead, we can expect to see higher resolution models and all issues we are seeing with hands and details will slowly be solved. The trend towards text-to-video, video-to-video, text-to-3D, and 3D-to-3D (most of which is already here) will only continue and become increasingly accessible and user-friendly. Additionally, I think prompt engineering may become less common as natural language becomes the preferred way to describe what we want. Maybe to some degree there’ll still be some prompt design going on, but I don’t think it will be like today.
The impact of AI on jobs will be significant, and some jobs will be affected sooner than others. For example, there was a recent announcement about an AI powered software that can perform Text-to-Figma (a design tool for digital designers to mockup user interfaces for apps and websites). We’ll just have to deal with this becoming an increasingly important part of our daily lives. It’s important to start learning these tools now before it’s too late.
Text-To-Figma by Galileo AI
[AI Art Weekly] Who is your favourite artist?
I’m a minimalist at heart and admire the timeless designs of Dieter Rams and Jesper Kouthoofd at Teenage Engineering. When I was younger, I wanted to redesign everything around me - streets, lamp posts, sidewalks, houses, parks, and household items - because I felt it lacked cohesion. Now that I’m older, I’ve learned to appreciate the imperfections of the world.
I’m also a big fan of Brian Donnelly, aka Kaws. Lorena is an amazing female artist whose colourful art I admire. My current favourite creator on YouTube is @GawxArt. Despite his young age, he has already achieved a lot and I think we’ll be seeing more of him in the future.
Music is also a big part of my life. I think I streamed well over 100,000 hours on Spotify last year and enjoy exploring new genres, artists, and subcultures. I recommend KOKOKO! and Cheikh Ibra Fam for anyone looking for something new to listen to.
For relaxation, I often hang out on poolside.fm, which is run by my good friend @marty.
“The Wings of Hermes” by Lorena Eloizaga
[AI Art Weekly] Anything else you would like to share?
I think that’s pretty much it. You can follow me on Twitter and if you’d like a closer connection, consider signing up for my weekly newsletter Inside My Head. I look forward to seeing you all on the interwebs.
Creation: Tools & Tutorials
These are some of the most interesting resources I’ve come across this week.
Last week I shared an almost flickerless example video created by @CoffeeVectors using Automatic1111 with InstructPix2Pix. @fffiloni now created a simple @Gradio app which lets you do the same thing – minus the hassle to install anything.
@fffiloni also created an app to generate audio from an image with last weeks AudioLDM model. If you want to take the above a step further, take a frame from your Pix2Pix-video, generate audio from it, and combine it with your video. Just take a look at this cute burning doggo.
If you just want to create audio from text instead of an image, @liuhaohe got you covered. There is also a Replicate model which gives you the ability to generate longer audio.
While InstructPix2Pix is good at manipulating styles of an image, @Microsoft’s Instruct-X-Decoder is good at object-centric instructional image inpainting/editing.
If you’re looking to sync Deforum animations to music, framesync is a handy tool to do just that. It’s similar to what I’ve been building but also lets you create keyframes without having to use an audio file.
“genie rowland pino percy lovely ponies moment seaside fra” by me playing around with PEZ Dispenser’s example prompt in MJ
And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:
- Sharing it 🙏❤️
- Following me on Twitter: @dreamingtulpa
- Leaving a Review on Product Hunt
- Using one of our affiliate links at https://aiartweekly.com/support
Reply to this email if you have any feedback or ideas for this newsletter.
Thanks for reading and talk to you next week!
– dreamingtulpa