AI Art Weekly #35

Hello there, my fellow dreamers, and welcome to issue #35 of AI Art Weekly! 👋

I had the honour this Thursday of speaking about AI art in front of a selected group of individuals at the Faculty of Law in Basel, Switzerland, who where curious about the development of the field over the last few months. I’ve compiled the talk into a Twitter thread in case you want to check it out.

Apart from that, this week was crazy in terms of released papers. My paper aggregation script returned over 200 results 😅🔫 Well… don’t forget to buy me a coffee so I can keep curating only the most interesting ones 😜

Anyway, here are this weeks highlights:

• Adobe brings Genrative AI into Photoshop with “Generative Fill”
• Mind-Video reconstructs videos from brainwaves 🤯
• CoDi can generate any-to-any combination of text, images, videos and audio
• Interview with AI artist Marina Ahmadova
• Unofficial DragGAN implementation released

---

Cover Challenge 🎨

---

News & Papers

Photoshop: Generative Fill

Generative AI art just got a lot more accessible this week. Adobe released a new feature called “Generative Fill” which enables text-to-image, inpainting and outpainting directly within Photoshop. The new feature is powered by their in-house model called Firefly which was trained on the Adobe Stock library.

Mind-Video: High-quality Video Reconstruction from Brain Activity

It was November last year when we were introduced to MinD-Vis, that one paper that was able to reproduce images from brainwaves. A lot of people thought this thing can generate any imaginable image out of thin air. Just to clarify, this isn’t the case. The output depends heavily on what Image/fMRI combination the model was trained on. Nonetheless, this is an incredible achievement and this week the team behind it brings us Mind-Video which is able to reproduce videos from brainwaves. I hope someone will invest a lot of money into this and builds a large enough dataset so I can visualize my dreams. Imagine a streaming platform where people could share their dreams 🤯

MMS: Massively Multilingual Speech by Meta AI

Meta AI revealed their MMS project this week which is able to transcribe AND generate speech for a 1000+ languages. Compared to OpenAI’s Whisper, models trained on the MMS dataset cover 11 times more languages and achieve half the word error rate. Impressive.

Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields

What comes after Text-to-Image, Text-to-Video, Text-to-3D? Apparently Text2Nerf. Text2NeRF, a text-driven 3D scene generation framework, combines the neural radiance field (NeRF) and a pre-trained text-to-image diffusion model to generate diverse view-consistent indoor and outdoor 3D scenes from natural language descriptions.

CoDi: Any-to-Any Generation via Composable Diffusion

Okay, now that we got Text-to-NeRF, what about we if we could combine it with other modalities? CoDi is a framework for any-to-any generation via composable diffusion. It can generate language, images, videos, and audio from any combination of the former. For example you can generate a video with audio from a text description, an image and an audio clip. Or generate new audio by combining an image with an existing audio clip. Any combination is possible.

Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models

Video-to-Video output so far always was a bit hard to get right because of flickering or not terrible coherency. Gen-1 by RunwayML is what I consider State of the Art when it comes to this task so far. The latest Control-A-Video aka Video-ControlNet comes pretty close and introduces the ability to guide video generation using depth maps, canny and HED edge detection as well as text descriptions. There is no code yet, but apparently this will be open-sourced 🥳

Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape

Sin3DM impresses me quite a bit. The method is able to generate variations of a 3D objects from only a single textured shape. Imagine you’ve a 3D asset you want to reuse but in a slightly different way? Sin3DM makes it possible to generate an infinite number of variations with different sizes and boundaries.

Voyager: An Open-Ended Embodied Agent with Large Language Models

Let’s talk about Minecraft for a minute. Voyager presents the development of an embodied agent that uses large language models to explore and acquire skills in Minecraft. It introduces three key components for the agent: an automatic curriculum, a skill library, and an iterative prompting mechanism. I think we aren’t that far away from being able to build worlds filled with NPCs that can learn by themselves and react to a player’s input organically.

More gems

• Text-3DH: Text-guided 3D Human Generation from 2D Collections
• MOD Extraction: Modulation Extraction for LFO-driven Audio Effects
• Restore Anything Pipeline: Segment Anything Meets Image Restoration
• WebGUM: Multimodal Web Navigation with Instruction-Finetuned Foundation Models
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
• SAD: Segment Any RGBD

---

Interview

In this weeks interview I had the honour to interview Belgium based artist Marina Ahmadova. Marina catched my eye back in January when she released her now sold out drop called “Alius Orbis” on Foundation. Happy to have had the chance to pick her beautiful brain. Let’s jump in!

What’s your background and how did you get into AI art?

I have had the privilege of pursuing multiple educational paths, having studied monument restoration, architecture, and fine arts. My journey began with a focus on becoming a restorer of architectural landmarks, which eventually led me to delve into the realm of architecture itself. Additionally, I successfully completed my artistic education, further enriching my creative abilities.

Upon graduating from university, I embarked on a teaching career, sharing my knowledge and passion with aspiring minds. However, the majority of my professional journey has been dedicated to the captivating fields of interior design and industrial design. Collaborating with talented colleagues, we have had the opportunity to bring numerous remarkable creations to life.

Should you wish to explore my body of work, a simple search of my name would lead you to my easily accessible portfolio. While I choose not to reveal much about my personal life on Twitter, I do not shy away from sharing my real name and background, allowing others to appreciate the authenticity of my craft.

Design has been an integral part of my life, and I have had the pleasure of immersing myself in various projects. Among them, one of my most cherished endeavors has been the creation of collections of exquisite silk scarves. The intricate blend of artistry and functionality in this particular project holds a special place in my heart, symbolizing the harmony between beauty and practicality.

Do you have a specific project you’re currently working on? What is it?

While I am somewhat hesitant to discuss unrealized projects in advance, fearing that they may not unfold as I envision, I can share my current area of keen interest and the subject matter I have been immersing myself in through extensive reading: myths and fairy tales and their profound influence on our conscious and subconscious minds.

I find great fascination in exploring the intricate interplay between these timeless narratives and the human psyche. Delving into their symbolism, archetypes, and underlying messages, I seek to unravel the profound impact they have on our perception of the world and our own identities.

Furthermore, there is another project close to my heart that I yearn to pursue—a project that combines my passion for embroidery with the realms of Artificial Intelligence and art.

Do you have a favourite myth/fairy tale? Which one and why?

Within the realms of myth and fairy tales, there exists an archetype that holds a profound significance for me - the tale of Joan of Arc. Although not a traditional fairy tale, the enduring power of this remarkable figure has left an indelible mark on my soul. I still recall with vivid clarity the moments spent engrossed in the pages of Mark Twain’s revered book, which shed light on both the virtues and vulnerabilities of this enigmatic heroine. As a child, I was captivated by the story that unfolded, weaving a tapestry of courage, determination, and unwavering conviction. Years later, I had the privilege of visiting Rouen. To me, Joan of Arc is more than just a historical figure or a character from a story. She represents the embodiment of resilience, sacrifice, and the unyielding pursuit of truth.

What drives you to create?

The world around me serves as an inexhaustible wellspring of inspiration, urging me to create in every possible way. As a highly sensitive and emotional individual, I find that this aspect of my nature is both a source of strength and vulnerability. My profound connection to my emotions allows me to perceive and experience life with great intensity, immersing myself in the depths of every moment.

Reflection is an integral part of my being, as I am constantly drawn to introspection and contemplation. I am endlessly fascinated by the interplay between art and life, and all the intricate webs of interconnectedness that exist. It is through this perpetual reflection that I am able to absorb a continuous stream of information, feeding my insatiable curiosity for the world and its myriad forms of expression.

Art, in all its diverse manifestations, holds the key to my heart. In this chaotic and ever-changing world, art serves as my sanctuary.

What does your workflow look like?

I may surprise some with the revelation that my creative process can take various forms. It is a fascinating journey that can unfold in a multitude of ways. I find that the path to achieving the desired result can be as diverse as the inspirations that guide me.

At times, I immerse myself in the creation of a single artwork, investing hours upon hours into meticulous craftsmanship. I meticulously craft each detail, utilizing a range of software and tools, refining and polishing until the vision in my mind is brought to life.

Yet, there are instances where the creative spark ignites instantaneously. With just a few strokes of imagination, I am able to capture the essence of the concept, effortlessly manifesting it into reality. These moments of inspiration can emerge from unexpected sources, evoking a surge of creativity that translates into a finished piece within moments. Ultimately, the key lies not in the complexity or simplicity of the process, but rather in the inspiration and alignment of my inner self with the desired outcome.

What is your favourite prompt when creating art?

Like all AI artists, I cannot fully share the secrets of my creative process, my “prompt,” if you will. However, I can shed light on the principles behind crafting a textual narrative. In most cases, within a collection of images, each one is composed using different prompts, although they may appear cohesive as a single collection. I always incorporate my own touch, adding a unique element that renders my work irreplicable.

One of my earliest and most cherished creations was born from a prompt featuring two girls sitting across from each other. In a moment of inspiration, I added the word cats to the mix. This simple addition transformed the image into one of my absolute favorites, infused with a touch of whimsy and unexpected charm.

How do you imagine AI (art) will be impacting society in the near future?

I believe that AI art is already making an impact here and now. However, I understand that true uniqueness and artistic value still require the touch of a human artist. Not every generated piece can be considered art. For instance, I generate an enormous amount of content, but only a fraction—0.001%—is selected for further exploration and development. The future remains uncertain, and while it is intriguing to delve into philosophical discussions regarding the role of AI in art, we can only speculate. What we do know is that AI art is already an inevitable part of our lives, shaping the present and influencing the artistic landscape.

The fusion of artificial intelligence and creativity has given rise to new possibilities, challenging our understanding of what art is and how it is created. As technology advances, we find ourselves contemplating the future implications and contemplating the potential evolution of AI art. However, in this current moment, AI art exists as a complementary force, enhancing and augmenting human creativity rather than replacing it. It is essential to recognize that the human artist remains an integral part of the artistic process, infusing their unique perspective, emotions, and intentions into each creation.

Who is your favourite artist?

It’s amusing to admit, but I genuinely adore all forms of art and every artist who has contributed their unique touch to the creative tapestry of humanity. I am captivated by every period, every individual who has left their mark. From cave paintings to contemporary Belgian artists, from the art of ancient Egypt to Le Corbusier, from the modernists to Goya, from the American school of painting to the Cretan period — I adored them all. Each era, each style, carries its own essence and narrative, igniting a spark of inspiration within me.

I have a particular fondness for women artists like Paula Rego. I love performance art, minimalism, and land art. The sheer diversity and range of expression within the realm of art continually astound me.

When it comes to the community of AI artists, I hesitate to even begin listing the talented individuals involved. It is an awe-inspiring community, filled with remarkable talents and profound depths of thought.

Anything else you would like to share?

As I conclude, I would like to extend my heartfelt wishes to all artists, including myself, for a greater belief in their own abilities and creative endeavors. This unique journey of AI artists has already etched its place in history, and I am grateful to be a small part of it.

The path we tread as AI artists is one of exploration, innovation, and pushing the boundaries of what is possible. It is a testament to the incredible fusion of human ingenuity and technological advancements. In the face of uncertainty and the ever-evolving landscape of art and technology, it is crucial for us to nurture our confidence, to trust in the depths of our creativity, and to embrace the unique perspective we bring to the artistic realm.

---

Tools & Tutorials

These are some of the most interesting resources I’ve come across this week.

And that my fellow dreamers, concludes yet another AI Art weekly issue. Please consider supporting this newsletter by:

• Sharing it 🙏❤️
• Following me on Twitter: @dreamingtulpa
• Buying me a coffee (I could seriously use it, putting these issues together takes me 8-12 hours every Friday 😅)

Reply to this email if you have any feedback or ideas for this newsletter.

Thanks for reading and talk to you next week!

– dreamingtulpa