AI Reimagines 10 Famous Landscape Paintings
AI Reimagines 10 Famous Landscape Paintings
A collaborative effort between GPT-3, Midjourney, and myself.
Each section below comprises a famous landscape painting on the left and Midjourney’s recreation on the right. Above the images, I’ve written GPT-3’s description of the original painting and my translation to a syntactically correct prompt for Midjourney.
All the explanations are at the end. If you have any questions feel free to ask in the comments. I recommend looking at the paintings first. Enjoy!
The Starry Night by Vincent van Gogh (1889)
Description:
The painting is set at night, with a bright moon in the sky. The stars are shining brightly, and the landscape is bathed in a blue light. The trees and houses in the distance are silhouetted against the sky, and the whole scene has a peaceful, dreamlike quality.
Prompt:
oil on canvas painting + post-impressionism + cloudscape + night + bright moon in the sky + shining stars + landscape bathed in a blue light + trees and houses in the distance are silhouetted against the sky + a peaceful dreamlike quality, — ar 5:4.
The Hay Wain by John Constable (1821)
Description:
The painting features a hay wain, or wagon, being pulled by two horses as it crosses a river. The scene is set against a backdrop of mountains, trees, and clouds. The painting is known for its simple and idyllic depiction of rural life in England.
Prompt:
oil on canvas painting + romanticism + landscape + a hay wain pulled by two horses as it crosses a river + a backdrop of mountains, trees, and clouds in the background + simple and idyllic depiction of rural life in England, — ar 30:21.
Impression, Sunrise by Claude Monet (1872)
Description:
The painting is of the sun just peeking over the horizon, with the sky a beautiful orange and red. The water is a deep blue, and there are some small boats in the distance. The brushstrokes are thick and textured, making the painting look almost like a photograph.
Prompt:
oil on canvas painting + impressionism + cityscape + sun over the horizon + beautiful orange and red sky + deep blue water + small boats in the distance + thick and textured brushstrokes, — ar 21:16.
Hunters in the Snow by Pieter Bruegel the Elder (1565)
Description:
The painting is a winter landscape painting that depicts a group of men returning from a hunting trip. The men are tired and their dogs are barking. The snow is coming down and the trees are covered in snow. The men are carrying their dead prey on their shoulders.
Prompt:
oil on panel painting + northern renaissance + genre + winter landscape + a group of men returning from a hunting trip + tired men + barking dogs + snowing + trees are covered in snow + men carry their dead prey on their shoulders, — ar 34:25
The Wanderer Above the Sea Fog by Caspar David Friedrich (1818)
Description:
The painting is a landscape painting that features a man standing on a rocky outcropping with his back to the viewer. The man is looking out over a foggy valley with mountains in the distance. The painting has a somber tone and conveys a sense of loneliness.
Prompt:
oil on canvas painting + romanticism + allegorical painting + landscape painting + a man standing on a rocky outcropping + back view + The man is looking out over a foggy valley + mountains in the distance + somber tone + a sense of loneliness, — ar 63:79.
View of Toledo by El Greco (1596–1600)
Description:
In the painting, the city of Toledo is shown in the distance, with a few buildings and mountains visible. The sky is a deep blue, and the light reflects off of the river in the foreground. The colors are very sharp and bright, making the painting seem almost surreal.
Prompt:
oil on canvas painting + mannerism + late renaissance + cityscape + landscape + city of Toledo in the distance + a few buildings and mountains + deep blue sky + river in the foreground + almost surreal painting, — ar 70:63.
Estaque by André Derain (1905)
Description:
The painting is set in the town of Estaque, near Marseille, France. The town is built on a hill, and the houses are painted in a variety of colors. The rooftops are red, the walls are white, and the doors and windows are blue. The sky is a deep blue, and the sun is shining. The painting is very brightly colored and has a lot of movement.
Prompt:
oil on canvas + fauvism + landscape + town of Estaque, near Marseille, France + the town is built on a hill + houses painted in a variety of colors + red rooftops + white walls + blue doors and windows + deep blue sky + sun shining + very brightly colored painting + a lot of movement, — ar 99:80
Mont Sainte-Victoire by Paul Cézanne (1904–1906)
Description:
In “Mont Sainte-Victoire,” Paul Cézanne depicts a massive mountain perched above the French countryside. The artist has simplified the forms in the painting, distilling the scene down to its essentials. The result is a work that conveys the grandeur and stillness of the mountain. The muted colors and soft light add to the sense of calm and peace in the painting.
Prompt:
oil on canvas painting + post-impressionism + landscape + a massive mountain perched above the French countryside + simplified forms + the grandeur and stillness of the mountain + muted colors + soft light + sense of calm and peace, — ar 5:4
Water Lilies by Claude Monet (1897–1899)
Description:
Water Lilies by Claude Monet is a beautiful painting of lilies in a pond. The lilies are different shades of pink and white, and they are surrounded by green leaves. The water is calm and still, and the sky is reflected in the surface of the pond.
Prompt:
oil on canvas + impressionism + flower painting + lilies in a pond + lilies are shades of pink and white + surrounded by green leaves + water is calm and still + sky is reflected in the surface of the pond, — ar 4:2
The Storm on the Sea of Galilee by Rembrandt (1633)
Description:
In The Storm on the Sea of Galilee, Rembrandt depicts the Biblical story of the disciples in a small boat on the Sea of Galilee being terrified by a storm. The disciples are shown huddled together in the boat, their faces filled with fear. The waves are shown crashing against the boat, and the sky is filled with dark, menacing clouds.
Prompt:
oil on canvas + baroque + religious painting + Biblical story of the disciples in a small boat on the Sea of Galilee + terrified by a storm + disciples huddled together in the boat + faces filled with fear + waves crashing against the boat + sky filled with dark, menacing clouds, — ar 4:5
What is Midjourney?
If you follow my work, you may have noticed that I’ve started to use cover images created with Midjourney. Let me explain a little bit about it before going into the process I followed for this article.
For those of you who don’t know about Midjourney, it’s an independent research lab that came out of stealth a few months ago with the intention of “exploring new mediums of thought [and] expanding the imaginative powers of the human species.”
Midjourney is also the name of the text-to-image model I used to create the images. It started as a closed beta for a very limited number of digital artists and researchers. A couple of weeks ago they opened the beta for anyone to play with it.
It works through Discord. You go to the server and send prompt requests to a bot that then furthers them to the algorithm. Then the bot returns the images to you. As of now, there are 700K+ users. Still, it’s pretty fast. You get four images per prompt in 60 seconds. You can then do variations or upscaling to improve the results or search for something that better fits what you want. In contrast to DALL·E 2, it doesn’t have an impainting feature, which they want to add before long.
Midjourney business model is a tiered subscription system. You can spend a limited number of prompts for free before having to pay the lowest rate which is $10/month. At $30/month you have unlimited prompts. Contrary to the brand new OpenAI’s business model for DALL·E 2, Midjourney avoids limiting your creativity through economic pressure.
They’ve recently updated their algorithm to version 3. The main feature is, as you may have noticed from the images above, that all are notably beautiful. They’ve tweaked the algorithm to do that in purpose. David Holz, Midjourney founder, said recently that the update was only half as good as they wanted and that a bigger update is coming.
But even if it feels like a generic type of beauty — always the same — it has me spellbound. Midjourney’s sensibility with details, colors, and a sort of magical texture is unmatched. In my opinion, not even DALL·E 2 — which is too realistic — can compare with Midjourney in this sense.
A final tip. If you plan to use it, I recommend playing with the “ — stylize” parameter in your prompts. Try 5000–10000 and see what you get. Also, if you pretend to use it more seriously, read the user manual.
Ok, now to the experiment.
How I did the experiment
I’ve been looking for new ideas to explore with AI visual generators for a few weeks now. Because the models are popular now it’s very difficult to come up with something truly original. I’ve seen people combine GPT-3 with DALL·E 2 to create stories, so this isn’t original in that sense.
However, this experiment provides a way to visually contrast the creative mastery of some of the best human painters in history with the ability of these AI models. I created an almost automated end-to-end process with GPT-3 and Midjourney — although I’m in the loop to help them communicate.
I chose famous landscape paintings for three reasons. They’re very recognizable, GPT-3 can easily provide a visual description to design a prompt, and Midjourney excels at painting landscapes.
Once I selected the paintings, I had to prompt GPT-3 to give me the descriptions. I used a generic simple prompt: “Describe with visual details [name of painting] by [name of author] in one paragraph.” I took the first or second completions without editing, even if I knew GPT-3 had missed some elements.
I then adapted the description to a prompt that Midjourney could understand. Midjourney prompts contain elements separated by different symbols: a plus (+), a comma (,), or a double colon (::). Depending on how much you want Midjourney to combine the elements (:: is the strongest break, while the other two are pretty much the same) you may want to choose differently, although the results may surprise you anyway. I used + to combine the concepts.
I added four elements to all prompts apart from GPT-3’s description. In the beginning, I included the medium (e.g. oil on canvas), style (e.g. impressionism), and genre (e.g. landscape). In the end, I added the aspect ratio of the original to make a clearer comparison.
I didn’t prompt the author or the name of the painting to force Midjourney to take only indirect inspiration — as conceived by GPT-3 — from the originals.
I disclosed the prompts for you to try yourself. I’ve realized it’s becoming a common practice among digital artists (and users of text-to-image models in general) to not disclose the prompts. I don’t intend to sell these paintings or gatekeep the skills that are required to create art with AI, so I’ve shared the prompts unedited.
If you want to know more about prompting, I recommend PromptoMANIA as a starting point. If you want to create great prompts without having to learn prompting skills, try this prompt generator hosted on Hugging Face.
If you want to improve beyond that, you’ll have to practice and experiment. I’m not anywhere near the level of some AI artists that have been honing their craft since the beginning of this new AI art scene.
After sending the prompts to Midjourney, it gave me four images to choose from. I upscaled the ones that reminisced me of the original and then chose one.
What I think of the results
As a general note, I have to say that most paintings, although very well painted, are far from recreations of the originals. One reason is that GPT-3 can’t give precise details unless those are very representative of the painting (it tends to be generic).
If I used human-made detailed descriptions of the paintings, I’d have gotten better results. Still, text-to-image models are generally pretty bad at compositionality. That means they have a hard time understanding “X is above Y” or “X is at the right of Y.” They don’t get the relative positions of elements and those are often key distinctive features of the artworks.
Another reason is that I’ve constrained GPT-3 and Midjourney for this experiment. The prompt to GPT-3 is simple and I took the first or second generations. From Midjourney I chose the painting from the first batch of results without editing or impainting.
One key takeaway from this experiment is that both GPT-3 and Midjourney can give you impressive results as long as you don’t intend to guide them to a predefined result. They’re great with generic creations but not with specific ones (DALL·E 2 is better at this).
In summary, I wanted to see how the GPT-3+Midjourney combo — with me in the loop — would reimagine the original human-made artworks. It was a communication experiment between humans and AIs.
As a final reminder: This isn’t an experiment to prove anything about these AIs’ abilities. It’s just to open a door to their otherwise opaque minds.
Very creative experiments! Your article made me wonder whether Midjourney memorizes its training set, i.e. if it can produce these paintings when explicitly prompted with the name and author of the painting. And it does so reasonably, see what it produces for "The Starry Night by Vincent van Gogh": https://media.discordapp.net/attachments/998046753087553546/1002635924175335574/prinnori_The_Starry_Night_by_Vincent_van_Gogh_8fd81eef-1f44-43c9-b1c1-dc411bd22ffc.png
I think what you're uncovering is an interesting property of these models to do something like a soft semantic search for images: you *almost* asked for a particular painting, and it *almost* gave you back that painting.
It'll be interesting to see these models being augmented with explicit retrieval capabilities. People might want to seamlessly switch between retrieving existing images (e.g. stock photos) and creating new ones, especially in the brainstorming phase of a more elaborate design workflow. Similarly to how LaMDA generates language, but also retrieves evidence.