The short version: AI image generators don’t draw the way a human does, they start with random static and slowly reshape it, step by step, into something that matches the words in your prompt, guided by patterns learned from hundreds of millions of image-caption pairs. There’s no brush, no layers, no sketch stage. It’s statistics dressed up as art, and once you see that, prompts stop feeling like magic and start feeling like a control panel you can operate.
The three things every image generator needs before it can make anything
Before a single pixel appears, three things have to exist: a huge dataset of images paired with text descriptions, a model trained on that dataset, and a way to turn your typed words into numbers the model can use. Take those away and there’s no picture, no matter how clever the prompt.
The dataset is the part nobody likes to talk about. Stable Diffusion was trained on a slice of LAION-5B, a dataset of roughly 5 billion image-text pairs scraped from the open web, alt text and all. Midjourney and DALL-E 3 have never published their full training sets, which tells you something on its own. These models learned what a “cat wearing a raincoat in the style of a 1970s travel poster” looks like because someone, somewhere, uploaded thousands of raincoats, cats, and 1970s posters to the internet with captions attached, and none of those original creators were asked or paid. That’s not a side note, it’s the whole foundation the industry is built on, and it’s why several artists and Getty Images have taken these companies to court.
The text encoder is the translator. Most modern generators use something built on CLIP (Contrastive Language-Image Pre-training), a model OpenAI released in 2021 that learned to match images with captions by studying 400 million pairs. CLIP doesn’t generate pictures, it just gets extremely good at knowing whether a caption and an image belong together. That matching skill is what lets a generator “understand” that your prompt about a “moody watercolour lighthouse at dusk” should pull toward certain colours, brush textures, and compositions rather than others.
What happens between you hitting enter and the image appearing
This is the part most explanations skip, and it’s the actual mechanism, so here’s the real sequence for a diffusion model, which is what Stable Diffusion, DALL-E 3, Midjourney, and Google’s Imagen are all built on:
- Your prompt gets converted into a numerical representation (an embedding) by the text encoder.
- The system generates a canvas of pure random noise, essentially TV static, in a compressed format called latent space rather than full pixel resolution, which is why this whole process is fast enough to run on a phone app.
- The model looks at that noisy canvas and, guided by your text embedding, predicts what noise needs to be removed to nudge the image slightly closer to matching your words.
- It removes a bit of that predicted noise. The image is still mostly static, but very slightly less so.
- This repeats, typically somewhere between 20 and 50 times depending on the tool and your settings, each pass shaving away a bit more noise and sharpening the shapes.
- Around step 10 to 15, you’d start seeing rough blobs of colour and composition if you watched it live. By step 30 or so, recognisable shapes emerge. The last handful of steps are mostly fine detail: textures, edges, small object correction.
- The final latent image gets decoded back up into a full-resolution picture you can see and download.
That’s it. There’s no drawing tool, no digital pen. It’s denoising, guided by a prompt, done dozens of times in a few seconds. The “creativity” is really the model’s trained sense of probability: given this text, what pixel arrangement is most likely to satisfy someone who typed that?
A real test I ran that shows why prompts behave the way they do
Last year I was building a set of social graphics for a client’s product launch and decided to run the same prompt through three tools back to back to see how differently they’d interpret it. The prompt was: “a woman in her fifties working at a laptop in a bright home office, warm natural light, confident, photorealistic.”
Midjourney gave me something gorgeous and cinematic, but the woman looked closer to 35, styled like a stock photo model, and the laptop screen was blank in a way that felt slightly off. DALL-E 3, working inside ChatGPT, read “fifties” and “confident” more literally and produced someone who looked closer to the right age, but the room lighting was flatter and less flattering. Stable Diffusion, run through a free interface, needed extra prompt engineering (adding “50 years old, natural skin texture, no airbrushing”) before it stopped defaulting to a younger, smoother face.
The lesson wasn’t which tool “won”. It was that every one of them defaults to whatever appeared most often in its training data for that combination of words, and the internet’s training data is heavily skewed toward young, conventionally styled faces when you say “confident woman working”. The model isn’t picturing a person, it’s averaging millions of tagged photos that once carried similar captions. If most of those photos were of younger women, that’s your baseline, and you have to fight the prompt to get away from it. That’s the bit that surprised me most once I understood the mechanics: bias in these images isn’t a bug that gets patched, it’s baked into the maths of what “most likely” means.
Why word order and specific detail change everything
Because the whole system is guided by that text embedding at every single denoising step, small wording changes shift the probability the model is chasing. A few things I’ve confirmed through hundreds of hours of prompting for client work:
- Words placed earlier in the prompt tend to carry more weight in most tools, particularly Midjourney, so lead with the subject, not the mood.
- Vague adjectives like “nice” or “professional” do almost nothing, because they match too many things in the training data to pull the image anywhere specific. “Warm tungsten lighting, shallow depth of field, shot on 50mm” does far more, because those are technical terms tied to a narrower, more consistent set of training images.
- Naming an artist or photographer’s style (a habit that’s caused its own legal headaches) works precisely because that person’s actual body of work was in the training data, tagged with their name.
- Negative prompts (telling the tool what to avoid, which Stable Diffusion supports directly and Midjourney handles with a “–no” parameter) work by pushing the denoising away from certain regions of that probability space, not by “removing” anything after the fact.
Which tool fits which job
After several years of running these tools for client campaigns, here’s the practical breakdown I give people who ask me where to start:
- Midjourney (from about $10 a month for the basic plan) for anything that needs to look striking and stylised fast, moodboards, concept art, social hooks.
- DALL-E 3, built into ChatGPT Plus at $20 a month, for anything that needs to follow instructions precisely, like text inside images or specific product placement.
- Adobe Firefly for commercial work where licensing safety matters, because Adobe trained it on Adobe Stock and licensed content, which means far fewer copyright question marks for client-facing use.
- Stable Diffusion, free and open source, for anyone who wants to run it locally, fine-tune it on their own brand imagery, or avoid subscription fees entirely, at the cost of a steeper learning curve.
If you’re trying to work out which of these fits into your business rather than just being a fun toy for a Tuesday afternoon, that’s exactly the kind of practical decision I help clients with through AI consulting for small businesses, because the tool matters far less than knowing what job you’re trying to solve.
The bit worth sitting with
None of these models “know” what a lighthouse is, or what dusk looks like, in any sense a person would recognise. There’s no understanding happening, only an extremely well-trained pattern of probabilities built from other people’s photographs and paintings, scraped without permission in most cases, compressed into billions of numerical weights. When the output looks stunning, it’s because the training data was rich enough that the average was beautiful. When it looks wrong, hands with six fingers, text that’s gibberish, a face that’s slightly uncanny, that’s the same mechanism failing rather than succeeding, because hands and text are hard to average consistently across millions of photos taken from wildly different angles.
Knowing that changes how you use these tools. You stop expecting intention and start expecting statistics, and you write prompts accordingly: specific, technical, and willing to describe exactly what you want rather than trusting the model to “get” your vision. It never had a vision to begin with.
Frequently asked questions
Do AI image generators copy existing images directly?
No, not in the sense of pasting one photo into a new picture. The model learns statistical patterns from training images and generates new pixel arrangements through denoising, so the output is new. However, because the training data included copyrighted work used without permission or payment, and because a model can sometimes reproduce a distinctive style or composition very closely, several lawsuits are currently testing whether this counts as copyright infringement, including cases brought by Getty Images and a group of visual artists in the US.
Why does my AI-generated text inside images look like gibberish?
Text is one of the hardest things for diffusion models because letters need pixel-perfect precision, and the model is only working with probabilities, not actual character logic. It’s improved a great deal (DALL-E 3 handles short text far better than earlier tools) but anything beyond a few words is still a gamble, and most professionals still add text in Canva or Photoshop after generating the base image.
How many images does it take to train one of these models?
Stable Diffusion’s public training set drew from LAION-5B, roughly 5 billion image-text pairs, though only a filtered subset was used for training. OpenAI and Midjourney haven’t disclosed exact figures for their own datasets, but the scale is understood to be in the hundreds of millions to billions of images.
Can I stop my own photos or artwork from being used to train these models?
Some platforms now offer opt-out tools, Stability AI and LAION added a mechanism through the site “Have I Been Trained” that lets creators flag images for removal from future training runs, and some stock photo sites now let contributors opt out by default. It won’t remove your work from models already trained, and enforcement across the whole industry is patchy at best.
Further reading
Related reading: 6 Frequent Blunders to Avoid When You Add Alt Text to Your Images for Accessibility and AI Images for Business: Tools, Prompts and Strategy That Work.
Want the complete version? Read where I break down AI marketing.
