Three different things under one name
Behind the words reference image hide three sharply different mechanisms. One carries over the style: palette, lighting, brushwork, filmic character. Another carries the content: one specific face, one specific product. The third carries the structure: what sits where in the frame, how a body is posed, where the edges run.
Most of the disappointment comes from expecting one type to do another one's job. A style reference will not carry over your character; a character reference will not adopt the composition; a structural reference brings no palette with it. On a tool's interface this often is not visible, because all three show up under the same "upload an image".
Style reference
This is the most widespread and the least risky one. You upload an image whose mood you want, and the model takes the palette, the light handling and the pictorial character from it. Meanwhile the prompt is free to say something else: a different subject, a different place.
What to know about it: it specifically does not copy over the objects and the people — one provider spells that out in its documentation. And it has an awkward side effect that has a name of its own in the literature: content leakage. If there is a red sofa in the style reference, something red and large tends to show up in the generated images, even when you did not ask for it. The more distinctive the content of the reference, the stronger this gets.
Character reference
Here the point is not the mood but the specific thing: the same face, the same product, the same coat. Today's models typically accept several references rather than one, and handle characters, objects and style separately — the limits differ by provider, but the order of magnitude is similar: a few characters, a few objects, a few style images at once.
Two practical tips. One is that the reference image should have a clean, featureless background — the fewer distracting elements in it, the less leaks out of it. The other is that the reach of a character reference is finite: with several characters the identities start to blur together. On that, and on when training is the better investment, character and product consistency goes into it thoroughly.
Structural reference
This is the least known one and yet the most precise. You are not handing over the content of the image but its skeleton: the outlines, the depth layers, a figure's pose, or simply a hand-drawn box layout. The model follows that structure and builds the surface from the prompt.
Which is why this method is the answer when you know exactly what should go where. Layout described in words is the weakest ability of today's models; a sketch, by contrast, does not ask for interpretation but for compliance. For the details of the method and the measurement data, controlling composition is the place to go.
The weight you can set
On most tools the strength of the reference is adjustable. Somewhere it is a slider, somewhere a number at the end of the prompt — the point is the same: how tightly should the model bind itself to the reference, and how much freedom should it leave the prompt.
Two extremes are worth knowing. At low weight the reference is only a faint direction: the mood is recognisable, but the model works freely. At high weight the reference is essentially copied and the prompt barely counts — what you typically get then is an image very close to the reference, which is rarely what you wanted. The usable range is almost always in between, and it is worth running the same prompt at two or three weights before you decide it does not work.
What not to mix
Four pairings that ruin the result on a regular basis.
- A strong style reference plus a detailed style description in the prompt. The two compete; usually the reference wins, and the style part of the prompt just dilutes the whole thing for nothing.
- The same image as both style and character reference. The model cannot tell what to copy and what to drop — leakage goes to maximum.
- Many references at once with contradictory content. Near the upper limit the result typically gets worse, not better. Fewer but cleaner references work better.
- A structural reference plus a "be creative" prompt. The skeleton is there precisely so that there is no freedom; ask for freedom as well and the two cancel each other out.
Common misconceptions
- "A reference image is like an example inside the prompt." It is not. The text of a prompt is interpreted by the model; a reference image is processed by a separate mechanism, at a different weight and with a different effect.
- "A higher-resolution reference will be more accurate." References are usually scaled down before processing anyway. Cleanliness is what counts, not resolution.
- "Whatever I set once stays set at the provider." Reference-handling features come and go: there was a tool where a new version withdrew the earlier character and object references and replaced them with something else. Worth re-checking once a year what the current state is.
- "A reference image is legally neutral." Not necessarily. Working from another artist's style reference is not purely a question of taste — it is worth deliberately building your reference set out of your own material.
What to take away
Before you attach an image, ask the question: what do you want to carry over from it — a mood, a specific thing, or a structure? The answer decides which reference type you have to use, and once that is settled, most of the "it is not doing what I ask" complaints disappear on their own.