Image Generation
From text to image in practice: linking text and images, generation parameters, editing, and ethical considerations.
The components
- The text encoder: turns the prompt into representations.
- The diffusion model: generates the image (or its latent representation) conditioned on the text.
- The decoder: turns the latent representation into an image.
Control parameters
| Parameter | Effect |
|---|---|
| Prompt | Describes the content, style, lighting and composition |
| Negative prompt | What to avoid |
| Guidance scale | How strongly to adhere to the text |
| Number of steps | Quality versus time |
| Seed | Reproducibility |
| Dimensions | Image resolution and aspect ratio |
An example with the Diffusers library
import torch
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
g = torch.Generator("cuda").manual_seed(42)
img = pipe("a watercolor illustration of a library with Arabic calligraphy, soft light",
negative_prompt="blurry, distorted, text artifacts",
guidance_scale=7.0, num_inference_steps=30, generator=g).images[0]
img.save("library.png")Requires a GPU, and the weights (several gigabytes) are downloaded on first use. Check the model's license and terms of use before commercial use. Many models handle English prompts better than Arabic ones, and may not draw Arabic text inside the image correctly.
Editing
- Image to image (img2img): transforming an existing image according to a prompt.
- Inpainting (inpainting): regenerating a region specified by a mask.
- Structural control: guiding generation with edges, pose or depth.
Ethics
- Copyright: legal questions about training data and imitating the styles of living artists.
- Deepfakes: misleading images of real people.
- Bias: stereotyped depictions of professions and groups.
- Transparency: disclosing generated images, and standards such as C2PA for documenting the source of content.
Lesson terms
- CLIP
- A model that learns joint representations of images and texts through contrastive learning on image-text pairs.
- Content provenance
- Documented metadata showing the source of digital content and how it was created and modified.
- Image inpainting
- Regenerating a specific region of an image while keeping the rest of it.
Abstract
Text-to-image systems combine a text encoder (often inspired by CLIP or a language transformer) with a diffusion model that generates the image conditioned on the text representation. The user controls the prompt, the negative prompt, the guidance scale, the number of steps and the random seed, and the systems allow partial editing and image-to-image transformation. This technology raises issues of copyright, deepfakes and bias, and watermarks and content provenance standards are used to limit misuse.
Review questions
-
What is the role of the random seed (seed) in image generation?
It determines the initial noise, so fixing it along with the other settings reproduces almost the same image.
-
Why is the negative prompt used?
To steer generation away from unwanted elements such as distortion, text or poor quality.
Sources and references
Complete the lesson
You have completed all the lessons in the track! You can now request the track completion certificate.
Request the certificate