Generative AI and AI Agents Intermediate 12 minutes

Image Generation

From text to image in practice: linking text and images, generation parameters, editing, and ethical considerations.

Your progress in the track0%

The components

  1. The text encoder: turns the prompt into representations.
  2. The diffusion model: generates the image (or its latent representation) conditioned on the text.
  3. The decoder: turns the latent representation into an image.

Control parameters

ParameterEffect
PromptDescribes the content, style, lighting and composition
Negative promptWhat to avoid
Guidance scaleHow strongly to adhere to the text
Number of stepsQuality versus time
SeedReproducibility
DimensionsImage resolution and aspect ratio

An example with the Diffusers library

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
g = torch.Generator("cuda").manual_seed(42)
img = pipe("a watercolor illustration of a library with Arabic calligraphy, soft light",
           negative_prompt="blurry, distorted, text artifacts",
           guidance_scale=7.0, num_inference_steps=30, generator=g).images[0]
img.save("library.png")

Requires a GPU, and the weights (several gigabytes) are downloaded on first use. Check the model's license and terms of use before commercial use. Many models handle English prompts better than Arabic ones, and may not draw Arabic text inside the image correctly.

Editing

  • Image to image (img2img): transforming an existing image according to a prompt.
  • Inpainting (inpainting): regenerating a region specified by a mask.
  • Structural control: guiding generation with edges, pose or depth.

Ethics

  • Copyright: legal questions about training data and imitating the styles of living artists.
  • Deepfakes: misleading images of real people.
  • Bias: stereotyped depictions of professions and groups.
  • Transparency: disclosing generated images, and standards such as C2PA for documenting the source of content.

Lesson terms

CLIP
A model that learns joint representations of images and texts through contrastive learning on image-text pairs.
Content provenance
Documented metadata showing the source of digital content and how it was created and modified.
Image inpainting
Regenerating a specific region of an image while keeping the rest of it.

Abstract

Text-to-image systems combine a text encoder (often inspired by CLIP or a language transformer) with a diffusion model that generates the image conditioned on the text representation. The user controls the prompt, the negative prompt, the guidance scale, the number of steps and the random seed, and the systems allow partial editing and image-to-image transformation. This technology raises issues of copyright, deepfakes and bias, and watermarks and content provenance standards are used to limit misuse.

Review questions

  1. What is the role of the random seed (seed) in image generation?

    It determines the initial noise, so fixing it along with the other settings reproduces almost the same image.

  2. Why is the negative prompt used?

    To steer generation away from unwanted elements such as distortion, text or poor quality.

Sources and references

  1. Radford et al. (2021) Learning Transferable Visual Models From Natural Language Supervision (CLIP) — arXiv
  2. Ramesh et al. (2022) Hierarchical Text-Conditional Image Generation with CLIP Latents — arXiv
  3. Hugging Face — Diffusers documentation
  4. C2PA — Coalition for Content Provenance and Authenticity

Complete the lesson