Diffusion Models Explained: How Stable Diffusion and Sora Generate Images and Video

Diffusion models are generative AI models that learn to create new data by reversing a gradual process of adding noise to training examples. Once trained, they start from pure random noise and transform it, step by step, into a meaningful output such as a photograph-like image or a video clip.

Two of the best-known systems built on this idea are Stable Diffusion, a family of openly available text-to-image generation models, and Sora, OpenAI’s text-to-video generation model. Both create visual content using diffusion-based techniques, but their architectures and goals differ.

This guide explains how diffusion models work and how the same principle extends from generating single images to generating video that moves coherently over time.

Interactive tool · Generative AI roadmap

Your roadmap to becoming a Generative AI Engineer

See what you already know, what to learn next and how long the switch to a GenAI engineering role takes. Tap any step for what to study and build.

✦ Personalise your roadmap3 quick questions · takes 30 seconds

Your name optional Current role Select your role Student / Fresher Software Engineer (Backend) Software Engineer (Frontend) Full-stack Engineer Data Analyst Data Scientist Machine Learning Engineer DevOps / Cloud Engineer QA / Test Engineer Solutions / Sales Engineer Technical Consultant Product Manager Non-technical role Years of experience Select experience 0–1 years (fresher) 1–3 years 3–5 years 5–8 years 8+ years Target companies optional

AI familiarity

NoneHaven't used AI tools for work BeginnerUse ChatGPT-style tools IntermediateBuilt side projects with LLM APIs AdvancedShipped AI features to production

Tech stack optional · pre-filled for your role

Choose your role to pre-fill a typical stack.

Python Java JavaScript / TypeScript Go C++ SQL React / Vue / Angular Node.js Django / Flask / FastAPI Spring AWS GCP Azure Docker Kubernetes Machine Learning / scikit-learn PyTorch / TensorFlow LLM APIs (OpenAI, Claude, Gemini) LangChain / LlamaIndex Vector databases No coding background + Edit tech stack

Generate my roadmap →

What's inside8 stages · 30 steps to job-ready

1. Python & engineering foundations
2. ML & deep learning essentials
3. How LLMs work
4. Prompting & building LLM apps
5. RAG & model customisation
6. Agents & tool use
7. Production, evaluation & safety
8. Portfolio & job readiness

Browse the full roadmap without personalising

Learn this Level up You have it Done

Show only what I need

Stage 1 · Start here

Python & engineering foundations

Python for AI work ~3 wks

Data handling with SQL & pandas ~2 wks

Git, APIs & JSON ~1 wk

Maths you actually need ~2 wks

Stage 2 · Core concepts

ML & deep learning essentials

Machine learning basics ~2 wks

NLP & embeddings ~2 wks

Neural networks with PyTorch ~3 wks

Stage 3 · Understand the engine

How LLMs work

Transformer architecture ~2 wks

Hugging Face & open models ~2 wks

Models, tokens & context windows ~1 wk

Small language models optional ~1 wk

Stage 4 · Start shipping

Prompting & building LLM apps

Prompt engineering ~1 wk

LLM apps with APIs ~2 wks

Structured outputs & function calling ~1 wk

LangChain & LlamaIndex ~1 wk

Stage 5 · Make models useful

RAG & model customisation

Embeddings & vector databases ~2 wks

Fine-tuning & PEFT (LoRA) ~2 wks

Multimodal & vision models optional ~2 wks

Retrieval-augmented generation ~2 wks

Preference tuning (RLHF, DPO) optional ~2 wks

Stage 6 · Automate workflows

Agents & tool use

Agent design patterns ~2 wks

Tools, MCP & system integration ~2 wks

Orchestration with LangGraph ~2 wks

Stage 7 · Make it reliable

Production, evaluation & safety

Containers & deployment ~2 wks

Evaluation (evals) ~2 wks

Monitoring, cost & latency ~1 wk

Cloud AI services ~2 wks

Guardrails & responsible AI ~1 wk

Stage 8 · Get hired

Portfolio & job readiness

Interview preparation ~1 wk

Capstone project & portfolio ~3 wks

Show the full roadmap (8 stages · 30 steps)

GenAI glossary 17 terms

LLM Transformer Token Context window Prompt engineering Embeddings Vector database RAG Fine-tuning LoRA Agent Tool calling MCP Evals Guardrails Hallucination Prompt injection

Tap a term to see what it means. You will hear these in every GenAI interview.

LLM

Large language model: a neural network trained on huge amounts of text to predict and generate language.

Transformer

The neural network architecture behind modern LLMs, built around the attention mechanism.

Token

A chunk of text (often part of a word) that a model reads and writes. Pricing and limits are counted in tokens.

Context window

The maximum number of tokens a model can consider at once, including your prompt and its answer.

Prompt engineering

Designing instructions and examples so a model reliably produces the output you need.

Embeddings

Numeric vectors that capture meaning, so similar texts end up close together.

Vector database

A database built to store embeddings and find the most similar ones quickly.

RAG

Retrieval-augmented generation: fetching relevant documents and giving them to the model so answers are grounded.

Fine-tuning

Further training a pre-trained model on your own data to specialise it.

LoRA

Low-rank adaptation: a parameter-efficient fine-tuning method that trains small add-on weights instead of the whole model.

Agent

An LLM system that plans steps, calls tools and acts towards a goal, often over several turns.

Tool calling

Letting a model call functions or APIs, such as search or a database query, with structured arguments.

MCP

Model Context Protocol: an open standard for connecting AI applications to tools and data sources.

Evals

Systematic tests that measure how well an AI system performs on representative cases.

Guardrails

Checks on inputs and outputs that keep an AI system safe, on-topic and compliant.

Hallucination

When a model states something false or unsupported with confidence.

Prompt injection

An attack where malicious text tricks a model into ignoring its instructions.

IIT Delhi CEP × Varsity

Close your gaps with a structured programme

The Certificate Programme in Generative AI from CEP, IIT Delhi covers transformers, fine-tuning, RAG, agents and responsible AI through live weekend classes and hands-on projects.

• 6 months
• Live online
• Weekend classes

Enquire about the programme View programme details

How do Diffusion Models Work?

Diffusion models learn to reverse a process that gradually adds noise to data. To generate something, they start with random noise and repeatedly remove the noise they predict until a recognisable output appears. The five steps below cover training first, then generation.

Step 1 : Understand the Forward Diffusion Process

The forward diffusion process adds a small amount of random noise to a training image, then repeats this many times until the image is almost indistinguishable from static. The model learns from these noisy versions paired with the clean original.

This forward process is only used in training. It is not how the final image is created during generation.

Step 2 : Train the Model to Predict Noise

Next, the model receives a noisy image and a timestep, a number showing how much noise was added. It learns noise prediction: estimating the exact noise that was mixed into the original.

Image diffusion models commonly use a U-Net, a neural network that captures both overall layout and fine detail. Training adjusts its parameters to shrink the gap between predicted and actual noise.

Step 3 : Start With Random Noise

Generation begins with a randomly sampled noise pattern that looks nothing like a picture. The trained model estimates which parts of it should be removed, using patterns learned during training.

In text-conditioned diffusion models, a text prompt steers each of these estimates towards content that matches the description.

Step 4 : Gradually Denoise the Image

The denoising process repeats a simple loop: predict the noise, remove part of it and feed the cleaner result back in. Early steps set the composition, while later steps add texture and detail.

Each pass is called a sampling step. More steps usually improve the result but slow generation down. The chosen sampler also affects speed and quality.

Step 5 : Produce the Final Output

After the final step, the result is converted into a viewable image. In latent diffusion models, a decoder turns the compressed result back into full-resolution pixels.

The image is a new output built from learned patterns, not a training picture pulled from storage. However, models can occasionally reproduce training images that appeared many times in their data.

Key point: Diffusion models generate outputs by learning to reverse noise, not by directly drawing an image from a text prompt in one step.

What is the Mathematics Behind Diffusion Models?

Diffusion models combine probability distributions, which describe how likely different values are, with iterative denoising, which improves a sample a little at a time. Three beginner-friendly concepts explain most of the mathematics, and none of them needs advanced calculus.

What is Gaussian Noise?

Gaussian noise is a set of random values drawn from a normal distribution, the familiar bell curve, where most values sit near zero. One such value is added to each pixel, or to each number in a latent representation.

It is popular because adding Gaussian noise to Gaussian noise still gives Gaussian noise. That means training can jump straight to any noise level in one calculation.

What is the Noise Prediction Objective?

The noise prediction objective measures how well the model identifies the noise inside a corrupted sample. The original Denoising Diffusion Probabilistic Models (DDPM) paper showed that this simple target produces high-quality images. A noisy sample and the training loss are written as:

x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε

Loss = ‖ε − ε_θ(x_t, t)‖²

Each symbol has a simple meaning, listed below. Read together, the formulas say: mix noise into an image, ask the model to guess that noise, then penalise the error.

• x_0: the clean image or latent representation.
• ε (epsilon): the Gaussian noise actually added.
• t: the timestep, which sets the noise level.
• ᾱ_t (alpha-bar): a value between 0 and 1 that shrinks as t grows, so later timesteps keep less of the original image.
• x_t: the resulting noisy sample.
• ε_θ(x_t, t): the model’s predicted noise, where θ (theta) stands for its learnable parameters.
• ‖…‖²: the squared error between actual and predicted noise.

What are Timesteps in Diffusion Models?

Timesteps number the stages of adding noise, from t = 0 (clean) to a final step T (almost pure noise). The original DDPM setup used T = 1,000. Because the timestep is a model input, one network learns to handle every noise level.

Generation runs from high noise to low noise. High-noise steps decide the layout, while low-noise steps refine edges, lighting and texture.

Key point: The model learns a denoising function that works across different noise levels, allowing it to generate new outputs from random noise.

How does Stable Diffusion Generate Images?

Stable Diffusion is a family of latent diffusion models. It runs much of the diffusion process in a compressed latent space rather than on full-resolution pixels. Created by CompVis with Stability AI and Runway, it builds on latent diffusion research by Rombach et al.

What is Latent Diffusion?

An encoder compresses each image into a smaller latent representation, and diffusion takes place in that compressed space. Stable Diffusion v1 shrinks images by a factor of 8, turning a 512×512 RGB image (786,432 values) into a 64×64×4 latent (16,384 values).

That roughly 48-fold reduction matters, because pixel-space diffusion models often take hundreds of GPU days to train.

How does Text Conditioning Work?

A text encoder converts the prompt into text embeddings, which are numbers that capture meaning. Stable Diffusion v1 uses a frozen (pretrained, never updated) CLIP ViT-L/14 text encoder. Cross-attention then lets the denoising network check these embeddings at every step.

For “a red bicycle in a rainy street”, the parts of the image forming the bicycle draw on “red” and “bicycle”. The background draws on “rainy street”, which steers the model towards wet surfaces and reflections.

What Role do the U-Net and VAE Play?

In common Stable Diffusion architectures, the U-Net does the denoising. It takes the noisy latent, the timestep and the text embeddings, then predicts the noise to remove.

The Variational Autoencoder (VAE) has two halves. Its encoder compresses training images into latents, and its decoder turns the final latent into pixels. Text-to-image generation needs only the decoder. Newer versions differ: SDXL uses a U-Net about three times larger, while Stable Diffusion 3.5 uses transformer-based MMDiT models.

How does Stable Diffusion Turn a Prompt into an Image?

The full pipeline combines the text encoder, the denoising network and the VAE. These five steps show the order in which each part acts during text-to-image generation, from the written prompt to the finished image:

1. Convert the prompt into text embeddings.
2. Initialise a latent representation with random noise.
3. Use the denoising network and text conditioning to refine the latent representation.
4. Repeat the denoising process over multiple sampling steps.
5. Decode the final latent representation into an image.

Key point: Stable Diffusion uses latent-space denoising and text conditioning to transform random noise into an image guided by a user’s prompt.

How does Sora Generate Videos Using Diffusion?

Sora is OpenAI’s video generation model, first previewed in February 2024. Generating video involves more than creating individual images, because the model must also learn movement, timing and consistency across frames.

OpenAI shut down the Sora app on April 26, 2026 and the Sora API on September 24, 2026. Its technical report is still a key public reference for diffusion-based video generation.

From Text Prompts to Video Representations

A video prompt sets the subjects, actions, setting, camera and visual style. According to OpenAI’s Sora technical report, Sora was trained on highly descriptive captions, and GPT expanded short prompts into detailed ones.

The model works with both spatial and temporal information, because an action such as a bicycle turning a corner only exists across several frames.

What are Spacetime Patches?

Patches are small units of visual data, similar to tokens in a language model. Sora compressed video in both space and time, then split it into spacetime patches covering image regions across time.

As a diffusion transformer, Sora learned to predict clean patches from noisy ones. Patches let it train on videos of different lengths, resolutions and aspect ratios, with images treated as one-frame videos.

How does Sora Generate Motion and Maintain Consistency?

Sora learned movement patterns from large amounts of video. OpenAI said these abilities emerged from scale rather than from built-in 3D rules. Because Sora denoised the patches for the whole clip together, it could keep track of motion, object identity, camera movement and scene continuity at once.

This is still hard. OpenAI said Sora could model short- and long-range dependencies “often, though not always”, and long clips could lose coherence.

How is Video Generation Different from Image Generation?

Image generation deals with spatial relationships within one frame. Video generation must also show how those relationships change over time.

If frames were generated independently, a red bicycle could change colour or position at random between frames. A video model therefore has to produce coherent motion across the whole clip.

Key point: Sora extended generative visual modelling to video by learning spatial and temporal patterns, rather than treating each frame as an unrelated image.

Stable Diffusion vs Sora: What Is the Difference?

Stable Diffusion and Sora are both generative AI models that create visual content using diffusion-based techniques. They target different output formats and face different challenges, though. Stable Diffusion produces still images, while Sora had to model change over time.

Key Differences Between Stable Diffusion and Sora

The table compares both systems across seven parameters, from purpose to typical use cases. It reflects publicly documented capabilities, which vary by version. OpenAI never disclosed Sora’s full architecture, so its column describes the approach at a high level.

Parameter	Stable Diffusion	Sora
Primary purpose	Image generation	Video generation
Input	Text prompts and supported image inputs	Text and supported visual inputs, depending on version
Output	Images	Video clips
Core approach	Latent diffusion	Diffusion-based video generation
Main challenge	Image quality and prompt alignment	Motion, temporal consistency, and scene continuity
Visual information	Primarily spatial	Spatial and temporal
Typical use cases	Artwork, design, illustrations, image editing	Video creation, visual storytelling, and scene generation

Why Video Generation is More Complex than Image Generation

A 10-second clip at 24 frames per second contains 240 frames, and each one must agree with its neighbours. Physical interactions make this harder. OpenAI noted that earlier video models would “morph objects and deform reality”, for example by having a missed basketball shot teleport into the hoop.

Small errors that go unnoticed in a single frame can become visible flicker during playback.

Key point: Stable Diffusion and Sora share diffusion-based generative principles, but video generation introduces additional temporal modelling challenges.

What are the Applications of Diffusion Models?

Diffusion models in AI support creative, commercial, scientific and technical work. The same denoising approach that creates images from scratch can also edit pictures, extend videos and generate synthetic data. Four application areas stand out.

Image Generation and Digital Art

Text-to-image generation helps artists, designers and marketers create illustrations, artwork, concept designs and visual assets from prompts. Quick iteration makes it easy to experiment with styles, and many teams treat outputs as drafts for human refinement.

Video Generation and Content Creation

Text-to-video generation can produce short clips for storyboarding, pre-visualisation and testing concepts. OpenAI showed Sora animating still images, extending videos and restyling footage. In advertising and filmmaking, this may speed up early concept work.

Image Editing and Restoration

Inpainting replaces a selected area, such as an unwanted object, while outpainting extends an image beyond its borders. Super-resolution adds plausible detail when enlarging images. The latent diffusion paper reported state-of-the-art inpainting and highly competitive super-resolution results.

Scientific and Medical Research

Researchers use diffusion models for synthetic data and scientific imaging. One team trained latent diffusion models on 31,740 brain MRI scans from UK Biobank and released 100,000 synthetic images.

Such data can support research and model development. However, generated medical data needs validation and cannot replace real clinical evidence.

Key point: Diffusion models are useful beyond generating creative images; they can also support editing, restoration, simulation, and research workflows.

What are the Limitations of Diffusion Models?

Diffusion models still face practical challenges that affect quality, reliability and cost. How serious each one is depends on the model, version and use case. Five limitations matter most for anyone using diffusion models in real projects.

High Computational Requirements

Training and high-resolution generation need a lot of GPU power. Longer, higher-resolution videos add more patches to process at every step. Optimisations at generation time, such as lower precision, quantisation and samplers that need fewer steps, help. Even so, Stable Diffusion v1 needed at least 10 GB of VRAM.

Slow Generation Times

Traditional diffusion models run many denoising steps, one after another. The consistency models paper notes this typically needs 10–2,000 times more compute than models that generate in a single step. Better samplers reduce the wait, though faster generation can cost detail or prompt accuracy.

Inaccurate Text and Object Relationships

Models often get precise instructions wrong, producing four apples instead of five or putting a cat on the wrong side of a dog. Complex prompts with many objects make these errors more likely. Simplifying prompts, generating several candidates and checking outputs against the request all help.

Motion and Physical Consistency Challenges

Video models can produce unnatural movement or objects that change shape mid-clip. OpenAI said Sora could not accurately model glass shattering, and that even Sora 2 “makes plenty of mistakes”. Consistency over time is therefore still a key test for any video model.

Bias, Copyright, and Responsible Use

Training data shapes outputs. The Stable Diffusion v1 repository warns that the model “mirrors biases and (mis-)conceptions” in its data. Copyright, consent, deepfakes and representation also raise concerns. Safeguards include safety filters, watermarking and consent controls over a person’s likeness.

Key point: Diffusion models can create impressive visual outputs, but computational cost, prompt accuracy, consistency, and responsible use remain important challenges.

What is the Future of Diffusion Models?

Research is making diffusion models faster, better, easier to control and more efficient. Some of these techniques are already widely used, while others are still active research with no fixed timeline for wide adoption.

Faster and More Efficient Sampling

Denoising Diffusion Implicit Models (DDIM) keep standard training but sample 10–50 times faster. Distillation trains a fast “student” model to match a slower “teacher”, a form of teacher-student knowledge distillation.

Progressive distillation cut samplers from up to 8,192 steps to 4, and consistency models support one-step generation.

Better Control Over Generated Images and Videos

Newer models follow prompts more closely and give more control over where things appear. Inputs beyond text, such as reference images, editing instructions, depth maps and pose outlines, allow finer control. OpenAI said Sora 2 could follow instructions spanning several shots while keeping the scene consistent.

Unified Image, Video, and Multimodal Generation

Interest is growing in models that work across several types of media. Sora was trained on images and videos together, and Sora 2 generated synchronised dialogue and sound effects. Combining text, images, video and audio could allow more flexible creative workflows, though these abilities are still developing.

Key point: Future improvements are likely to focus on making diffusion-based generation faster, more controllable, and more consistent across visual formats.

Conclusion

Diffusion models turn random noise into images and video by learning, step by step, which noise to remove. Stable Diffusion made this efficient for still images by working in latent space and steering generation with text. Sora showed the same principle stretching across time, where every frame must agree with the next.

The common thread is control: denoising creates the content, conditioning shapes it and temporal consistency makes video believable. To build hands-on skills with generative models, explore the IIT Delhi CEP Certificate Programme in Generative AI, offered by IIT Delhi’s Continuing Education Programme in collaboration with InterviewBit by Scaler (Varsity).

Frequently Asked Questions

What is the difference between diffusion models and autoregressive image generation models?

Autoregressive models build an image piece by piece, predicting one token at a time, much as a language model writes text. Diffusion models refine the whole image at once over several denoising steps. Each approach balances speed, quality and control differently.

How much GPU memory is needed to run Stable Diffusion locally?

It depends on the model, resolution, precision and optimisations. The original Stable Diffusion v1 repository listed at least 10 GB of VRAM. Stability AI states that Stable Diffusion 3.5 Medium needs 9.9 GB, not counting text encoders. Half precision and quantisation can lower requirements.

Can Stable Diffusion generate high-resolution images without losing detail?

Yes, within limits. Models work best near the resolution they were trained on. Stable Diffusion v1 was fine-tuned on 512×512 images, and pushing far beyond that can repeat subjects. SDXL trains on multiple aspect ratios with a refinement model, and many workflows upscale after generation.

What is classifier-free guidance in diffusion models, and how does it affect image quality?

Classifier-free guidance combines predictions made with and without the prompt to push each step towards the prompt. A guidance scale sets the strength, and the original Stable Diffusion scripts default to 7.5. Higher values follow the prompt more closely but reduce variety, and extreme values can oversaturate images.

How do sampling schedulers such as DDIM and Euler affect diffusion model outputs?

Schedulers decide how much noise to remove at each step. DDIM can follow a fixed, non-random path, which gives good images in fewer steps and the same result every time for a given seed. Euler-style samplers solve the denoising equation in small steps. Each scheduler can change sharpness and the number of steps needed.

What is the difference between image-to-image and text-to-image diffusion?

Text-to-image diffusion starts from pure random noise guided by a prompt. Image-to-image diffusion starts from an existing image, adds a controlled amount of noise, then denoises it with a prompt. A strength setting controls how much of the original survives.

How can users maintain character consistency across AI-generated images?

Reusing seeds, settings and detailed character descriptions helps but rarely guarantees consistency. Stronger options include reference-image conditioning, image-to-image editing, LoRA adapters and fine-tuning methods such as DreamBooth, which personalises a model using only 3–5 images of a subject.

Why do AI-generated images sometimes contain distorted hands, faces, or text?

Hands have many joints and poses, and they often appear small or partly hidden in training images, so models learn them less reliably. Small faces lose detail when images are compressed into latents. Text needs exact letter shapes and spelling. Newer models handle typography better, but mistakes still happen.

Can diffusion models be fine-tuned to generate images in a specific artistic style?

Yes. DreamBooth, textual inversion and LoRA adapt a pretrained model using a small set of examples. LoRA trains small add-on weights instead of the full network, as explained in this guide to how LoRA and other fine-tuning methods update a pretrained model. Only use images you have the rights to use.

How do developers evaluate the quality of images and videos generated by diffusion models?

Developers combine automated metrics with human review. Fréchet Inception Distance (FID) compares generated and real images, where lower is better; DDPM reported 3.17 on CIFAR-10. CLIP-based scores measure how well an image matches its prompt. Video evaluation also checks consistency over time, motion quality and physical plausibility.

What is the difference between open-source diffusion models and proprietary video generation models?

Open-weight models such as Stable Diffusion can be downloaded, run locally and fine-tuned under their licence terms. Proprietary models such as Sora were only available through the provider’s apps or API, and their architecture was not disclosed. Access can also end, as it did when OpenAI shut down Sora in 2026.

How are diffusion models used to create synthetic training data for machine learning?

Diffusion models can create extra examples for rare classes, unusual conditions or privacy-sensitive fields such as medical imaging. The brain MRI study above set age, sex and brain volumes as inputs to control its outputs. Synthetic data must be checked for realism and flaws, and models trained on it should still be tested on real data.

Sources

Denoising Diffusion Probabilistic Models – Ho, Jain & Abbeel (2020)

Stable Diffusion: Official CompVis Repository

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis – Podell et al. (ICLR 2024)

Introducing Stable Diffusion 3.5 Stability AI

Classifier-Free Diffusion Guidance Ho & Salimans (2022)

Denoising Diffusion Implicit Models Song, Meng & Ermon (ICLR 2021)

Progressive Distillation for Fast Sampling of Diffusion Models Salimans & Ho (ICLR 2022)

Consistency Models Song, Dhariwal, Chen & Sutskever (ICML 2023)