Self-Attention vs Cross-Attention: A Worked Example
Key Insights
- Self-attention allows tokens to exchange information within the same sequence, creating context-aware representations.
- Cross-attention connects two sequences by using queries from one sequence and keys and values from another.
- Both mechanisms use scaled dot-product attention: query-key scores are scaled by the square root of the key dimension, passed through softmax, and used to combine value vectors.
- Multi-head attention runs several attention operations in parallel, allowing Transformers to learn different relationships between tokens.
- Masked self-attention uses a causal attention mask to prevent autoregressive decoders from accessing future positions during generation.
- In encoder-decoder Transformers, cross-attention allows decoder states to retrieve relevant information from encoder outputs; related mechanisms can also connect modalities in multimodal models.
- The attention score matrix has dimensions
for self-attention and
for cross-attention, where
and
are the query and key sequence lengths.
Self-attention is a neural network mechanism that allows every token in a sequence to attend to other tokens in the same sequence to understand their meaning and context. Cross-attention, in contrast, allows tokens in one sequence to retrieve relevant information from another. Both use queries (Q), keys (K) and values (V), but the key difference lies in where these vectors originate.
Understanding the self-attention vs cross-attention difference is essential to understanding how Transformers build contextual representations and connect information across sequences. This guide walks through a worked numerical example of the self-attention mechanism, applies the same calculation to cross-attention, and compares their inputs, calculations and practical uses.
Your roadmap to becoming a Generative AI Engineer
See what you already know, what to learn next and how long the switch to a GenAI engineering role takes. Tap any step for what to study and build.
What's inside8 stages · 30 steps to job-ready
- Python & engineering foundations
- ML & deep learning essentials
- How LLMs work
- Prompting & building LLM apps
- RAG & model customisation
- Agents & tool use
- Production, evaluation & safety
- Portfolio & job readiness
Python & engineering foundations
What to study
- Core Python: data types, functions, classes, modules
- Virtual environments and package management (pip, uv)
- Writing clean, testable scripts
What to build
A command-line script that reads a CSV, cleans it and writes a summary report.
What to study
- SQL joins, aggregations and window functions
- pandas for cleaning and reshaping data
- Working with text data at scale
What to build
A notebook that cleans a messy text dataset and prepares it for a model.
What to study
- Git branching and pull requests
- How REST APIs work: HTTP methods, status codes, auth headers
- Reading and writing JSON
What to build
Call a public API from Python, handle errors and save the results as JSON.
What to study
- Vectors, matrices and dot products
- Probability and distributions
- Gradients and optimisation, intuitively
What to build
Implement cosine similarity and gradient descent from scratch in NumPy.
ML & deep learning essentials
What to study
- Supervised vs unsupervised learning
- Train/test splits, overfitting and evaluation metrics
- scikit-learn workflows
What to build
Train and evaluate a text classifier with scikit-learn.
What to study
- Tokenisation and text preprocessing
- Word embeddings and semantic similarity
- Sequence models (RNN, LSTM) and why transformers replaced them
What to build
Build a semantic search over 500 short texts using sentence embeddings.
What to study
- Layers, activations, loss functions and backpropagation
- Training loops, batching and GPUs
- PyTorch tensors and autograd
What to build
Train a fully connected network on a small dataset and plot its learning curve.
How LLMs work
What to study
- Self-attention and multi-head attention
- Encoder, decoder and encoder-decoder models
- Positional encoding and why scale matters
What to build
Walk through a minimal transformer implementation and explain each block in a short write-up.
What to study
- The Transformers library, model hub and datasets
- Running open-weights models locally
- Fine-tuning a pre-trained model for a task
What to build
Fine-tune a small pre-trained transformer for sentiment or topic classification.
What to study
- Closed vs open-weights model families
- Tokens, context windows, temperature and sampling
- Comparing models on cost, latency and quality
What to build
A comparison table of three models answering the same 20 prompts, with cost and latency.
What to study
- Quantisation, pruning and distillation
- When a small model beats a large one
- On-device and low-cost inference
What to build
Quantise an open model and compare its speed and quality against the original.
Prompting & building LLM apps
What to study
- System prompts, few-shot examples and chain-of-thought
- Prompt test sets and versioning
- Reducing hallucinations through prompt design
What to build
A prompt library with a 30-example test set and before/after scores.
What to study
- Calling LLM APIs: streaming, rate limits, keys
- Serving with FastAPI and simple UIs with Streamlit
- Caching and cost control
What to build
A small chat app with streaming responses, deployed for others to try.
What to study
- JSON schema outputs and validation
- Function / tool calling
- Retries and graceful failure
What to build
An invoice or resume parser that returns validated JSON.
What to study
- Chains, prompts and output parsers
- Document loaders and retrievers
- When a framework helps and when it gets in the way
What to build
Rebuild your chat app with LangChain and compare the code with your plain-API version.
RAG & model customisation
What to study
- Embedding models and similarity search
- Vector databases (e.g. pgvector, Pinecone, Weaviate)
- Chunking strategies and metadata filters
What to build
Index a document set in a vector database and query it with filters.
What to study
- Supervised fine-tuning vs prompting vs RAG
- LoRA and other parameter-efficient methods
- Preparing instruction datasets
What to build
LoRA fine-tune an open-weights model on a narrow task and compare it with prompting alone.
What to study
- Vision transformers and CLIP-style models
- Image captioning and visual question answering
- Combining text and image retrieval
What to build
An image search app that finds photos from a text description.
What to study
- The RAG pipeline end to end
- Re-ranking, hybrid search and citations
- Measuring retrieval quality
What to build
A document Q&A assistant that cites its sources, with a retrieval-quality report.
What to study
- Instruction tuning and reward models
- RLHF and DPO, conceptually and with TRL
- Where alignment fits in real products
What to build
Train a small reward model and run a preference-tuning loop on a toy task.
Agents & tool use
What to study
- ReAct, planning and reflection patterns
- Single-agent vs multi-agent setups
- Memory: short-term, long-term and retrieval
What to build
A research agent that searches, reads and summarises sources on a topic.
What to study
- Model Context Protocol (MCP) servers and clients
- Connecting agents to databases, CRMs and internal APIs
- Permissions and audit trails for tool calls
What to build
An MCP server that exposes two tools from a real API, used by your agent.
What to study
- Graphs, state and checkpoints
- Human-in-the-loop approvals
- Error handling in multi-step workflows
What to build
Rebuild your research agent as a LangGraph workflow with an approval step.
Production, evaluation & safety
What to study
- Docker images for AI apps
- CI/CD basics
- Secrets and environment configuration
What to build
Containerise your RAG assistant and deploy it with a CI pipeline.
What to study
- Building evaluation datasets
- LLM-as-judge and its limits
- Regression testing prompts and models
What to build
An eval harness that scores your assistant on 50 cases and runs on every change.
What to study
- Tracing LLM calls
- Tracking token cost and latency
- Dashboards and alerting
What to build
Add tracing and a cost dashboard to your deployed app.
What to study
- Managed model platforms on AWS, GCP and Azure
- Serverless deployment and autoscaling
- Data privacy and region choices
What to build
Deploy your app on one cloud using a managed model endpoint.
What to study
- Hallucination, bias and privacy risks
- Prompt injection and defences
- PII handling and content filtering
What to build
Add input and output guardrails to your assistant and document the risks you tested.
Portfolio & job readiness
What to study
- GenAI system design questions
- Explaining trade-offs: RAG vs fine-tuning, cost vs quality
- Coding rounds in Python
What to build
Mock-interview notes for three GenAI system design questions.
What to study
- Choosing a domain problem (healthcare, fintech, edtech)
- Writing clear READMEs and architecture notes
- Recording a short demo
What to build
A client-ready AI agent for one domain, with a public demo and write-up.
Fine-tuned text classifier
Adapting a pre-trained transformer to a real task.
Fine-tune a small pre-trained transformer on a labelled dataset, evaluate it properly and compare it with a classic ML baseline.
Structured data extractor
Turning messy documents into clean JSON.
Use an LLM API with schema-validated outputs to extract fields from invoices or resumes, with a prompt test set to track accuracy.
Document Q&A assistant
Retrieval-augmented answers with citations.
Index a document collection in a vector database and build an assistant that answers questions with cited sources and a retrieval-quality report.
LoRA fine-tune of an open model
Customising a model when prompting isn't enough.
Fine-tune an open-weights model with LoRA on a narrow task and document when it beats prompting and RAG, and when it doesn't.
Reward model & preference tuning
Teaching a model what good looks like.
Train a small reward model on preference data and run a preference-tuning loop, with a write-up of what changed.
Multimodal image search
Finding images from plain-language descriptions.
Embed images and text in a shared space and build a search app that retrieves photos from a description.
Tool-using research agent
An agent that plans, acts and asks for approval.
Build a LangGraph agent that searches, reads and summarises sources, calls tools through an MCP server and pauses for human approval.
Deployed GenAI service
From notebook to a monitored production API.
Containerise an LLM app, deploy it on a cloud platform with CI/CD, and add tracing plus a cost and latency dashboard.
Eval harness with guardrails
Proving your AI app works, and stays safe.
Create a 50-case evaluation set, automate scoring on every change, and add input and output guardrails against prompt injection and PII leaks.
Domain AI agent capstone
A client-ready agent for a real industry problem.
Pick a domain such as healthcare, fintech or edtech and ship an agent that combines retrieval, tools, evals and guardrails, with a public demo and write-up.
GenAI glossary 17 terms
Tap a term to see what it means. You will hear these in every GenAI interview.
- LLM
- Large language model: a neural network trained on huge amounts of text to predict and generate language.
- Transformer
- The neural network architecture behind modern LLMs, built around the attention mechanism.
- Token
- A chunk of text (often part of a word) that a model reads and writes. Pricing and limits are counted in tokens.
- Context window
- The maximum number of tokens a model can consider at once, including your prompt and its answer.
- Prompt engineering
- Designing instructions and examples so a model reliably produces the output you need.
- Embeddings
- Numeric vectors that capture meaning, so similar texts end up close together.
- Vector database
- A database built to store embeddings and find the most similar ones quickly.
- RAG
- Retrieval-augmented generation: fetching relevant documents and giving them to the model so answers are grounded.
- Fine-tuning
- Further training a pre-trained model on your own data to specialise it.
- LoRA
- Low-rank adaptation: a parameter-efficient fine-tuning method that trains small add-on weights instead of the whole model.
- Agent
- An LLM system that plans steps, calls tools and acts towards a goal, often over several turns.
- Tool calling
- Letting a model call functions or APIs, such as search or a database query, with structured arguments.
- MCP
- Model Context Protocol: an open standard for connecting AI applications to tools and data sources.
- Evals
- Systematic tests that measure how well an AI system performs on representative cases.
- Guardrails
- Checks on inputs and outputs that keep an AI system safe, on-topic and compliant.
- Hallucination
- When a model states something false or unsupported with confidence.
- Prompt injection
- An attack where malicious text tricks a model into ignoring its instructions.
Close your gaps with a structured programme
The Certificate Programme in Generative AI from CEP, IIT Delhi covers transformers, fine-tuning, RAG, agents and responsible AI through live weekend classes and hands-on projects.
Why is Attention Needed in Transformers?
Attention enables a Transformer to determine which tokens are relevant to a particular token and combine their information into context-aware representations.
How does attention connect related tokens?
Consider the sentence: “The animal was tired because it ran.”
To interpret “it,” a model needs information about the earlier words, particularly “animal.” Attention allows a token’s representation to incorporate information from other relevant positions rather than processing every token independently.
Boundary: Self-attention creates these connections within a sequence. The following calculations show how it works.
How does Self-Attention Work Step by Step?
The self-attention mechanism projects the same input sequence into Q, K and V matrices. It calculates query-key similarity scores, applies softmax to obtain attention weights, and uses those weights to combine value vectors.
What do queries, keys and values represent?
- Query (Q): Represents what a token is looking for.
- Key (K): Represents the features used to match a token against a query.
- Value (V): Contains the information combined to produce the output.
The projections are:
Q = XWQ
K = XWK
V = XWV
Here, X represents the input token representations, while WQ, WK and WV are learned projection matrices.
Worked example: Calculate self-attention using small matrices
Consider a two-token sequence with the following deliberately simplified matrices:
Q = K=[[1, 0], [0, 1]]
V = [[1, 0], [0, 2]]
These matrices are illustrative; they are not trained model parameters.
Step 1: Calculate the raw attention scores.
Multiply Q by the transpose of K:
QKᵀ = [[1, 0], [0, 1]]
Each token has a score of 1 for itself and 0 for the other token.
Step 2: Apply scaled dot-product attention.
The self-attention formula is:
Attention(Q, K, V) = softmax(QKᵀ / √dk)V
Here, dk is the key-vector dimension, which is 2. Divide the scores by √2:
QKᵀ / √2 ≈ [[0.707, 0], [0, 0.707]]
Scaling helps prevent large dot products from producing excessively concentrated softmax distributions.
Step 3: Apply row-wise softmax.
Softmax converts each row of scores into weights that sum to 1:
Attention weights ≈ [[0.67, 0.33], [0.33, 0.67]]
These rounded values indicate how much each token attends to both positions.
Step 4: Multiply the weights by V.
[[0.67, 0.33], [0.33, 0.67]] × [[1, 0], [0, 2]]
Output ≈ [[0.67, 0.66], [0.33, 1.34]]
Each output combines information from both value vectors. The first token assigns more weight to the first value, while the second assigns more weight to the second.
Why is the score divided by the square root of the key dimension?
As the key dimension increases, dot products can become large in magnitude. Large scores can make softmax distributions overly concentrated and produce small gradients. Dividing by √dk helps control their scale during training.
Boundary: This example demonstrates one attention head. Transformers can use multiple heads and masks to capture different relationships or restrict which tokens can interact.
What are Multi-Head Attention and Masked Self-Attention?
Multi-head attention runs multiple attention operations in parallel, while masked self-attention restricts which positions a token can access.
How does multi-head attention work?
Each head uses its own learned Q, K and V projections. Different heads can learn different relationships, such as nearby dependencies or longer-range contextual links. Their outputs are concatenated and passed through an output projection.
However, individual heads are not guaranteed to specialise in a particular relationship; their behaviour depends on what the model learns.
What does a causal mask do?
A causal attention mask prevents a token from attending to future positions. For three tokens, the allowed positions can be represented as:
[[1, 0, 0],
[1, 1, 0],
[1, 1, 1]]
Here, 1 means attention is allowed and 0 means it is blocked. In implementations, blocked scores are typically masked before softmax.
Masked self-attention is used in autoregressive decoders, including GPT-style models, so predictions do not use future tokens that would be unavailable during generation.
Boundary: Masking changes which positions can contribute, not where Q, K and V originate. Cross-attention changes that source.
How does Cross-Attention Work?
Cross-attention uses queries from one sequence and keys and values from another. In a standard encoder-decoder Transformer, decoder states provide Q, while encoder outputs provide K and V.
How does cross-attention work in an encoder-decoder Transformer?
In machine translation, the encoder processes a source sentence, such as “The cat sleeps.” The decoder generates the target-language sentence while accessing relevant information from the encoded source.
The projections are:
Q = XdecoderWQ
K = XencoderWK
V = XencoderWV
The same scaled dot-product attention formula applies. The key difference is that the query and key-value matrices come from different sequences.
A small cross-attention example
Consider one decoder query and two encoder positions:
Q = [[1, 0]]
K = [[1, 0], [0, 1]]
V = [[2, 0], [0, 4]]
Step 1: Calculate and scale the scores.
QKᵀ = [[1, 0]]
Since dk = 2:
QKᵀ / √2 ≈ [[0.707, 0]]
Step 2: Apply softmax.
Attention weights ≈ [[0.67, 0.33]]
Step 3: Multiply by V.
[[0.67, 0.33]] × [[2, 0], [0, 4]]
Output ≈ [[1.34, 1.32]]
The decoder query assigns more weight to the first encoder position, while still incorporating information from the second. All decimal values in these examples are rounded approximations.
Where is cross-attention used beyond translation?
Cross-attention is used in encoder-decoder tasks such as translation and summarisation. It can also connect different modalities in multimodal models, such as allowing text representations to attend to image features. The exact design varies, and not every multimodal architecture uses cross-attention in the same way.
Boundary: Both mechanisms use the same core calculation. Their difference lies in how the input representations are connected.
Self-Attention vs Cross-Attention: What Is the Difference?
The main self-attention vs cross-attention difference is the origin of Q, K and V. Self-attention derives all three from the same sequence, while cross-attention uses Q from one sequence and K and V from another.
| Feature | Self-attention | Cross-attention |
|---|---|---|
| Input source | One sequence | Two sequences or representations |
| Query (Q) | Same sequence as K and V | Querying sequence |
| Keys (K) and values (V) | Same sequence as Q | Separate sequence |
| Main purpose | Build context within a sequence | Transfer information across sequences |
| Typical use | Encoder attention; decoder masked attention | Encoder-decoder attention |
| Attention score matrix | Input length × input length | Query length × key/value length |
| Example | Connecting words in a sentence | Decoder attending to source tokens |
How can you identify which type of attention a model uses?
Check where Q, K and V originate. If all three come from the same sequence, it is self-attention. If Q comes from one sequence and K and V from another, it is cross-attention.
A causal mask does not turn self-attention into cross-attention. It simply restricts which positions within the same sequence can contribute.
Boundary: These distinctions provide a practical way to recognise attention operations when studying Transformer architectures or reading their implementations.
Conclusion
Self-attention builds contextual relationships within a sequence, whereas cross-attention allows one sequence to retrieve information from another. Both rely on Q/K/V projections, scaled dot-product attention and softmax, but differ in their input sources and typical uses. Working through the attention score matrix and output calculation makes this distinction easier to understand.
To explore these concepts alongside Transformer architecture and other Generative AI foundations, explore the Generative AI programme by IIT Delhi, delivered on Varsity by InterviewBit.
Frequently Asked Questions
Can self-attention and cross-attention use the same attention formula?
Yes. Both can use scaled dot-product attention. The difference is whether Q, K and V originate from one sequence or from separate sequences.
Does cross-attention always require an encoder-decoder architecture?
No. Encoder-decoder Transformers are a common example, but cross-attention can also connect different modalities or representations.
Can the query and key/value sequences have different lengths?
Yes. The output has one row per query position, while the attention score matrix has one column per key position.
Why do attention weights add up to one?
Softmax normalises the scores across the available positions for each query, so the weights sum to one, subject to numerical rounding.
Does self-attention preserve the original token embeddings?
Not necessarily. It produces context-enriched representations by combining value vectors according to the attention weights.
Is masked self-attention the same as cross-attention?
No. Masked self-attention restricts access to positions within a sequence. Cross-attention connects queries from one sequence to keys and values from another.
What happens if all attention scores are equal?
Softmax assigns equal weights to all unmasked positions. The output becomes the average of their value vectors.
Does every Transformer layer contain cross-attention?
No. Encoder-only and decoder-only Transformers can use self-attention without a separate cross-attention layer. Standard encoder-decoder Transformers typically include cross-attention in the decoder.
How does sequence length affect the attention matrix?
For self-attention with sequence length n, the score matrix contains n × n entries. For cross-attention, it contains nq × nk entries, where nq is the query length and nk is the key/value length.
What should I calculate first when solving an attention problem by hand?
Identify the origins and dimensions of Q, K and V. Then calculate QKᵀ, scale the scores, apply softmax row-wise, and multiply the resulting weights by V.