AI · LESSON 12 / 25 · English
The Transformer Architecture
Attention helps representations use context.
Content reviewed: 22 October 2026 · தமிழில் படிக்க ↗
Understand it in plain language
A Transformer processes representations using attention and other components, including feed-forward layers and positional information. Attention weights combine information from different positions. Encoder, decoder, and encoder–decoder variants support different tasks. Transformers are influential, but not every modern AI system uses one, and attention weights alone are not a complete explanation of a model decision.
How the AI approach works
A Transformer builds contextual representations: the representation of a token can depend on other relevant tokens. Each input token becomes an embedding with positional information. Attention then mixes information across positions. Feed-forward layers transform each position further, while residual connections and normalization support the stacked computation.
In scaled dot-product attention, learned projections form queries Q, keys K, and values V. Query–key similarity gives scores QKᵀ/√dₖ. After an appropriate mask, softmax normalizes scores into weights, and those weights combine V. A query asks what information is relevant here; keys determine matching; values provide the information to combine. These are vectors learned for a task, not literal database questions and answers.
Multi-head attention performs several such mixtures in different learned subspaces. A causal mask prevents a decoder from attending to future positions during next-token training. Encoder models can use bidirectional context for representation tasks; encoder–decoder models combine input encoding with conditioned generation; decoder-only models generate autoregressively. Architecture and training objective must be considered together.
Attention has costs: ordinary full attention compares many pairs of positions, making long sequences expensive. Implementations and variants change this tradeoff. A high attention weight is not a complete causal explanation of an answer, and a longer context window does not ensure that every important detail is used correctly.
Break the idea into steps
- Represent tokens or other inputs as vectors and include position information.
- Compute attention scores, normalize them, and combine value vectors.
- Repeat transformations across layers; masks control which positions may be used.
Worked example: connect the method to the task
Take one query q = [1, 0], two keys [1, 0] and [0, 1], and values [10, 0] and [0, 20]. The scaled scores are about 0.707 and 0. Softmax gives approximately 0.670 and 0.330, so the mixed value is about [6.698, 6.605]. The result retains information from both values with different weights.
The experiment calculates this actual vector attention operation. Its vectors are supplied by hand; it omits learned projections, multiple heads, positional encoding, and training. It is one component, not a complete Transformer.
Algorithm experiment: predict, change, explain
This calculates one scaled dot-product attention result with hand-set vectors. Change the query to [0, 1] and predict which value receives more weight. It does not train a Transformer.
import math
q = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0]]
values = [[10.0, 0.0], [0.0, 20.0]]
scores = [sum(a*b for a, b in zip(q, key)) / math.sqrt(len(q))
for key in keys]
# Subtracting the maximum improves numerical stability.
exp_scores = [math.exp(s - max(scores)) for s in scores]
weights = [e / sum(exp_scores) for e in exp_scores]
result = [sum(weights[j] * values[j][d] for j in range(len(keys)))
for d in range(len(values[0]))]
print("weights:", [round(x, 3) for x in weights])
print("mixed value:", [round(x, 3) for x in result])Predict first, then reveal the checked output
weights: [0.67, 0.33] mixed value: [6.698, 6.605]
Change one thing. Run it again.
Run this small, self-contained experiment in standard Python. No package installation or paid API is required. Change the suggested variable and explain the effect on the method before applying it to real data.
Let the editor finish loading, load this example, then press Run.
The free editor uses OneCompiler, an external service. Code you run is sent there. Do not enter passwords, API keys, or private student data. If an example is sent before the editor is ready, wait and load it again.
Apply the idea: check your reasoning
Think before revealing the answer
Why must a next-token decoder hide future tokens during training?
Reveal the answer and reasoning
Otherwise it could use the answer it is supposed to predict, creating a training task unavailable during generation. A causal mask enforces the permitted context for attention.
A common mistake: Do not claim that Transformers power every image generator or that attention is human-like understanding.
Before you move on
- Explain the main idea without repeating the technical label.
- Identify the approach, its assumptions, and the evidence needed to trust the result.
- Complete the challenge, then compare your reasoning with the answer.
Sources & a deeper next step
The refresh uses primary papers and official documentation. These optional links are in English; you can understand this lesson without reading them first. Product versions and availability can change.