Why LayerNorm (and RMSNorm) exist

Every transformer block has a normalization step. Pull it out and training falls apart in the first thousand steps. Why is this tiny operation load-bearing?

The picture version

Six pictures for a reader who has never watched a training run die. The prose below fills in the seams the pictures skip.

Why it exists

You’ve left something big running overnight — a render, an export, a backup — and come down in the morning to find it died at 3am with an error that explains nothing. Now picture the machine-learning version: you kick off a training run before bed, and in the morning the loss curve is a cliff. A few hundred steps of healthy descent, then a vertical line into nan. You check the data loader. You check the learning rate. You check for a bad batch. Everything looks fine. That’s the running example for the whole post — one training run that dies at step 800 — and the culprit is going to be the least interesting-looking line in the model.

Because inside every transformer block, sandwiched between the attention and the feed-forward, there is a tiny operation that looks like an afterthought: take the activations, subtract their mean, divide by their standard deviation, multiply by a learned scale. Five lines of code. No moving parts. Nobody talks about it.

Pull it out and at modern depth and learning rate the model usually doesn’t train. The loss explodes in the first thousand steps, or plateaus at the cross-entropy of “guess uniformly.” (You can sometimes rescue an un-normalized transformer with very careful initialization and residual scaling, but that’s its own line of research, not the default.)

So the question is: why does a deep network — billions of parameters, trillions of training tokens, attention heads doing all the linguistically interesting work — collapse without a five-line rescaling step?

The short version is that gradients in deep networks are a chain of multiplications, and chains of multiplications go to zero or infinity unless something keeps them tame. Normalization is what keeps them tame. Everything else in the architecture assumes it.

Why it matters now

If you’re building, fine-tuning, or even just reasoning about modern LLMs, the choice of normalization shows up in places you’d rather not have to think about:

• Architecture diffs across model families. Llama, Mistral, Gemma, Qwen and DeepSeek use RMSNorm. T5 uses a close cousin — a LayerNorm variant with no mean subtraction, which is RMSNorm in all but name and predates the RMSNorm paper. GPT-2 and GPT-3 use classical LayerNorm. When you port weights or compare papers, this is one of the first-line “is this the same shape” checks.
• Pre-LN vs. Post-LN. Where the norm sits inside the residual block strongly affects whether the model can be trained without a learning-rate warm-up. Pre-LN dominates recent large LLMs; the original 2017 transformer was Post-LN. (My read, not a sourced claim: that placement is part of why warm-up schedules were treated as a mandatory ritual for several years before Xiong et al. explained what they were compensating for.)
• Quantization and inference kernels. Norm layers are tiny in parameter count but they are sequential dependencies in the forward pass — every fused-attention kernel and every quantization scheme has to deal with them explicitly. RMSNorm has one fewer reduction to implement on a GPU; no model team has written down that this is why they picked it, so treat the kernel-simplicity story as a plausible motive rather than a stated one.
• Debugging training runs. “Loss is NaN at step 800” is, more often than not, an interaction between the norm, the residual stream, and learning-rate schedule. You can’t reason about it without knowing what the norm is doing.

The norm is small, boring, and load-bearing. It’s the structural beam in the wall — easy to forget about until you remove it.

The short answer

normalization layer = per-token rescale to a fixed magnitude + a learned scale the network can override it with

Picture to keep: a chain of a hundred amplifiers wired in series, with a technician standing between every pair resetting the volume to a fixed level — so no amount of accumulated gain or attenuation can run away down the chain. Like that, except the technician also hands the next amplifier a dial (the learned scale), so the network can still ask for a louder signal — it just has to ask on the record instead of getting it by accident.

A normalization layer rescales each token’s activation vector so its magnitude is roughly fixed, regardless of how the previous layers chose to amplify it. That fixed magnitude is what makes the gradient chain through a hundred-layer network behave. The two versions you’ll meet differ only in how much of the rescale they do:

LayerNorm = per-token (subtract mean, divide by std) + learned scale + learned shift
RMSNorm   = per-token (divide by RMS)                + learned scale

RMSNorm is the same idea with the mean-subtraction step removed, after Zhang and Sennrich (2019) found it performed comparably without it in their experiments.

How it works

The design falls out of a chain of failures. Each fix below exists because the previous attempt broke — and your step-800 nan is what the first break looks like from the outside.

Attempt 1: control the scale at initialization — and why it breaks

A deep network is a chain of matrix multiplications, nonlinearities, and adds. The gradient at the bottom is the product of all the Jacobians along the way. Think of each one as a multiplier. If the typical multiplier is bigger than 1, a hundred of them in a row blow up. If it’s smaller than 1, a hundred of them shrink to nothing. Either way you stop learning — and step 800 is where “blows up” becomes nan.

You can try to control this by careful initialization (Xavier, He, etc.) and by clipping gradients. Those help. They are not enough at the depth and learning rate transformers want to train at.

Normalization is the brute-force fix. After every layer (or twice per block, in a transformer), you reach in and rescale the activations so that their magnitude is fixed by construction. The network can no longer drift into a regime where one block multiplies by 100 and the next divides by 100. Activations stay in a roughly controlled range across depth, which tends to keep the chain of Jacobians from blowing up or collapsing — not as a guarantee, but as a strong empirical regularization.

There’s a learned scale (and, in LayerNorm, a learned shift) so the network can still choose a non-unit magnitude where it’s useful — it just has to ask for it explicitly through a parameter, instead of getting it accidentally through compounded weight scales.

Attempt 2: normalize across the batch — and why that breaks

The 2016 LayerNorm paper (Ba, Kiros, Hinton) was a response to a specific limit of Batch Normalization: BatchNorm normalizes each feature across the batch, which means the statistics depend on which other examples happen to be in the mini-batch. That’s fine for image classifiers with fixed-size batches in training. It’s a problem for sequence models, where you have variable lengths, padding, and — at inference — autoregressive generation that produces tokens one at a time. There is no “mini-batch of one token” that gives you meaningful BatchNorm statistics.

LayerNorm sidesteps this by normalizing per token, across the hidden dimension. Each token’s activation vector is normalized using only its own statistics. Training and inference behave identically. Batch size doesn’t matter. Padded positions don’t pollute anything. This is the property transformers need and it’s why the original transformer (and most that followed) reached for LayerNorm rather than BatchNorm, despite BatchNorm being older and more famous. Some later models switched to RMSNorm; almost none went back to BatchNorm. Note what this buys you for the step-800 run: whatever went wrong, it wasn’t “an unlucky batch poisoned the statistics.” LayerNorm made that class of failure impossible.

Attempt 3: put the norm after the residual add — and why it needs a crutch

The original “Attention Is All You Need” transformer placed LayerNorm after the residual add: x + Sublayer(x) then norm. This is Post-LN. It worked, but in practice it required a careful learning-rate warm-up that ramped from near-zero over thousands of steps; without warm-up, training tended to diverge — the step-800 cliff, again, from a different cause. The warm-up recipe circulated as practical folklore for a few years before anyone published the reason.

Xiong et al. (2020), On Layer Normalization in the Transformer Architecture, is that reason. In Post-LN, the gradients near the output layer are large at initialization, which means a sane learning rate at depth 100 looks like a learning-rate-from-hell at depth 1. Warm-up was a workaround: start with such a tiny learning rate that nothing diverges, then ramp once the network has shaped itself.

Move the norm inside the residual block — x + Sublayer(LN(x)), which is Pre-LN — and the Xiong analysis shows the gradients are well-behaved at initialization across all depths. In their experiments that let them drop warm-up entirely on the setups they tested. The broader practitioner claims that follow — deeper models, higher learning rates — are the community’s experience rather than results from that paper, so take them as the reason Pre-LN won, not as a theorem.

This is one of those changes where the diff is one line and the practical consequence is large — Pre-LN is one of several enabling factors (alongside better optimizers, gradient clipping, and init schemes) that make training very deep, very large transformers tractable. Pre-LN is the dominant choice in recent large LLMs.

Attempt 4: trim the half that wasn’t earning its keep

LayerNorm does two things to each token vector: subtract the mean (centering) and divide by the standard deviation (scaling). Zhang and Sennrich (2019) asked: do we actually need the centering?

Their hypothesis: the re-scaling invariance is what stabilizes training. The re-centering invariance is mostly cosmetic. They removed the mean subtraction, kept only the divide-by-RMS, and called it RMSNorm.

LayerNorm(x) = (x - mean(x)) / std(x) * gamma + beta
RMSNorm(x)   = x / sqrt(mean(x^2) + eps) * gamma

Empirically, in their paper and in many subsequent reports, models trained with RMSNorm reach roughly the same loss as LayerNorm — sometimes a hair better, sometimes a hair worse, broadly comparable. But RMSNorm is cheaper: one fewer reduction over the hidden dimension, no shift parameter, simpler GPU kernels. Their paper reports a 7%–64% reduction in running time per training step across the models and hardware they tested — a wide range, but real money at LLM training scale.

This is why Llama, Mistral, Gemma, Qwen, and DeepSeek use RMSNorm, and why T5’s no-mean-subtraction LayerNorm variant looks like the same idea arrived at independently. GPT-2 and GPT-3 used classical LayerNorm because they predate the RMSNorm result. One boundary worth naming: the normalization choice in GPT-4 and later closed models isn’t public, so this list stops at open weights and published papers.

The honest gaps

A few things are not fully understood, and the literature is still arguing:

• Why mean-centering doesn’t matter. Zhang and Sennrich showed empirically that you can drop it. The theoretical story for why re-centering is dispensable is still being filled in. Recent work (e.g. Re-Introducing LayerNorm: Geometric Meaning, Irreversibility and a Comparative Study with RMSNorm, 2024) revisits this with geometric arguments. There isn’t a single textbook proof.
• Whether normalization is strictly necessary or just very convenient. There’s a small but persistent line of work on norm-free transformers — careful init, residual scaling, no explicit norm. They sometimes match LayerNorm baselines on small models. They have not, so far, replaced LayerNorm/RMSNorm in production LLMs. My read is “norms are an empirically dominant local optimum, not a mathematical requirement,” but the question is genuinely open.
• The exact gradient story. “Normalization keeps gradients tame” is the right shape of the answer, but the precise mechanism by which Pre-LN gradients become uniform across depth involves some nontrivial linear-algebra accounting that’s easier to verify experimentally than to derive cleanly from first principles. The Xiong 2020 paper does the derivation; it’s not a one-liner.

The thing to walk away with: normalization is the part of the transformer that exists to keep the rest of the transformer trainable. It’s not where the model stores knowledge. It’s not what makes the architecture expressive. It’s the structural support that lets a hundred residual blocks chain together without the gradient landscape going to hell — and it’s why your step-800 nan was never a data-loader bug.

You started with LayerNorm = (subtract mean, divide by std) + learned scale and shift. What did this post add that the formula can’t show? — + where you put it. The arithmetic is five lines and barely changed since 2016; the thing that actually decided whether deep transformers train was moving the norm inside the residual block. Same operation, different position, different world.

Check yourself

Before you go — a colleague removes the learning-rate warm-up from a Post-LN transformer and the run diverges in the first 200 steps. They propose fixing it by lowering the learning rate permanently. What’s the cheaper fix, and what does their proposal cost them?

And one more — RMSNorm drops the mean-subtraction and keeps the divide-by-RMS. If someone built the opposite layer — subtract the mean, don’t divide — would you expect it to stabilize training?

Famous related terms

• LayerNorm — LayerNorm = per-token (subtract mean + divide by std) + learned scale + learned shift. The 2016 default. Still used in GPT-style models.
• RMSNorm — RMSNorm = per-token divide by RMS + learned scale. LayerNorm with the centering removed. The default in Llama, Mistral, Gemma, Qwen and DeepSeek; T5 uses an equivalent-in-spirit no-mean-subtraction variant that predates the name.
• BatchNorm — BatchNorm = per-feature normalize across the batch + learned affine. The original normalization, from 2015 (Ioffe & Szegedy). Dominant in CNNs, ill-suited to sequence models because batch statistics couple examples together.
• Pre-LN — Pre-LN = norm goes inside the residual block. x + Sublayer(LN(x)). The placement that often lets you skip or shrink learning-rate warm-up and train deep transformers stably.
• Post-LN — Post-LN = norm goes after the residual add. LN(x + Sublayer(x)). The 2017 original. Works, but typically requires warm-up; uncommon in big modern LLMs.
• Softmax — softmax(x) = exp(x) / Σ exp(x). Also a normalization, of a different flavor: it normalizes a vector to a probability distribution. Unrelated mechanism, related vibe.
• DyT (Dynamic Tanh) — DyT ≈ replace normalization with a learned tanh. Recent (2025) proposal to replace normalization layers with a learned tanh. Interesting, not yet standard. Worth watching, not betting on.

Going deeper

• Layer Normalization (Ba, Kiros, Hinton, 2016, arXiv:1607.06450) — the primary source; read it for the original case against BatchNorm, argued in the RNN era where the variable-length problem is starkest.
• On Layer Normalization in the Transformer Architecture (Xiong et al., 2020, arXiv:2002.04745) — answers “why does placement matter?”, with the gradient derivation behind Pre-LN vs. Post-LN and warm-up.
• Sebastian Raschka’s RMSNorm vs. LayerNorm — the explainer: the same LayerNorm/RMSNorm comparison worked through with pictures rather than equations.
• Root Mean Square Layer Normalization (Zhang & Sennrich, 2019, arXiv:1910.07467) — the rabbit hole, for “what exactly did RMSNorm delete, and what did the experiments actually show?”