Why floating-point addition isn't associative

Schoolroom math says (a + b) + c equals a + (b + c). On a real computer it doesn't, and that one fact ripples out into nondeterministic GPU reductions, irreproducible training runs, and LLM outputs that aren't bit-stable across hardware.

The picture version

Six pictures for a reader who has never thought about how a computer stores a number. The prose below fills in the seams the pictures skip.

Why it exists

You have almost certainly seen the small version of this. Open any Python prompt, or a JavaScript console, or a spreadsheet cell, and add 0.1 + 0.2. You get 0.30000000000000004. Everyone’s first reaction is “the computer got arithmetic wrong,” and everyone’s first explanation is “rounding.” Both are close enough — but the interesting consequence of that rounding isn’t a weird digit at the end. It’s this: the order you add numbers in changes the answer.

Which brings us to the version that costs people real time. You re-run the same training script with the same seed, the same data, the same model, the same hardware. The loss curve at step 10,000 is almost the same as last time — but not quite. Off by 0.0003. Run it again: off by something different. Nothing in your code changed. Nothing in your config changed. The bits going into the GPU are identical. The bits coming out aren’t. This is not a bug in CUDA or in your framework. It’s the consequence of a single, deeply unintuitive fact about computer arithmetic: floating-point addition isn’t associative. (a + b) + c and a + (b + c) can give different answers, and on modern hardware they routinely do.

The reason is finite precision. A real number has, in principle, infinitely many digits. A floating-point number has a fixed number of bits — 32, 16, 8 — split between an exponent (how big) and a mantissa (how precisely). After every single arithmetic operation, the result is rounded back to fit. Round, round, round. Once you accept that, the non-associativity is forced: which intermediate values you produce — and therefore which ones get rounded — depends on the order you produced them in. Different order, different intermediates, different rounding errors, different final answer.

One three-number example carries this whole post: a = 1e20, b = -1e20, c = 1. Before reading on, commit to a guess — does (a + b) + c equal a + (b + c), and if not, which one loses the 1?

This is not a flaw the standards committee forgot to fix. IEEE 754 — the spec almost every CPU and GPU implements — defines each operation as “compute the exact result, then round it to the nearest representable value.” Non-associativity follows from that definition rather than being legislated separately, and the committee accepted the consequence rather than missing it. What you get in exchange is that every individual operation is exactly specified and portable; what you give up is the algebraic laws of the real numbers.

Why it matters now

This sounds like a textbook curiosity. It isn’t — it’s the load-bearing explanation for several things engineers hit constantly.

• GPU reductions associate differently from serial code, and many paths are nondeterministic. Summing a million numbers on a GPU isn’t done left-to-right; it’s done as a parallel tree of partial sums whose shape depends on tile sizes, the kernel the runtime picked for the current input shape, and — for paths that use atomics or unordered work queues — the actual scheduling. Different shape or different reduction path, different rounding errors, different final logits. (A fixed kernel on the same hardware can be bit-stable; the surprises tend to come from atomics, library version drift, or shape-dependent kernel dispatch.)
• Training runs aren’t bitwise reproducible. Same script, same seed, same hardware — the loss curve is close but not identical. Frameworks expose flags to force order-stable reductions, but they cost throughput, so the default is “fast and almost-deterministic” rather than “slower and bit-exact.”
• Temperature-zero LLM sampling drifts. Even with greedy decoding, the argmax is computed over logits that are themselves the output of a long chain of GPU reductions. A near-tie between two tokens can flip based on which other requests happened to share the inference batch. That’s the thread the why-temperature-zero-isnt-deterministic post pulls on.
• Lower precision amplifies it. In BF16 or FP8, the rounding step throws away more bits per operation, so the same reordering produces a larger drift. The non-associativity is the same; the gap between “two valid answers” is just wider.

The short answer

float non-associativity = finite mantissa + rounding after every op

Picture to keep: a scale whose resolution gets coarser the more weight is already on it. Step on holding a paperclip and the reading doesn’t budge — the paperclip is finer than the tick marks at your weight. Put the paperclip on an empty scale and it registers fine. Whether a small quantity survives depends on what it was added next to, and the order of operations decides that. (Where the analogy breaks: a real scale rounds readings; a float rounds the stored value itself, so the loss is permanent — the paperclip isn’t sitting there un-measured, it’s gone.)

Real numbers carry as many digits as they need. Floating-point numbers don’t — every result gets rounded to a fixed mantissa width. Rounding is where the information loss happens, and information loss isn’t order-invariant. Re-arrange the additions, you re-arrange which intermediate values get rounded, and that changes the final answer. The same equation, evaluated two valid ways, gives two slightly different floats.

How it works

Here is the running example, run for real. In FP32, with a = 1e20, b = -1e20, c = 1:

import numpy as np
a, b, c = np.float32(1e20), np.float32(-1e20), np.float32(1.0)
(a + b) + c   # → 1.0
a + (b + c)   # → 0.0

Same three numbers. Same operator. Different parenthesization. Different answer. (You can paste those lines into a Python shell and reproduce the split yourself.)

Walk through (a + b) + c. First 1e20 + (-1e20) = 0 exactly — two equal- magnitude opposites cancel, no rounding needed. Then 0 + 1 = 1. The 1 survived because it never got near a giant number.

Walk through a + (b + c). First -1e20 + 1. The number -1e20 lives at a magnitude where the gap between consecutive FP32 values is enormous — much bigger than 1. (That gap has a name: ULP.) At 1e20, one ULP in FP32 is roughly 2^43 ≈ 9e12. Adding 1 to -1e20 produces a true result that lies between two representable FP32 values, and rounding picks the nearer one — which is -1e20 itself. The 1 was annihilated by the rounding step. Then 1e20 + (-1e20) = 0. The 1 is gone, never to return.

Re-association is what decided whether the small number survived. Pair it with its near-equal partner first and it lives. Pair it with a giant first and it’s silently rounded away.

The general shape: every floating-point operation is round(true_result). Re-association doesn’t change true_result, but it changes which intermediate gets rounded. If a small number meets a big number first, the small one is silently swallowed. If two big numbers meet first and cancel, the small one survives. The associative law is a property of the exact arithmetic underneath, not of the rounded arithmetic the hardware actually performs.

A few consequences fall out of this:

• Commutativity is not broken. a + b and b + a produce the same float in IEEE 754 (NaN payload edge cases aside). The rounding rule doesn’t care about the order of two operands. Re-grouping is the thing that breaks, not re-ordering of a single sum.
• Sum order matters in long reductions. Summing a million floats left-to-right, in pairs, or in a parallel tree gives three different answers. None of them is “wrong” — they’re all valid IEEE 754 computations of subtly different expressions. The answers closest to the exact sum typically come from pairwise/tree summation or from Kahan summation rather than from naive left-to-right — Kahan has a proven error bound; “tree beats sequential” is a strong tendency, not a theorem for every input.
• GPUs lean into this. A GPU matmul doesn’t sum left-to-right — it splits the dot product across thousands of threads and combines partial sums in a tree whose shape depends on tile choices. That’s one source of the “same matmul, different bits” problem — others include atomics, fused multiply-add paths, TF32 fallbacks, and library version drift. Frameworks like PyTorch expose deterministic-mode flags that force order-stable reductions, but at a real throughput cost. The exact source of nondeterminism in a given training run is hardware- and library-specific, and you generally have to read framework determinism docs and vendor numerical-precision guides to track it down case by case.

The honest seam: I’m describing the mechanism — finite mantissa plus per-op rounding plus reordering. The empirical question of how much this matters for a given workload (training a 70B model? doing a 3×3 matmul?) depends on numerical conditioning, precision, and reduction strategy in ways that don’t compress to a single rule. For a deeper treatment, the canonical reference is Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic (1991).

You started with float non-associativity = finite mantissa + rounding after every op. What did 1e20 + (-1e20) + 1 add to that line? — + re-grouping changes which value gets rounded, not what the true answer is. That’s why the fix is never “use more precision” (a wider mantissa just moves the threshold); it’s “control the order,” which is exactly what deterministic-mode flags buy you and exactly what they cost throughput for.

Check yourself

Before you go — you sum a million positive floats. Your colleague sorts them smallest-to-largest first and gets a different (and closer to exact) total. Why does sorting help, and would sorting largest-to-smallest help too?

And: your inference server returns bit-identical logits for a prompt when you send it alone, but slightly different ones when the same prompt shares a batch with other requests. Is this a bug in the model weights?

Famous related terms

• ULP — ULP = gap between adjacent floats at this magnitude. Under round-to-nearest, an addend below half a ULP rounds away entirely. The reason 1e20 + 1 swallows the 1.
• Kahan summation — Kahan sum = running total + a compensation term that recovers lost bits. A clever trick that recovers most of the precision a naive long sum throws away. Costs a few extra ops per element.
• Denormals (subnormals) — denormal = float below the smallest normal value, encoded with reduced precision. The format’s escape hatch for “almost zero” — slow on many CPUs, and often flushed to zero in fast-math GPU paths.
• TF32 — TF32 = FP32 inputs rounded to TF32 precision inside tensor cores + FP32 accumulate. Trades a few mantissa bits for big throughput; another place rounding-order shows up.
• BF16 vs FP16 — different ways to spend a 16-bit budget; the format you pick changes how much the non-associativity hurts.
• Why temperature 0 isn’t deterministic — the most visible downstream consequence.

Going deeper

• IEEE 754-2019 — the primary source; go here for the exact statement of the rounding rules that make non-associativity unavoidable, rather than taking anyone’s word (including this post’s) for it.
• David Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic, ACM Computing Surveys 23(1), 1991 — the explainer, and the best answer to “how much error does a given computation actually accumulate?” Thirty-plus years old and still the reference people cite.
• Nicholas Higham, Accuracy and Stability of Numerical Algorithms — the rabbit hole for when “just be careful with the ordering” stops being good enough and you need real error analysis for a specific algorithm.