Why matrix multiplication is the bottleneck of modern ML

Modern ML is mostly one operation in a trench coat. Understanding why matmul dominates explains hardware, software, and why GPUs eat the world.

The picture version

Five pictures for a reader who has never wondered what a graphics card is for. The prose below fills in the seams the pictures skip.

Why it exists

Every time ChatGPT writes a word, Midjourney paints a pixel, or your phone unlocks with Face ID, the hidden grunt work behind the scenes is the same one operation: multiplying enormous tables of numbers together. That’s matrix multiplication — “matmul.” It’s why a single high-end GPU costs more than the rest of a gaming PC combined: the GPU is essentially a factory built to do this one calculation millions of times in parallel. Take this operation away and modern AI doesn’t just slow down — it disappears.

Open the profiler on almost any modern model — a transformer, a diffusion model, a ResNet from a decade ago — and you’ll see the same picture. One operation dominates the FLOP count, most of the memory traffic, and most of the wall-clock time: matrix multiplication, multiplying two 2D tables of numbers so that each output entry is a dot product of a row and a column. Everything else — the activations, the normalizations, the softmaxes — is a small share of the arithmetic budget.

That’s a strange thing to discover. Why would a field as varied as “vision, language, audio, robotics, protein folding” all collapse onto a single linear-algebra primitive? It’s not because researchers picked it; it’s because everything else stopped scaling, and matmul didn’t.

Why it matters now

This one fact silently shapes decisions you’re already making:

• GPUs dominate ML because they’re matmul firehoses. Most of a modern training GPU’s advertised FLOPs come from its matrix-multiply units, not its general-purpose lanes.
• The reason an LLM bills you per token, not per “thought,” is that each token costs roughly one matmul-pass through the model’s weights.
• Quantization (FP16 → INT8 → INT4) pays off twice over on matmul: fewer bits means both cheaper arithmetic in the matrix units and less memory traffic to feed them.
• “Why is my model slow?” answers almost always reduce to: a matmul is the wrong shape, the weights don’t fit in cache, or you spend more time moving them than multiplying.

The short answer

modern neural net ≈ a stack of (matmul + cheap nonlinearity)

Picture to keep: a factory floor of thousands of identical multiply-and-add machines, and one conveyor belt feeding them. Matmul is the job shaped so that a single trip down the belt keeps every machine busy; most other operations leave much of the floor idle waiting on the belt. (Where the picture breaks: the belt isn’t the only way to waste the floor — a matmul that’s too small, or one tiled badly, leaves the machines idle too. High intensity is a property of the kernel, not automatically of the operation.)

A neural network layer, stripped of branding, is: take a vector of inputs, multiply it by a learned weight matrix, add a bias, apply a cheap elementwise function (ReLU, GELU). Stack a hundred of those. That’s it. Even attention is matmul wearing a different hat — attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V is two matmuls with a softmax in between, and producing Q, K, and V in the first place is three more. (Softmax itself isn’t elementwise — each output depends on the whole row — but it’s cheap next to the multiplies around it.)

So when you train or run a model, you are mostly asking your hardware to multiply matrices. Faster matmul = faster everything.

How it works

The naive expectation. Hardware gets faster every year, so whatever operation a researcher invents will get faster too. Design the model you think is right; the silicon will catch up. That was roughly true through the 1990s, and it’s the intuition most people still carry.

Why it breaks. Compute got cheap much faster than memory got fast. That gap is the memory wall, and it means the number that decides an operation’s fate isn’t how many FLOPs it does — it’s how many FLOPs it does per byte it drags out of memory. Adding two vectors does one FLOP for two loads and a store: the chip finishes instantly and then sits idle waiting on memory. Feed a modern accelerator that workload and you use a rounding error’s worth of its arithmetic capability. Every operation that isn’t dense enough to hide its own memory traffic quietly fell off the Pareto frontier — not because it was slower in theory, but because the machine spent its time waiting.

What survives the filter. Multiplying two N×N matrices is O(N³) arithmetic against O(N²) data. Written well — tiled so that a block of the matrix is loaded once and reused across many multiplies — every value pulled from HBM earns its trip many times over. That ratio, arithmetic intensity, grows with N: matmul is the rare operation that gets better at hiding the memory wall the bigger you make it. That’s the survivor — and note the “written well,” because a badly tiled matmul falls right back onto the memory wall like everything else.

Then the loop tightens. Around the late 2000s and early 2010s, deep learning beat hand-engineered features at vision, then speech, then language, and the winning models all shared a shape: many layers, each layer a big matmul. Convnets (a convolution can be unrolled into a matmul via im2col, and for years that was the common implementation), RNNs, transformers — different architectures, same primitive. Then GPUs added dedicated matmul units (NVIDIA shipped tensor cores in Volta, 2017), and the gap widened again: a tensor-core matmul can be several times faster than the same FLOPs run through generic vector lanes. So a model built out of matmuls runs far faster than the same FLOP budget spent on anything else, and researchers followed the speed. Hardware selected for matmul, then architectures selected for the hardware.

The seam. Matmul’s dominance isn’t a law of nature — it’s a feedback loop between hardware and architectures. There are operations (sparse attention, structured matrices, state-space models) that are mathematically cheaper but currently slower in practice because the hardware isn’t shaped for them. Whether that loop ever breaks — whether something dethrones matmul — is genuinely open. Mamba and friends are the most credible recent challengers, but as of early 2026, transformers still win at the frontier, and the frontier still runs on matmul.

One more thing worth naming: the theoretical exponent of matrix multiplication is below 3 — Strassen’s algorithm is O(N^2.807), and there’s a long line of asymptotically faster algorithms going down toward ~2.37. Almost none of them show up in production ML kernels. The usual explanation — worse constants, worse numerical stability, worse cache and tiling behaviour at the sizes we actually run — is the standard account rather than something I can point at a benchmark for. What’s not in doubt is what ships: production GEMM is the classical cubic algorithm, tiled and hand-tuned to within an inch of its life. Which is the whole thesis in one sentence: the asymptotically cheaper algorithm loses to the one shaped like the machine.

So: back to the opening. Why does the grunt work behind ChatGPT, Midjourney, and Face ID collapse onto the same operation, when the three problems have nothing in common? Not because anyone chose it. Because the memory wall filters hard, and matmul is the shape that comes through it.

You started with modern neural net ≈ a stack of (matmul + cheap nonlinearity). What did this post add? — + and the memory wall, not elegance, is what selected it. Matmul didn’t win on FLOP count; it won on FLOPs-per-byte, and every “why is my model slow” question is really “which of my operations lost that ratio.”

Check yourself

Before you go — a colleague replaces a big dense layer with a sparse one that does 10× fewer FLOPs, and reports the model got slower on the same GPU. What happened?

And one more — a 70B model generating one token at a time for a single user gets nowhere near the GPU’s advertised FLOP number, but the same GPU serving 64 users at once gets much closer. Why would batching change the arithmetic at all?

Famous related terms

• GEMM — GEMM = General Matrix Multiply — the BLAS-3 routine C = αAB + βC. Almost every ML framework’s matmul call eventually lands in a hand-tuned GEMM kernel.
• Arithmetic intensity — arithmetic intensity = FLOPs / bytes moved — the number that decides whether a kernel is compute-bound or memory-bound. Matmul wins by being high.
• Roofline model — roofline ≈ plot of arithmetic intensity vs achievable FLOPs — a back-of-envelope diagram for “is this kernel as fast as it can be?”.
• Tensor cores / MXUs — tensor core = matmul ASIC bolted onto a GPU SM — why a 2026 GPU does ML matmul much faster than its raw FP32 number suggests.
• im2col — im2col = unroll a convolution into a matmul — the trick that turned convnets into a matmul workload, which is why GPUs ate computer vision.

Going deeper

• Williams, Waterman, and Patterson, “Roofline: An Insightful Visual Performance Model” — for “how do I tell whether a given kernel is compute-bound or memory-bound?”, which is the question this whole post reduces to.
• Horace He, “Making Deep Learning Go Brrrr From First Principles” — for “what do I actually do about it?”, walking through compute-bound vs memory-bound vs overhead-bound on real PyTorch code.
• Rabbit hole: recent results on the matrix-multiplication exponent ω — for “is O(N³) really the best we can do?” (No, in theory. Yes, in every production kernel.)