Heads up: posts on this site are drafted by Claude and fact-checked by Codex. Both can still get things wrong — read with care and verify anything load-bearing before relying on it.
why → how

How does an AI 'see' a video?

Upload an hour-long video to Gemini and ask what happens at minute 40 — it answers. But no model 'watches' anything. It reads a flipbook, and the flipbook is missing most of the pages.

AI & ML intermediate Aug 6, 2026

Why it exists

You upload a recording of an hour-long lecture to Gemini and ask “where does she start talking about black holes?” — and it hands you a timestamp. You ask your phone’s photo app for “the clip where the dog jumps into the pool” and it finds the exact video. From the outside it looks like the AI sat down and watched the footage.

But think about what a video actually is. Even a single image is expensive for a model: one frame becomes a few dozen to a few hundred tokens in the model’s context window. Video is that, thirty times per second. A ten-minute clip at 30 fps is 18,000 frames; at roughly 258 tokens a frame that’s about 4.6 million tokens — several times more than the largest production context windows, for ten minutes of footage. An hour would be pure fantasy. So the puzzle is real: how does a machine that can’t afford to look at every frame answer questions about an hour of video?

The answer is that it never looks at every frame. In the recipe production systems document, the model reads video the way you’d skim a flipbook with most pages torn out: keep about one frame per second (Gemini’s documented, adjustable default), turn each surviving frame into image tokens, and lay those token blocks into the context in time order, often with timestamps written next to them. “Watching” is attention over a sparse, ordered stack of stills.

Why it matters now

Video input went mainstream in the last two years: Gemini takes hour-long uploads, ChatGPT’s Advanced Voice mode can share your phone’s camera, and computer-use agents work from streams of screenshots — video in all but name. Three places the flipbook mechanic shows up concretely:

The short answer

video input = ~1 frame per second sampled from the clip + each frame tokenized like an image + the blocks laid in time order (with timestamps) in one sequence

That’s the whole trick. A video is decomposed into a sparse sequence of stills; each still goes through the exact image pipeline you may already know — patches → vision encoder → projector → image tokens; and the resulting blocks sit in the context one after another, so attention can relate “the man picks up the ball” at 00:12 to “the dog has the ball” at 00:19. The compression line is lossy in two honest ways: some systems also interleave audio tokens from the soundtrack, and some research models tokenize small stacks of frames together instead of single stills. Both refinements are covered below.

How it works

A video was never anything but frames (plus a soundtrack)

There is no “video” data type to see. Once decoded for playback, a video is a stream of still images — typically 24 to 60 per second, which your brain fuses into motion — plus an audio track stored alongside. (On disk, codecs mostly store differences between frames rather than full images; the full frames are reconstructed at decode time.) So “seeing video” reduces to two already-solved problems: turning images into tokens, and (optionally) turning audio into tokens. The only genuinely new problem is scale: there are far too many frames to afford.

Step 1: throw away almost every frame

The saving grace is that consecutive frames are nearly identical — a person mid-sentence looks the same at frame 301 and frame 302. This temporal redundancy is the same reason video files compress so well, and it means most frames can be dropped with little loss of content, even though all sense of smooth motion dies.

Production systems are unusually public about this step. Google’s Gemini docs state that video is sampled at 1 frame per second by default (the rate is adjustable), each frame costing 258 tokens at standard resolution (66 at low resolution), with a 1M-token context fitting about an hour of video — three hours in low-resolution mode. Run the arithmetic and the fantasy becomes tractable: that ten-minute clip drops from 18,000 frames to 600, from ~4.6M tokens to ~155,000 visual tokens (plus another ~19,000 if the soundtrack is tokenized too — more on that below). Still expensive — video is the most token-hungry thing you can put in a context — but possible.

Step 2: each surviving frame becomes image tokens

Every sampled frame now goes through the standard image pipeline: cut into patches, encoded by a vision transformer, projected into the language model’s embedding space. This post won’t re-derive that machinery — the image post covers it — but one consequence matters here: everything that’s true of image input is true of every frame. Tiny on-screen text smaller than a patch gets smeared; low-resolution mode makes that worse. A blurry frame yields blurry tokens.

Step 3: time gets written back in

A bag of stills isn’t a video — order and timing are the point. Two things restore them. First, the frame blocks are laid into the context in chronological order, so sequence position itself carries “before” and “after,” the same way word order does. Second, systems commonly attach explicit timestamps to the frames — Gemini’s docs have you reference moments as MM:SS, which only works because the model can associate frames with clock time. That’s also how “what happens at minute 40?” becomes answerable: the frames sitting near the 40:00 labels are right there in the context to be looked up.

There’s a more radical research answer worth knowing about — a different family of models from the flipbook pipeline above. Video transformers from 2021 tokenize the video natively instead of sample-then-stacking: ViViT cuts it into tubelets, little space-time boxes, so a single token can carry motion, while TimeSformer keeps per-frame patches but splits attention in two — one pass across time between frames, another across space within a frame. That factorization exists because attention cost grows with the square of sequence length, and video-length sequences are enormous. A caveat, stated plainly: the internals of closed production models aren’t public, so whether any given assistant uses pure frame sampling or something tubelet-shaped under the hood is not something I can verify. The frame-sampling account above is what the public docs and the open-model recipes describe.

Step 4: the frames are silent

Sampled frames carry no sound. If the system wants the model to know what was said, it must tokenize the audio track separately — Gemini does, at a documented 32 tokens per second, alongside the visual stream. Plenty of vision-language models, especially open ones, process only the frames — which produces a disorienting failure mode: the model describes the argument in the video perfectly and has no idea what either person said. When a video answer seems deaf, it often literally is.

Why the failure modes look the way they do

Going deeper