How does an AI 'see' a video?

Upload an hour-long video to Gemini and ask what happens at minute 40 — it answers. But no model 'watches' anything. It reads a flipbook, and the flipbook is missing most of the pages.

The picture version

The whole idea in seven pictures, for someone who has never thought about what a video is to a computer. The prose below has the numbers, the caveats, and the seams the pictures skip.

Why it exists

You upload a recording of an hour-long lecture to Gemini and ask “where does she start talking about black holes?” — and it hands you a timestamp. You ask your phone’s photo app for “the clip where the dog jumps into the pool” and it finds the exact video. From the outside it looks like the AI sat down and watched the footage.

But think about what a video actually is. Even a single image is expensive for a model: one frame becomes a few dozen to a few hundred tokens in the model’s context window. Video is that, thirty times per second. A ten-minute clip at 30 fps is 18,000 frames; at roughly 258 tokens a frame that’s about 4.6 million tokens — several times more than the largest production context windows, for ten minutes of footage. An hour would be pure fantasy. So the puzzle is real: how does a machine that can’t afford to look at every frame answer questions about an hour of video?

The answer is that it never looks at every frame. In the recipe that production systems document, the model reads video the way you’d skim a flipbook with most pages torn out: keep about one frame per second (Gemini’s documented default), turn each surviving frame into image tokens, and lay those token blocks into the context in time order, often with timestamps written next to them. “Watching” is attention over a sparse, ordered stack of stills. That lecture recording is the example this post keeps coming back to.

Why it matters now

Video input is now ordinary: Gemini takes hour-long uploads, ChatGPT’s Advanced Voice mode can share your phone’s camera, and computer-use agents work from streams of screenshots — video in all but name. Three places the flipbook mechanic shows up concretely:

• Cost and limits are frame arithmetic. Google’s docs put default video processing at ~300 tokens per second of footage — which is why a model with a 1M-token context tops out around an hour of video, and why long uploads get billed like small books.
• Fast motion is invisible by design. At one frame per second, anything that happens between samples never reaches the model. Google’s own documentation warns that fast action sequences lose detail at 1 fps — that’s not a model weakness, it’s the sampling.
• “Did it hear the video?” is a separate question. The sampled frames are silent. Whether the model knows what was said depends on whether the system also tokenizes the audio track — some do, many don’t.

The short answer

video input = ~1 frame per second sampled from the clip + each frame tokenized like an image + the blocks laid in time order (with timestamps) in one sequence

Picture to keep: a contact sheet — one printed still per second of footage, laid out in order with the clock time written under each — spread on a desk where the model can look at any of them at once. Not a screen playing; a wall of stills with the gaps between them simply gone.

That’s the whole trick. A video is decomposed into a sparse sequence of stills; each still goes through the exact image pipeline you may already know — patches → vision encoder → projector → image tokens; and the resulting blocks sit in the context one after another, so attention can relate “the man picks up the ball” at 00:12 to “the dog has the ball” at 00:19. The compression line is lossy in two honest ways: some systems also interleave audio tokens from the soundtrack, and some research models tokenize small stacks of frames together instead of single stills. Both refinements are covered below.

How it works

A video was never anything but frames (plus a soundtrack)

There is no “video” data type to see. Once decoded for playback, a video is a stream of still images — typically 24 to 60 per second, which your brain fuses into motion — plus an audio track stored alongside. (On disk, codecs mostly store differences between frames rather than full images; the full frames are reconstructed at decode time.) So “seeing video” reduces to two already-solved problems: turning images into tokens, and (optionally) turning audio into tokens. The only genuinely new problem is scale: there are far too many frames to afford.

Break 1: every frame is unaffordable → keep one per second

The naive design — decode the lecture and hand the model all 108,000 frames — died back in the hook, on arithmetic alone. The saving grace is that consecutive frames are nearly identical — a person mid-sentence looks the same at frame 301 and frame 302. This temporal redundancy is the same reason video files compress so well, and it means most frames can be dropped with little loss of content, even though all sense of smooth motion dies.

Production systems are unusually public about this step. Google’s Gemini docs state that video is sampled at 1 frame per second by default, each frame costing 258 tokens at standard resolution (66 at low resolution), with a 1M-token context fitting about an hour of video — three hours in low-resolution mode. Treat these as the right order of magnitude rather than a formula: Google’s own pages quote slightly different per-second figures in different places (~300 tokens/second on the video page, 263 in the token guide), and the numbers move as the models do. Run the arithmetic and the fantasy becomes tractable: that ten-minute clip drops from 18,000 frames to 600, from ~4.6M tokens to ~155,000 visual tokens (plus another ~19,000 if the soundtrack is tokenized too — more on that below). Still expensive — video is the most token-hungry thing you can put in a context — but possible.

Break 2: a still isn’t tokens either → the image pipeline, unchanged

Sampling leaves 3,600 stills of the lecture, which the model still can’t read — a transformer consumes vectors, not pixels. Fortunately this problem was already solved for single images. Every sampled frame now goes through the standard image pipeline: cut into patches, encoded by a vision transformer, projected into the language model’s embedding space. This post won’t re-derive that machinery — the image post covers it — but one consequence matters here: everything that’s true of image input is true of every frame. Tiny on-screen text smaller than a patch gets smeared; low-resolution mode makes that worse. A blurry frame yields blurry tokens.

Break 3: a bag of stills isn’t a video → time gets written back in

Tokenize each frame independently and you’ve thrown away the thing that made it footage. “Where does she start talking about black holes?” is unanswerable from an unordered pile of lecture stills, however many there are — order and timing are the question. Two things restore them. First, the frame blocks are laid into the context in chronological order, so sequence position itself carries “before” and “after,” the same way word order does. Second, systems commonly attach explicit timestamps to the frames — Gemini’s docs have you reference moments as MM:SS, which only works because the model can associate frames with clock time. That’s also how “what happens at minute 40?” becomes answerable: the frames sitting near the 40:00 labels are right there in the context to be looked up.

There’s a more radical research answer worth knowing about — a different family of models from the flipbook pipeline above. Video transformers from 2021 tokenize the video natively instead of sample-then-stacking: ViViT extracts spatio-temporal tokens, in one variant cutting the clip into tubelets, little space-time boxes, so a single token can carry motion, while TimeSformer keeps per-frame patches but splits attention in two — one pass across time between frames, another across space within a frame. That factorization exists because attention cost grows with the square of sequence length, and video-length sequences are enormous. A caveat, stated plainly: the internals of closed production models aren’t public, so whether any given assistant uses pure frame sampling or something tubelet-shaped under the hood is not something I can verify. The frame-sampling account above is what the public docs and the open-model recipes describe.

Break 4: the frames are silent → audio is a separate stream or nothing

Here’s the one that catches people out with a lecture recording. Everything above is a picture pipeline. Sampled frames carry no sound. If the system wants the model to know what was said, it must tokenize the audio track separately — Gemini does, at a documented 32 tokens per second, alongside the visual stream. Plenty of vision-language models, especially open ones, process only the frames — which produces a disorienting failure mode: the model describes the argument in the video perfectly and has no idea what either person said. When a video answer seems deaf, it often literally is.

Why the failure modes look the way they do

• Fast actions vanish. A card trick, a golf swing’s wrist position, the exact moment of a collision — if it happens between two one-second samples, it was never in the context at all. The model isn’t bad at fast motion; it never saw it.
• Counting repetitions is unreliable. “How many push-ups did he do?” requires reconstructing a continuous action from sparse stills — some reps fall between samples, and near-identical frames are hard to keep tallied, the same family of weakness as counting letters in a word.
• Long videos trade detail for length. Fitting three hours instead of one means switching to the low-resolution mode — 66 tokens per frame instead of 258, in Gemini’s documented tiers — so fine detail like small text and distant faces degrades exactly when the video is long. And a video that fills the context is subject to the usual lost-in-the-middle effect: recall is strongest near the start and end of the footage.
• Precise timing is fuzzy. “At exactly 12:07.4” is below the sampling resolution. At the default rate, the model’s clock ticks once per second.

You started with video = frames sampled + tokenized like images + laid in time order. What did the failure chain add? — + a separate audio stream, or silence, and that’s the one people forget. Every other limit in the list above follows from the sampling rate, but “it didn’t know what she said” follows from a stream that may never have been tokenized at all.

Check yourself

Before you go — you ask about a two-minute clip of a card trick and the model describes the magician’s patter and hands perfectly, but gets the crucial sleight-of-hand wrong. Would uploading the same clip at 4K help?

And one more — you upload a three-hour recording and the model answers well about the opening and the closing but gets vague about the middle. Two different mechanisms in this post could each cause that. What are they, and how would you tell them apart?

Famous related terms

• Frame sampling — frame sampling = keep ~1 frame per second + drop the rest — the single decision that makes video affordable and causes most video failure modes.
• Image tokenization — image input = patches → vectors → tokens beside the words — the per-frame machinery this whole post leans on.
• Tubelet / spacetime patch — tubelet = an image patch extended through a few frames of time — how research video transformers (ViViT) and video generators (Sora) tokenize motion itself.
• Factorized space-time attention — factorized attention = attend within a frame + attend across frames, separately — TimeSformer’s trick for taming quadratic cost on video-length sequences.
• Optical flow — optical flow = a per-pixel arrow field of motion between two frames — how pre-transformer video models fed motion to a network explicitly, most famously the two-stream networks of 2014.
• Temporal redundancy — temporal redundancy ≈ the reason video compresses — consecutive frames are nearly identical, which is what makes throwing most of them away survivable.

Going deeper

• Gemini API video understanding docs — the primary source, for “what does this actually cost me, and what will it miss?”: sampling rate, per-frame token prices, audio tokenization, and how much footage fits in a context.
• ViViT: A Video Vision Transformer (Arnab et al., 2021) — answers “what would it look like to not cheat with frame sampling?”, working through spatio-temporal tokenization and the factorized-attention variants that keep it affordable.
• Video generation models as world simulators (OpenAI, 2024) — the rabbit hole, for “if video is just spacetime patches, can you run the pipeline backwards?” — which is roughly what a video generator does.