Why AI Video Training Is Limited by Tokens and Labels, Not Playback Speed

AI systems can process video faster than its playback speed. The harder problem is representing footage efficiently and teaching models what its frames mean. That distinction helps explain why video is attracting attention as text becomes harder to scale, while also showing why adding more footage alone won’t solve the training data problem.

Video processing doesn’t run at playback speed

A model doesn’t need to play a video from beginning to end as a person would. Gemini, for example, samples frames at roughly one per second by default and processes the audio alongside them. Those inputs are converted into tokens, the units the model handles. Since much of this work can run in parallel, a 90-minute recording doesn’t automatically require 90 minutes of processing.

Google’s published estimate puts video processing overhead at about two seconds plus roughly half a second for each minute of footage. Applied to a 90-minute video, that comes to less than a minute of overhead, according to How does Gemini process videos?. The estimate describes the processing overhead; it doesn’t mean every video task, from upload to a useful answer, will finish in that time.

Training follows a different path from playback, too. Video can be split into frames and encoded in batches before training begins, much as text is converted into tokens ahead of model training. There’s no requirement for a model to sit through the footage in real time. Longer clips still mean more material to process, but runtime alone doesn’t set the pace.

Redundant frames make video expensive

The bigger constraint is the volume of tokens needed to represent a clip. Google’s pricing breakdown estimates that an hour of low-resolution video uses about 475,000 tokens: roughly 360,000 for frames and 115,000 for audio. An hour of spoken-word transcript, by comparison, might use 10,000 to 15,000 tokens, according to Understand and count tokens.

That gap reflects how video carries information. Adjacent frames often show almost the same scene: a person speaking may barely change position from one second to the next. Processing each frame can therefore spend tokens describing repeated visual details, even when the meaningful action is brief.

Google’s agentic video processing offers one response: the model can select relevant portions instead of treating every frame as equally useful. The company’s reported reduction of up to 88% in token use comes from avoiding some of that redundant processing, as described in Google cuts Gemini video analysis tokens by up to 88%. That’s a change in the cost of representing a video, not evidence that the model has overcome a playback-speed limit.

Footage lacks the labels that text carries

Even an efficient representation can leave a model without the context needed to learn from a clip. Text often describes itself: a sentence or paragraph supplies language that can be used as a training signal. Video usually doesn’t explain what is happening. A clip of someone lifting a cup and setting it down may have a broad caption, but not an account of the action, its sequence or the cause and effect involved.

That shortage makes video supervision difficult. Models need to learn patterns across time, not just recognize objects in individual frames. They may need to distinguish an action from its result or infer how objects moved, yet much online footage has no detailed annotation that spells out those relationships.

One direction is self-supervised learning, which aims to draw training signals from the video itself instead of relying on human labels for every example. The approach shifts the research challenge: systems need to learn useful structure from unlabeled sequences. For practitioners, that means the supply of footage and the supply of training-ready data are different things.

Why the push toward video is growing

Demand for video data is rising as the supply of high-quality web text tightens. One widely cited estimate puts annual growth in scrapable text below 10%, while training data demand roughly doubles each year; the projections cross around 2028, according to The data wall is important. Those figures are estimates, but they capture the pressure behind the search for other sources.

Video offers a vast pool of sensory information, though turning it into model input is costly and difficult. Yann LeCun has argued that a four-year-old encounters more raw sensory data than any large language model’s text training set, largely through vision, as discussed in The Most Expensive Disagreement in AI. The comparison points to a type of information text doesn’t capture well; it doesn’t show that current systems can learn from video with the same ease.

The next gains depend on useful signal

AI has reached video, but the scale of that access shouldn’t be confused with an ability to learn efficiently from every clip online. Frame sampling and parallel processing address the time required to handle footage. Token reduction addresses the cost of carrying repeated visual information. Self-supervised methods target the lack of detailed labels.

For teams building video systems, the practical question is how much useful information a pipeline can extract from a clip, and what it costs to do so. Better selection of frames and stronger ways to learn temporal structure could make existing footage more valuable. The next meaningful progress will come from those improvements, not from making a model “watch” faster.

Discover how Fastly's edge cloud platform can help you optimize performance and scale data-heavy workloads.