AnimWeb-Bench: Benchmarking Animated Webpage Code Generation from Videos

Animation fidelity is low across the board. Even GPT-6 Astra, the strongest model, scores below 0.46 on every animation metric, and ten of the remaining thirteen models stay below 0.35. For every model, each animation score falls below each static score, and Velocity is the lowest of all eight metrics. Because the two dimensions are computed differently, their absolute values are not directly comparable. Two further observations, however, show that the gap reflects motion rather than scale. Animation metrics respond to the kind of motion on a page while static metrics do not, and human annotators prefer every evaluated model over the baseline less often on animation than on static appearance.

Current models fail to infer and encode component-level dynamics in three ways. First, some animated components are missing from the generated page altogether, and their unmatched trajectories lower all four animation scores. Second, even when a component is animated, its trajectory or manner of change may deviate from the reference. For example, sliding along a different path or scaling instead of sliding, which lowers Position and Size. Third, and most subtle, is the temporal behavior of motion, namely when and how fast each component moves. Components may move faster, slower, or in a different direction than in the reference, which lowers Velocity. They may also appear at the wrong time: elements that the reference reveals gradually or one after another may appear all at once in the generated page, leaving time steps at which only one of the matched components is present and thus lowering all four animation scores. Faithfully recovering how a webpage moves thus remains an open challenge for current models.

GPT-6 Astra leads all eight metrics, and Claude Fable 5.1 ranks second on every metric. Among open-weight models, Kimi K3 ranks third in static Text and CLIP, surpassing GPT-6 Sol, and fifth across all animation metrics, outperforming Claude Opus 5 and approaching GPT-5.6 Sol. DeepSeek V4.1 Flash and GLM 5.3 Flash perform on par with the proprietary Grok 4.7 and Qwen3.8 Max (0902), and all four outperform the two Gemini models and Doubao-Seed-2.1-pro across all metrics.

We compare each agent with its own underlying MLLM, and find that the benefit of agentic generation depends on both the agent and the input format. Qwen Code outperforms Qwen3.8 Max (0902) on all eight metrics under both settings, with relative gains of 4.3%–24.7% under Frames and 9.7%–36.1% under Video. Claude Code shows a reversal: it falls below Claude Opus 5 on all eight metrics under Frames, yet surpasses it on all eight under Video, with relative gains of 7.9%–21.7%. Codex, by contrast, trails GPT-5.6 Sol on six metrics under Frames and seven under Video, with the remaining metrics changing by at most 1.4%.

Comparing the two input formats, Video yields higher scores than Frames on all eight metrics for both Claude Code and Qwen Code, suggesting that video is generally the more informative input. Unlike the fixed 2 fps frames, a video allows an agent to choose its own sampling rate, for example, sampling densely around transitions, and can thus reveal intermediate states and timing that sparse frames miss. For Codex, however, Video outperforms Frames on only three metrics, indicating that this flexibility does not automatically translate into better reconstruction. The way in which an agent perceives visual input is therefore critical to its performance.

To understand why Codex lags behind, we examine its execution traces under Video. We observe that Codex often extracts frames from the video, tiles them into contact sheets (e.g., nine frames per image), and relies on these sheets rather than on the individual frames. Such reduced tiles may obscure fine visual details and subtle changes between frames. We therefore introduce two additional Video settings for Codex. In Video + Frame Inspection (Video+FI), the prompt instructs Codex to also inspect each source frame individually at its original resolution whenever it builds a contact sheet. Video + Frame Inspection + Render Feedback (Video+FI+RF) further provides a recording tool that renders the generated page into a video, allowing Codex to compare its output with the reference and revise the code iteratively.

The two enhancements improve Codex in different ways. Compared with Video, Video+FI improves all eight metrics by 7.3%–22.0% on the static metrics and 11.6%–15.6% on the animation metrics, and outperforms GPT-5.6 Sol on every metric. Building on Video+FI, Video+FI+RF further raises all four animation metrics by 4.0%–7.0%, whereas the static metrics change by only −0.3% to 2.5%. Frame inspection thus improves overall fidelity, while render feedback mainly benefits animation, likely because comparing the rendered page with the reference video exposes motion errors that are hard to detect from the code alone.

With both enhancements, Codex outperforms GPT-6 Sol on all eight metrics and Claude Fable 5.1 on six, second only to GPT-6 Astra on most metrics. These results suggest that the bottleneck of agentic reconstruction lies not only in code generation but also in perception and verification: an agent improves substantially once it observes the reference closely and checks how its own page actually moves. Because the frame-inspection instruction was derived from Codex's traces on this benchmark, we treat Video+FI and Video+FI+RF as diagnostic interventions that locate the bottleneck rather than as tuned methods.

Animation fidelity declines as the animations involve more motion. On animation metrics, Visibility Only scores the highest, followed by Property Change, while Spatial Motion scores the lowest, with a Velocity only 56% that of Visibility Only. This ordering reflects the demands of each group. Visibility changes only require models to determine when elements appear, and property changes alter the size, color, or content of components with limited motion, whereas spatial motion requires recovering how each element travels over time.

The static metrics behave differently: between Spatial Motion and Property Change, they differ by −0.8% to +7.2% with no consistent direction, whereas all four animation metrics drop by 8.7%–13.4%. Animation categories thus mainly affect how well models reproduce motion rather than appearance, which shows that Animation Fidelity captures differences that the static metrics miss.

Among individual animation types, the most challenging ones change the page rapidly and continuously, often across many elements at once, such as Card Flip, Twinkling, Text Scramble, Text Fragmentation, Shake, and Numeric Counter Animation. Their intermediate states shift at nearly every frame, making the timing difficult to infer from the reference video and to express precisely in code.

The easiest types, such as Font Weight Animation, Icon State Transition, and Text Substitution, are instead discrete transitions between two clearly visible states, requiring models only to identify the start and end states and connect them with a simple transition. This contrast is also reflected in the experimental results, where webpages with rapid, continuous changes score lower on all metrics than those with discrete state transitions.

In both dimensions, human evaluators rank the models in nearly the same order as our automatic metrics. GPT-6 Astra is preferred over the baseline on 94% of the webpages for static appearance and 92% for animation, followed by Claude Fable 5.1, Kimi K3, and Grok 4.7, whose win rates decline in the same order as their automatic scores.

DeepSeek V4.1 Flash and GLM 5.3 Flash perform close to the baseline, with comparable win and loss rates, consistent with their automatic scores differing from those of Qwen3.8 Max (0902) by less than 0.03 on every metric. Gemini 3.1 Pro Preview and Doubao-Seed-2.1-pro lose to the baseline on at least half of the webpages in both dimensions, again matching their low automatic scores. All eight metrics correlate positively with human preferences, with Spearman's ρ ranging from 0.667 to 0.737.

Compared with static appearance, every model obtains a lower win rate and a higher tie rate on animation. The rise in ties is particularly pronounced for weaker models: the tie rates of DeepSeek V4.1 Flash, GLM 5.3 Flash, Gemini 3.1 Pro Preview, and Doubao-Seed-2.1-pro rise from 12%–20% to 30%–40%.

When neither animation closely follows the reference, annotators find little basis to prefer one over the other, echoing our finding that reproducing animation remains challenging.