Vidu S2: The World's Leading Real-Time Interactive and Real-Time Editing Model

Try Vidu S2Enterprise APITechnical Report

Traditionally, video generation models have produced predetermined clips: users enter a prompt, wait for inference to complete, and receive a finished video.

Vidu S1 took AI video from offline generation to real-time interaction. Instead of producing a fixed clip, the model interprets user input as the video unfolds, responds to new voice commands, and continuously generates visual feedback.

Today, we are officially launching our next-generation model, Vidu S2.

Vidu S2 comprises Vidu S2-Avatar for real-time interaction and Vidu S2-Editing for real-time video editing. It also explores real-time spatial video generation and editing. End-to-end R&D for the S-series is led by Jintao Zhang, a doctoral researcher advised by Professor Jun Zhu and Head of Streaming Video Generation and Inference at Shengshu Technology.

Compared with Vidu S1, Vidu S2 expands real-time video beyond continuous generation and interaction to encompass complex motion, dynamic references, real-time editing, and spatial video. S1 showed that video could be generated and made responsive in real time. S2 answers a broader question: as a video unfolds, can it be continuously understood, controlled, and transformed in real time?

Video is no longer merely an output. It is becoming a responsive visual world that evolves continuously.

Built for real-time interaction and real-time editing

Real-time video is about more than low latency. A true real-time interaction and editing model must continuously ingest user input; reason over the current frame and interaction history; maintain long-term consistency across characters, motion, and scenes; adapt subsequent content to new voice commands, images, and instructions; preserve the source video's motion, timeline, and visual structure during editing; and stream high-quality video with low latency.

To meet these requirements, Vidu S2 introduces full-stack upgrades spanning data, model architecture, training algorithms, inference acceleration, and serving infrastructure.

This technical foundation supports two core product capabilities:

Vidu S2-Avatar: Real-time interactive model

• Vidu S2-Avatar: Supports spoken conversation and lets users direct complex avatar actions—including dance—through voice commands.
• Reference-guided live interaction: Uses instructions and reference images to let a character interact with objects in real time—for example, picking up and presenting a specified item, putting on a referenced garment, or moving into a referenced environment.
• 720P+ output: Generates real-time avatar video at 720P and above.
• Offline avatar generation: Produces high-quality, pre-rendered avatar videos that deliver scripted speech or perform predefined actions.

Vidu S2-Editing: Real-time editing model

• Applies the visual style of a reference image to a live video stream in real time.
• Changes a subject's clothing in real time using a garment reference image.
• Replaces the person in a live video stream with a referenced subject in real time.
• Replaces the background in real time using a reference image.

Real-time video models are thus evolving from continuous content generators into visual systems capable of continuous generation, interaction, and editing.

Vidu S2-Avatar: Extending the boundaries of real-time interaction

Vidu S1 already enabled real-time video generation and continuous interaction driven by voice instructions.

In S1, however, real-time character capabilities centered primarily on conversational avatars: output resolution was limited to 540P, character references remained largely fixed once streaming began, and control over large full-body movements such as dance was still limited. Vidu S2-Avatar builds on that foundation to broaden the scope of real-time interaction.

Stronger motion following: from voice-driven expression to voice-controlled behavior

To teach the model more complex behaviors, we add high-quality footage of solo dance performances alongside 2D and 3D animation data.

Rather than discarding videos with smooth camera movement, we apply a background-stabilization operator that mitigates the effect of camera motion on background consistency. The model can therefore learn a richer range of body movements without sacrificing long-term scene stability. We also redesign the video-captioning scheme.

Conventional captions focus primarily on what appears in a frame. Vidu S2 instead uses temporally structured descriptions that explicitly capture when actions occur, in what order, and with what outcome.

For example:

0–2.8 seconds: The character raises their right hand and waves at the camera.
2.8–5 seconds: The character lowers their arm, smiles, and gives a slight nod.

The model learns more than static semantics. It learns which action occurs first, what follows, and how the resulting state should persist.

This enables Vidu S2-Avatar to model complex motion and continuous behavior more accurately, while responding more quickly to action transitions and updated instructions.

Dynamic references: enabling interaction to evolve continuously

Traditional generative models typically require users to define all reference conditions before generation begins.

Once video generation starts, the character, clothing, objects, and scene are largely fixed.

Vidu S2-Avatar also allows users to introduce new reference images at any point during streaming.

Users can upload an object for the character to interact with in real time, a garment for the character to put on, or a new scene for the character to enter.

Reference images are no longer fixed conditioning inputs supplied only before generation. They become dynamic inputs throughout the real-time interaction.

To make this process more reliable, we introduce a VLM Agentic System.

Users can provide text, speech, an initial image, and reference images. The VLM agent interprets these inputs alongside the current frame to generate prompts for subsequent steps and continuously validate the model's output.

For example, when a user says:

“Pick up the cup, then smile.”

After the character picks up a cup, the system preserves the fact that the cup remains in the character's hand. A subsequent instruction to smile therefore does not undo the previous action.

Interaction therefore evolves from User Input → Model Generation to:

User Input → Current-State Understanding → Action Execution → Result Verification → State Maintenance → Response to the Next Instruction

This gives real-time characters a more persistent understanding of state and more coherent transitions between actions.

Self-Replay Forcing: Improving long-horizon real-time generation stability

One of the greatest technical challenges in real-time video is error accumulation.

Conventional offline diffusion models can denoise repeatedly over an entire video. Streaming generation, by contrast, must proceed chunk by chunk, with each new chunk conditioned on the model's previous outputs. Even a minor identity inconsistency, motion error, or visual artifact can therefore propagate over time, eventually causing identity drift, motion discontinuities, or severe visual degradation.

To address this problem, we propose Self-Replay Forcing (SRF). The model first performs a long-horizon autoregressive rollout under realistic inference conditions, producing a trajectory of its own generated frames. It then samples contiguous segments from that trajectory, re-noises them with diffusion noise, and performs differentiable causal replay. Compared with conventional Self-Forcing, SRF trains on histories that more closely match the model's actual inference distribution. Multiple blocks within a replay segment also remain in the same computation graph, allowing losses from later blocks to backpropagate through earlier ones.

Put simply, the model learns not only how to generate the next segment, but also how to prevent errors in its own earlier outputs from compounding over time.

We further combine DPO with Streaming NFT to improve visual quality, motion naturalness, and instruction following during streaming generation.

720P · 25–42 FPS: Decoupling temporal dynamics from spatial detail

Vidu S2-Avatar uses a lightweight, single-step refiner to recover high-resolution detail from the backbone's low-resolution output. The backbone models motion and long-range temporal dynamics, while the refiner reconstructs local appearance details. This division of labor enables 720P real-time output with limited additional compute.

Training data is filtered by effective visual fidelity rather than nominal resolution alone, with texture detail, edge clarity, and compression degradation considered jointly. Two videos labeled 1080P can retain very different amounts of usable detail depending on how heavily they have been compressed.

For inference, we build on techniques from TurboDiffusion and TurboServe, combining hybrid attention, low-precision computation, and operator fusion to reduce compute overhead. In the editing pipeline, the encoder, generative backbone, refiner, and decoder are scheduled on a shared execution timeline, with GPUs reused across stages to minimize idle capacity and end-to-end latency.

Vidu S2-Editing: Real-time editing for video in progress

Vidu S2-Avatar addresses the question:

How can we generate a continuously responsive character in real time?

Vidu S2-Editing addresses a different one:

How can the video itself be edited in real time as it unfolds?

Traditional video editing typically follows this workflow: Record Video → Import Footage → Edit → Render → Export

Vidu S2-Editing transforms it into: Live Video Stream → Real-Time Editing → Live Output

Users can define the edit with text instructions or control the target appearance with a reference image. Video is moving from post-generation modification to being modified as it unfolds.

Frame-Aligned Attention: Changing visual content while preserving the original motion

The challenge in real-time editing is not simply changing the image. It is preserving the original video after the change. During outfit replacement, the subject's hands should remain where they were. After subject replacement, the character's motion and camera path must not drift. After background replacement, the spatial relationship between the foreground and the scene must remain stable. A model that simply regenerates a similar clip can easily introduce motion drift, temporal misalignment, or structural changes to the subject.

Vidu S2-Editing therefore introduces Frame-Aligned Attention. Its core principle is simple: each target frame attends only to the source frame at the same temporal position.

This preserves the source video's motion, timing, pose, camera movement, and scene structure as faithfully as possible. Meanwhile, the reference image remains available to every target frame, providing new style, character, clothing, or background information.

The result is:

Source Video: preserves motion / timing / structure

↓

Reference: provides style / identity / clothing / scene

↓

Real-Time Edited Video

Vidu S2-Editing therefore does not regenerate a merely similar video. It reconstructs the video in real time while preserving the original motion and timeline.

50+ styles: Enabling visual expression to change in real time

To enable real-time style editing, we train a video generation model conditioned on surface normals. The model first captures the source video's spatial structure and motion, then reconstructs a new visual appearance from a reference image. This allows it to change the style while preserving the source video's spatial structure and motion.

Vidu S2-Editing currently generalizes across 50+ realistic and non-realistic styles, including cel animation, cyberpunk, Monet-inspired painting, sketching, and Chinese gongbi painting. Style is no longer a fixed parameter set before generation. It becomes a visual variable that can change in real time while the video is running.

Real-time outfit, subject, and background replacement: From visual effects to real-time reconstruction

Beyond style transfer, Vidu S2-Editing supports full video-content editing.

Real-time outfit replacement

The model changes clothing while the subject remains in motion, preserving garment boundaries, textures, and occlusion relationships between the hands and the clothing.

In the virtual try-on examples described in the technical report, Vidu S2-Editing keeps the garment stable through large body and arm movements while transferring the outfit across the full body.

Real-time subject replacement

The model can transfer a reference character's complete visual identity—including facial identity, hairstyle, clothing, and body appearance—to the source video while preserving the source video's pose, motion, camera path, and spatial scene structure.

Real-time background replacement

As the subject moves, the model can replace the background with a new environment while preserving the foreground pose and motion, scene geometry, and spatial relationship between foreground and background.

Real-time editing is therefore more than a collection of local visual effects. It is the ability to reconstruct the video world in real time.

Real-time spatial video: From 2D video to spatial interaction

With Vidu S2, we also explore Real-Time Spatial Video Generation and Editing.

For Vidu S2-Avatar, monocular real-time video can be converted into synchronized left- and right-eye views through depth estimation and stereo conversion.

Vidu S2-Editing supports both: Monocular Video → Real-Time Editing → Spatial Video Conversion

and: Stereo Video → Joint Real-Time Editing → Left- and Right-Eye Output

Style transfer, virtual try-on, character replacement, and background replacement can all be applied to spatial video. This gives real-time video a technical foundation for VR, XR, and passthrough applications. In the future, real-time-generated video may no longer be confined to a 2D screen.

The entire visual space can be generated, edited, and continuously transformed through user interaction in real time.

The technology stack behind Vidu S2

Real-time interaction and editing do not depend on a single breakthrough. They require coordinated advances in model architecture, training algorithms, inference optimization, and system deployment.

At the attention layer, we use layer-wise hybrid attention, selecting among SageAttention, SpargeAttention, and Sparse-Linear Attention according to each layer's sensitivity to numerical precision. At the linear layer, we use per-block W8A8 GEMM to reduce the compute and memory cost of linear projections.

At the kernel layer, we use Triton/CUDA kernel fusion, CUDA Graphs, and launch optimization to reduce kernel launches, synchronization overhead, and intermediate-tensor traffic. For multi-GPU inference, Ulysses-style context parallelism and quantized inter-GPU tensor transfers reduce communication overhead.

For Vidu S2-Editing, the VAE encoder, backbone, refiner, and VAE decoder dynamically share GPUs along a common execution timeline. When one module does not need a GPU, another can reuse it immediately, reducing idle time and further lowering end-to-end inference latency.

Together, these model- and system-level optimizations provide the technical foundation for Vidu S2's real-time interaction and editing capabilities.

Leading performance across multiple benchmarks

On StreamAV-Bench, Vidu S2-Avatar achieves the best results on every metric reported in the technical report, including visual aesthetics, visual quality, audio quality, audio-visual semantic alignment, audio-visual synchronization, instruction following, subject consistency, and background consistency.

Vidu S2-Editing also achieves leading results on video-editing benchmarks.

In human-preference evaluations, Vidu S2-Avatar is preferred to Runway Character GWM-1, PixVerse Image Avatar, and HeyGen in overall quality, motion, expression, semantic adherence, and long-term stability.

Vidu S2-Editing is likewise preferred to XMax-X2.0 and Decart-Lucy2.5 in overall quality, video quality, temporal consistency, and semantic adherence.

In StreamAV-Bench comparisons, S2-Avatar achieves the best performance across all nine metrics, including visual quality, audio quality, audio-visual alignment, and subject and background consistency.

Vidu S2-Editing delivers leading results across key dimensions, including instruction compliance, reference fidelity, motion preservation, and temporal consistency.

From real-time generation to real-time interaction and editing

From Vidu S1 to Vidu S2, we continue to pursue the same goal:

To evolve video from one-time generated content into an interactive medium that can respond and change continuously.

S1 enabled real-time generation and continuous interaction. S2 further extends real-time capabilities to complex motion, dynamic references, real-time editing, and real-time spatial video. Video is no longer simply generated for viewing. It can understand and respond to the user as it unfolds, changing continuously through interaction.

From real-time generation, to real-time interaction, to real-time transformation.

This is how Vidu S2 expands the boundaries of real-time video—and how the Vidu S series continues to pursue a responsive, continuously evolving visual world.

Vidu S2 is now available. Try it for yourself!

🎮 Try Vidu S2: https://vidu.com/vidu-stream

🛠️ API Platform: http://platform.vidu.com/vidu-stream/doc

📄 Technical Report: https://arxiv.org/pdf/2609.11638

The Vidu S series is dedicated to delivering the ultimate visual interaction experience.