DiTaiListener: Controllable High Fidelity
Listener Video Generation
with Diffusion

Maksim Siniukov*   Di Chang*   Minh Tran   Hongkun Gong   Ashutosh Chaubey   Mohammad Soleymani University of Southern California
*Equal Contribution    
Paper arXiv Code Button Image Model

Baseline Comparison on ViCo

Abstract

Generating naturalistic and nuanced listener motions for extended interactions remains an open problem. Existing methods often rely on low-dimensional motion codes for facial behavior generation followed by photorealistic rendering, limiting both visual fidelity and expressive richness. To address these challenges, we introduce DiTaiListener, powered by a video diffusion model with multimodal conditions. Our approach first generates short segments of listener responses conditioned on the speaker's speech and facial motions with DiTaiListener-Gen. It then refines the transitional frames via DiTaiListener-Edit for a seamless transition. Specifically, DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener head portrait generation by introducing a Causal Temporal Multimodal Adapter (CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter integrates speakers' input in a causal manner into the video generation process to ensure temporally coherent listener responses. For long-form video generation, we introduce DiTaiListener-Edit, a transition refinement video-to-video diffusion model. The model fuses video segments into smooth and continuous videos, ensuring temporal consistency in facial expressions and image quality when merging short video segments produced by DiTaiListener-Gen. Quantitatively, DiTaiListener achieves the state-of-the-art performance on benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion representation (+6.1% in FD metric on VICO) spaces. User studies confirm the superior performance of DiTaiListener, with the model being the clear preference in terms of feedback, diversity, and smoothness, outperforming competitors by a significant margin.

Long Video Generation Strategy Comparison with Prompt Travel and Teacher Forcing

Text-Conditioned Listener Generation

Listener Generation on RealTalk

Out-of-Domain Listener Generation

BibTeX

If you find DiTaiListener useful for your research or applications, please cite DiTaiListener using this BibTeX:

@misc{siniukov2025DiTaiListener,
      title     = {DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion},
      author    = {Maksim Siniukov and Di Chang and Minh Tran and Hongkun Gong and Ashutosh Chaubey and Mohammad Soleymani},
      year      = {2025},
      archivePrefix = {arXiv},
    }

Acknowledgement

This research work was sponsored by the Army Research Office and was accomplished under Cooperative Agreement Number W911NF-25-2-0040. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.