MegActor: Harness the Power of Raw Video for Vivid Portrait Animation

Shurong Yang*, Huadong Li*, Juhao Wu*, Minhao Jing*†, Linze Li, Renhe Ji, Jiajun Liang, Haoqiang Fan
Megvii Research
*Equal contribution Lead this project Corresponding author

MegActor features:

Usability: animates a portrait with video while ensuring consistent motion.

Reproducibility: fully open-source and trained on public available datasets.

Efficiency: ⚡200 V100 hours of training to achieve pleasant motions on portraits.

Introduction

Despite raw driving videos containing richer information on facial expressions than intermediate representations such as landmarks in the field of portrait animation, they are seldom the subject of research. This is due to two challenges inherent in portrait animation driven with raw videos: 1) significant identity leakage; 2) irrelevant background and facial details such as wrinkles degrade performance.

To harness the power of raw videos for vivid portrait animation, we proposed a pioneering conditional diffusion model named MegActor. First, we introduced a synthetic data generation framework for creating videos with consistent motion and expressions but inconsistent IDs to mitigate the issue of ID leakage. Second, we segmented the foreground and background of the reference image and employed CLIP to encode the background details. This encoded information is then integrated into the network via a text embedding module, thereby ensuring the stability of the background.

Finally, we further style transfer the appearance of the reference image to the driving video to eliminate the influence of facial details in the driving videos. Our final model was trained solely on public datasets, achieving results comparable to commercial models. We hope this will help the open-source community.

Emoji-driven examples

With MegActor, people can migrate the motion of their favorite emojis to their own portraits..

VASA-1 Cases

EMO Cases

Self driven of EMO results

Cross driven of EMO results

BibTeX

@misc{yang2024megactor,
      title={MegActor: Harness the Power of Raw Video for Vivid Portrait Animation}, 
      author={Shurong Yang and Huadong Li and Juhao Wu and Minhao Jing and Linze Li and Renhe Ji and Jiajun Liang and Haoqiang Fan},
      year={2024},
      eprint={2405.20851},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}