Controllable Generation of Synthetic Video Data With Human Motion
summary
A PyTorch-compatible toolkit generates controllable human-action videos by combining pose transfer and 3D-grounded animation, releases 100 controllable avatars, and produces roughly 3,600 synthetic videos per dataset for augmentation. A participant study compares Animate Any, Magic Animate and X-Avatar and finds X-Avatar yields the best semantic and motion consistency. Experiments train on 225 real plus 225 synthetic videos and evaluate on 50 reals, showing up to 2–3x boosts on Toyota Smarthome and NTU RGBD with single-person, no-object limitations.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:22Motivation for synthetic videos
- 2:56Pose transfer for synthetic data
- 4:01Evaluating pose transfer methods
- 6:52Comparing pose transfer models
- 7:55Quality goals for synthetic data
- 8:37Identity diversity with Random People
- 9:20Avatar creation pipeline
- 10:45Reenacting identities on backgrounds
- 11:29Experimental setup and generation
- 12:56Results on action recognition
- 14:22Limitations and ethics
- 15:29Contributions and release
- 16:14Q&A
related talks
03mBuilding artificial people: Endless possibilities and the dark side
41mSocial Behavior Prediction from Video Observations
22mMovie Gen Video: State-of-the-art Video Generation
35mDeep learning for 3d human motion capture
17mThe State of Generative Media
58mUsing Unity to Generate Synthetic data and Accelerate Computer Vision Training
21mVisualizing robot minds, and other fun stuff
41mComputer Vision for Remote AR