Learning Long-Term Motion Embeddings for Efficient Kinematics Generation
Learning Long-Term Motion Embeddings for Efficient Kinematics Generation
研究提出了一种通过长时运动嵌入来高效生成运动学的方法，直接对从追踪器模型获取的大规模轨迹中学习到的嵌入进行操作，将场景动态建模效率提升了数个数量级。该方法能够根据文本提示或空间戳指定的目标，高效生成长而真实的运动序列，避免了传统全视频合成在探索多种可能未来时效率低下的问题。
Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64×. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.
- † CompVis at LMU, Germany
- ‡ Munich Center for Machine Learning
Related readings and updates.
Naturalistic Head Motion Generation From Speech
Synthesizing natural head motion to accompany speech for an embodied conversational agent is necessary for providing a rich interactive experience. Most prior works assess the quality of generated head motion by comparing them against a single ground-truth using an objective metric. Yet there are many plausible head motion sequences to accompany a speech utterance. In this work, we study the variation in the perceptual quality of head motions…
On the Role of Lip Articulation in Visual Speech Perception
Generating realistic lip motion from audio to simulate speech production is critical for driving natural character animation. Previous research has shown that traditional metrics used to optimize and assess models for generating lip motion from speech are not a good indicator of subjective opinion of animation quality. Devising metrics that align with subjective opinion first requires understanding what impacts human…
来源：Apple Machine Learning Research · machinelearning.apple.com