People communicate with more than words. A raised hand, a change in posture or a facial expression can help convey meaning and emotion. Now, researchers have developed an artificial intelligence (AI) system that can translate speech into coordinated movements of the body, hands and face, potentially making digital characters and virtual presenters more expressive and lifelike.
Developed by researchers at Tohoku University and the University of Hong Kong, DynaConTalk generates three-dimensional (3D) human movements that correspond to spoken content. Unlike approaches that generate only individual gestures, the system coordinates different parts of the body while maintaining smooth movement over longer speeches. Users can also guide the generated motion by specifying poses or providing reference examples.
Generating realistic full-body movement from speech presents several challenges. Large changes in body posture can overshadow smaller movements, such as finger gestures and facial expressions, making them difficult for AI models to learn. This can result in repetitive gestures or unnatural shaking. Another problem occurs when separately generated movement segments are joined together, sometimes creating noticeable changes in speed or rhythm.
To address these issues, DynaConTalk separates movement into slower and faster components while keeping them synchronized. This is similar to separating sound into low- and high-frequency components. By learning these different types of movement together, the system can capture both broad changes in posture and subtle, brief gestures. It also dynamically adjusts how much it relies on different features of speech, including rhythm, voice characteristics and spoken words, as it generates movement.
The system also uses information from previously generated movements to guide the next segment. This allows it to maintain continuity across longer sequences, reducing abrupt transitions and helping produce more natural movement during extended speeches.
The researchers evaluated DynaConTalk using two publicly available datasets of speech and human movement, BEAT2 and SHOW. They also conducted a study in which 30 participants compared videos generated by three different systems across 15 speech examples. DynaConTalk received the highest average ratings in all four evaluation categories: naturalness, correspondence with spoken content, synchronization and movement diversity. Automated evaluations further showed that its generated movements more closely resembled the distribution of recorded human movements.
The technology could have applications in character animation for video games and films, virtual presenters, online education and remote communication. For example, creators could specify a character's initial pose or provide a reference movement, then let the AI generate the surrounding gestures and expressions. This offers a more flexible way to create animated characters whose movements complement what they are saying.
The team hopes to extend the approach to a wider range of whole-body movements, including walking. However, existing datasets contain relatively limited examples of large-scale movement, while speech alone does not provide enough information to determine how a person should move through space. The researchers plan to explore larger motion datasets and additional guidance about movement paths to address these limitations.
The findings were published in ACM Transactions on Graphics on November 9, 2026. The research is also scheduled to be presented at SIGGRAPH Asia 2026, which will take place in Kuala Lumpur, Malaysia, from December 1 to 4.
- Publication Details:
Title: DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Authors: Yifei Zhu, Yangyang Cai, Mingyi Shi, Miao Cheng, Lin Gu, Taku Komura, Yoshifumi Kitamura
Journal: ACM Transactions on Graphics, Vol. 45, No. 6, Article 194 (scheduled for December 2026)
DOI: 10.1145/3842553
