AI Decodes Pianists' Muscle Activity Via Video

AI and human-movement research intersect in a study that enables precise estimation of hand muscle activity from standard video recordings. Using a deep-learning framework trained on a large, comprehensive multimodal dataset from professional pianists, the researchers introduce a system that accurately reconstructs muscle activation patterns without sensors. This advancement provides a low-cost, non-invasive method for analyzing fine motor control, optimizing rehabilitation strategies, enhancing performance training, and informing future developments in human-machine interaction.

From Pose to Muscle: Inferring Pianists' Hand Muscles from Video

Hand movements during piano performance depend on precise coordination between small muscles hidden beneath the skin. Tracking these signals has traditionally required electromyography (EMG) sensors, which are expensive, intrusive, and technically complex. A research team led by Professor Hideki Koike from the Department of Computer Science, School of Computing, Institute of Science Tokyo (Science Tokyo), Japan, and Dr. Shinichi Furuya from Sony Computer Science Laboratories, Japan, has now addressed this challenge using artificial intelligence.

Their new framework, Piano Keystroke-Pose-Muscle Network (PianoKPM Net), estimates miniature hand muscle activity using only video recordings. Their findings were published online on September 19, 2025, and will be presented at the 39^th Conference on Neural Information Processing Systems (NeurIPS 2025), held in San Diego, USA, on December 2, 2025. The system is built on a new dataset, PianoKPM, that captures how expert pianists move, press, and control their hands with exceptional precision. It includes 12.6 hours of synchronized data from 20 professional pianists performing seven distinct musical tasks. Each performance was recorded with multi-view videos at 60 frames per second, 3D hand poses, 1 kHz keystroke data, audio, and 2 kHz EMG signals from six small hand muscles. The dataset contains more than five million pose frames and 28 million EMG samples, creating the first detailed map linking visible motion with internal muscle activity. "Leveraging this dataset, we propose PianoKPM Net to infer high-frequency EMG from pose data," exclaims Koike.

Using this foundation, PianoKPM Net learns to infer muscle behavior from video data. By combining cues from hand poses and keystrokes, the model reconstructs muscle signal timing and strength. In comparative tests against advanced baselines, such as NeuroPose and CodeTalker, PianoKPM Net achieved higher accuracy in predicting both the amplitude and timing of muscle activation. Even with pianists and musical pieces not included in training, the model maintains strong performance, confirming its adaptability and generalization.

This approach transforms a simple camera into a non-invasive tool for studying muscle coordination. This allows observation of how skilled pianists control subtle muscle movements to achieve speed, control, and precision. This makes a detailed physiological assessment possible without attaching any sensors to the body, reducing both cost and discomfort.

The technology has potential far beyond the piano. In sports science, it could track muscle exertion to enhance training accuracy and prevent overuse injuries. In rehabilitation, it could monitor recovery progress, providing clinicians with continuous feedback without physical attachments. It could also improve human–machine interaction systems, where understanding a user's muscle effort helps refine robotic assistance and gesture-based interfaces. "Together, the PianoKPM Net and PianoKPM dataset create a foundation for affordable access to internal physiological and muscle activity signals, supporting progress in human augmentation and advanced human–machine interaction," explains Koike.

The research team plans to make both the dataset and model publicly available. This open release will allow scientists and developers to advance studies in motor learning, embodied intelligence, and assistive robotics. Widespread access will help standardize benchmarks for motion and muscle activity estimation, accelerating development in multiple fields. By linking vision and physiology, PianoKPM Net offers a new method to study fine motor control. It replaces complex EMG setups with accessible video-based analysis, creating opportunities for performance research, clinical evaluation, and human–technology design. The system marks a clear step toward affordable, AI-driven analysis of skilled movement, where invisible muscle patterns can finally be observed and understood through vision alone. In the future, this technology could contribute to remote skill education by enabling use over low-latency communication networks, even in environments without access to expensive biological measurement equipment.

Authors:

Ruofan Liu^1,2, Yichen Peng¹, Takanori Oku², Chen-Chieh Liao¹, Erwin Wu¹, Shinichi Furuya², and Hideki Koike¹

Title:

From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography

Journal:

Proceedings of 39^th Conference on Neural Information Processing Systems (NeurIPS 2025)

Affiliations:

¹Department of Computer Science, School of Computing, Institute of Science Tokyo, Japan

²Sony Computer Science Laboratories, Japan