As any athlete will tell you: perfect practice makes perfect. But for individuals who do not have regular access to coaches or trainers, maintaining good form can be tricky. In fact, during the Covid-19 pandemic when many people were exercising at home, the U.S. Consumer Product Safety Commission reported a 48% rise in injuries related to at-home exercise . In hopes of preventing some of these injuries and extending the expert guidance of coaches, researchers from Drexel University and Michigan State University have developed a prototype of a program that uses artificial intelligence and computer vision to analyze video and provide form coaching in real time.
The program, which integrates biomechanical modeling with computer vision and a vision-language model, is designed to provide live, personalized feedback and explanations of the guidance it offers during an exercise — a feat that has proven to be challenging for most fitness coaching apps. The researchers published their work ahead of presenting their prototype, called BioCoach , at the Conference on Computer Vision and Pattern Recognition, hosted by the Institute of Electrical and Electronics Engineers and the Computer Vision Foundation in June.
"Many people who exercise at home with videos and apps don't get high-quality assessment of their movements," said Feng Liu, PhD , an assistant professor in Drexel's College of Engineering and Computing , who led the research. "Feedback is often too generic or simply encouragement but no actual form coaching. Our goal with BioCoach is to provide timely, specific cues grounded in body motion, closer to the kind of guidance a knowledgeable coach would give."
Feng's Visual Intelligence Lab at Drexel applies advanced computer vision, machine learning and 3D human-body modeling to study problems in exercise coaching, clinical gait assessment and classroom education.
To prepare BioCoach, the team started with an exercise-video coaching benchmark — the publicly available Qualcomm Exercise Video Dataset (QEVD), which includes hundreds of hours of exercise footage along with time-stamped coaching feedback.
The feedback included only short coaching comments, such as "lower your body more." So the researchers created a new version by re-annotating it with more detailed biomechanical targets, "increase elbow flexion to 90 degrees at the bottom," for example. They also added short rationale for the guidance, such as "increase hip/knee flexion to distribute load."
