A University of Washington team created Audeo, a system that can generate music using only visual cues of someone playing the piano.Patrick Tomasso/Unsplash
Anyone who's been to a concert knows that something magical happens between the performers and their instruments. It transforms music from being just "notes on a page" to a satisfying experience.
For journalists
A University of Washington team wondered if artificial intelligence could recreate that delight using only visual cues - a silent, top-down video of someone playing the piano. The researchers used machine learning to create a system, called Audeo, that creates audio from silent piano performances. When the group tested the music Audeo created with music-recognition apps, such as SoundHound, the apps correctly identified the piece Audeo played about 86% of the time. For comparison, these apps identified the piece in the audio tracks from the source videos 93% of the time.
The researchers presented Audeo Dec. 8 at the NeurIPS 2020 conference.
"To create music that sounds like it could be played in a musical performance was previously believed to be impossible," said senior author Eli Shlizerman, an assistant professor in both the applied mathematics and the electrical and computer engineering departments. "An algorithm needs to figure out the cues, or 'features,' in the video frames that are related to generating music, and it needs to 'imagine' the sound that's happening in between the video frames. It requires a system that is both precise and imaginative. The fact that we achieved music that sounded pretty good was a surprise."
