How do we recognize what one person is saying when others are speaking at the same time? This simple, yet peculiar question forms the basis of the cocktail party problem, a phrase coined by the cognitive scientist Colin Cherry in 1953. It describes how the brain acts to focus on one sound or voice in a packed, noisy room. However, for anyone who wears a hearing aid, the cocktail party problem is a real problem. With a lot of help from machine learning, PhD researcher Luan Fiorio set out to solve the problem so that those wearing a hearing aid in the future can enjoy a cocktail party for what it is meant to be, and that's just a party.
"The cocktail party problem describes how the human auditory system focuses on one sound over many others - which is a problem that one might encounter at any busy cocktail party," says Luan Fiorio.
Those with normal hearing can isolate specific voices from a room full of voices, but people with hearing loss face difficulties in isolating one voice from many other voices, noises, and reverberations. "Hearing aids and other hearing devices can help, and recent developments in machine learning models have made them even better, but this advancement requires lots of computational power."
Guitar-amplified curiosity
Some of Fiorio's close relatives deal with hearing loss, and one even uses a hearing aid. However, his initial motivation to work on improving hearing aids for his PhD research stems not from hearing aids.
"I grew up playing guitar, and I was always curious about how guitar amplifiers worked and the processing algorithms behind them. This led me to the field of audio processing and then later I had contact with research on the problem of audio processing in hearing aids. The fact that my PhD research could have a positive impact on hearing aid technologies, and by association on the lives of many people, was also a huge motivating factor during my PhD journey."
Label issues
One key issue that Fiorio explored was the training of the software in hearing aids. "Hearing aids should be able to identify different sounds in an environment, which can be used to suppress the issue associated with the cocktail party problem. This means that the devices must be able to recognize different sounds such as voices, as well as characteristics from the acoustic environment," says Fiorio.
Normally, hearing aid software (in this context neural network models) is trained via supervised learning, where every audio signal has a label describing it. However, this can pose some problems.
"Labels carry biases. For instance, one person may label a certain sound as coming from a metro station while someone else labels it as being from an airport," says Fiorio. "To avoid this label bias, I didn't use labels! Instead, I used unsupervised learning where it is possible to make neural networks learn certain tasks without having the 'correct answer' at hand. This is done through some interesting mathematics and probability theory, and it is one of my favorite topics within my thesis."
Machine learning is here to stay
Machine learning played a central role in Fiorio's research, and he's quick to acknowledge its importance. "It is difficult to think of modern audio processing without machine learning. Every major hearing aid company has at least one device in their catalogue which employs a deep learning-based approach, and almost all modern devices use machine learning."
Machine learning approaches prove to be handy when used in situations where noise is unpredictable and acoustic conditions can change suddenly - which are both typical of the cocktail party problem.
Testing without people
The principal focus of Fiorio's work was on training the algorithms that could be found in hearing devices. The training was validated using a combination of objective metrics and by listening to the audio clips thar were outputted by the software where possible. Unfortunately, he was unable to test the software in hearing aids worn by individuals.
"Without being part of a hearing aid company, it is difficult to test the proposed algorithms in devices. While this is not necessarily good for researchers based in academia, it presents a gap for testing devices focused on hearing-based research. I am only aware of one start-up that produces hearing devices on a small scale. Hearing aid companies often have their own testing equipment, but their use is mostly in-house."
Future hearing aids
As computer chips become more efficient, and given the power of machine or deep learning, hearing aids look destined to lean on more machine learning approaches in the future.
"These algorithms rely on efficient hardware implementation to compensate for their computational cost, which is a bottleneck too," Fiorio notes. "A more efficient implementation would mean lower battery consumption."
Fiorio believes that his work on training hearing aids to recognize sounds without the need to label sounds will be pivotal. "On-the-fly learning for hearing aids during operation is the ultimate goal. In this way, any user has access to a specialized device that adapts to their needs based on their personal preferences and the environment around them. The first step toward this achievement is to forget labels and focus at the audio content itself."
Fiorio will continue his work on revolutionizing the future of hearing aids with a new position as a research scientist at GN Hearing, which is part of the Danish company GN Store Nord.
