Loneliness has a critical impact on the mental health of citizens, particularly among the elderly. Robots capable of perceiving and responding to human emotions can serve as heart-warming companions to help lift the spirits. A research team at The Hong Kong Polytechnic University (PolyU) has discovered that the combined power of music and empathetic speech in robots with artificial intelligence (AI) could foster a stronger bond between humans and machines. These findings underscore the importance of a multimodal approach in designing empathetic robots, offering significant implications for their application in health support, elder care, education and beyond.
The research project, A Talking Musical Robot over Multiple Interactions, was led by Prof. Johan HOORN, Interfaculty Full Professor of Social Robotics of the School of Design and the Department of Computing at PolyU, in collaboration with Dr Ivy HUANG at The Chinese University of Hong Kong. The study investigated how music and empathetic speech could enhance the emotional resonance of on-screen robots, revealing that music can act as a powerful adjunct to empathetic speech.
As part of the study, the team examined how Cantonese-speaking participants interacted with empathetic robots across three interactive sessions. The findings showed that combining music and speech significantly increased the participants' perceived empathy of the machines.
"Our data indicate that the presence of music continued to enhance the robot's resemblance to humans in later sessions," explained Prof. Hoorn. "One interpretation is that music made the interaction feel more like a real conversation with a personality, something human counsellors might do by playing music to comfort their clients, which in turn made the robot seem more lifelike or socially present."
However, the research pointed out that the impact of music could diminish over time when the participants became attuned to the music after repeated sessions, highlighting the importance of tailoring interaction strategies to individual users' needs to sustain effective human-robot interaction. The study suggested that empathetic robots should be designed to adapt their responses to user feedback and context, for example, by adjusting various musical elements or gradually personalising dialogue content to maintain sustained relevance of empathy.
Prof. Hoorn emphasised: "Our research points to the significance of multimodal communication encompassing music, speech and more through empathetic robots. It holds considerable promise for application in real-world settings, particularly in the fields of mental health support and elderly care. The integration of empathetic robots capable of delivering tailored musical experiences and engaging in sensitive conversation could provide meaningful companionship and emotional support to individuals who may experience loneliness or social isolation."
Prof. Hoorn is leading another project, "Social Robots with Embedded Large Language Models Releasing Stress among the Hong Kong Population", which has received funding of over HK$40 million from the Research Grants Council Theme-based Research Scheme.
Concurrently serving as Associate Director of the PolyU Research Institute for Quantum Technology, Prof. Hoorn is set to explore quantum-inspired models of human affect to better capture and respond to the inherent vagueness and ambiguity of emotional experience. Unlike traditional computational systems that struggle with the fluid and context-dependent nature of affective responses, quantum models can represent emotional states as probabilistic superpositions, reflecting the genuine uncertainty and complexity of human feelings.
"What excites me the most is the possibility of developing social robots that not only recognise the complexity of human affect but also embrace it. These robots could offer support that is adaptable, open-ended and compassionate, similar to the individuals they are designed to help," added Prof. Hoorn.
The study has been published in ACM Transactions on Human-Robot Interaction, a leading peer-reviewed interdisciplinary journal in the field.
