Virtual reality and prosthetics are advancing rapidly thanks to technological innovations, but both are still missing one key element — a sense of touch, also known as haptic feedback.
"A major challenge for both prosthetic limb users and people who use virtual reality is a lack of haptic feedback," says Craig Chapman, associate professor in the Faculty of Kinesiology, Sport, and Recreation and lead author on a study recently published in Nature Scientific Reports.
The human brain needs haptic feedback to feel confident in its own movements and to experience the feeling of embodiment. Without it, technologies like VR will struggle to truly feel real, and prosthetic limb users won't quite feel like their prosthetic is a natural extension of their body. But accurately replicating the experience of touching an object is not something we have the technology for, explains Chapman.
"If you think about your hand as a sensor, how many signals it can pick up, there would be thousands if not tens of thousands of sensors in your hand. There's no device that's capable of stimulating all of those sensors."
Take the visual realm as a comparison. As Chapman explains, there are displays that can show incredibly dense images with millions of pixels: "We can produce things for your eye that are almost as good as the real world at producing stimulation. For the hand, we are light-years away."
To explore the effect of haptic feedback with virtual reality and prosthetic limb users in mind, Chapman designed an experiment where he was able to both take away and return haptic feedback to study participants.
For the experiment, participants wore a virtual reality headset and a hand-tracking glove while reaching for a virtual box. In one scenario, there was no physical box present — it existed only in the virtual world. In the other scenario, the researchers provided a real, physical box that participants could lift, feel the weight in their hands and sense the texture and temperature with their fingertips. The researchers painstakingly matched the virtual reality conditions with the real-world conditions to ensure the sole difference between the scenarios was the presence of haptic feedback.
They found that participants' behaviour changed dramatically between the two scenarios. When participants had haptic feedback, they looked at an object for about half a second before they touched it, then immediately moved their eyes to the next target, which Chapman says aligns with normal behaviour we would expect in such a situation. When there was no haptic feedback, participants' gaze remained fixated on the object for much longer after it was interacted with.
"The really critical finding is that in moments of interaction, a lack of haptic feedback forces vision to stay locked at the location of interaction. It's not free to move ahead," says Chapman. "And what we see in the real world is that the eyes always want to look ahead, to go to the next thing."
The different behaviour in the two scenarios comes down to something called embodiment, a complex concept Chapman says contains three main dimensions: ownership, agency and location. For someone to feel a sense of embodiment, there needs to be a perception that an object is part of their body (ownership), that they are in control of that object (agency) and that their body is where they perceive it to be (location).
By observing eye-hand co-ordination and the way haptic feedback affects it, Chapman and his collaborators have identified an objective way to measure the degree of embodiment someone feels in a particular situation.
"That's a really big step, because if you can measure embodiment, then people designing prosthetic limbs, people designing virtual reality experiences, they're going to now have a new way to determine whether or not a person is embodying that experience."
"If we can improve embodiment, we can likely get these tools to feel even better."
The research was a collaboration with Ewen Lavoie, a resident physician who was formerly a PhD student in Chapman's lab, and Jacqueline Hebert, a professor in the Division of Physical Medicine and Rehabilitation who leads the BLINC (Bionic Limbs for Improved Natural Control) Lab.
