A technical glitch during a Zoom call shouldn't be that big a deal, or lead someone to make a judgment about the person on the other end.
But in actuality, glitches during face-to-face video calls - even when the glitch does not affect the transmission of information - can shatter the illusion of being across the table from the other person, evoking a creepy or eerie feeling known as "uncanniness."
This can have major implications in critical life events, new Cornell-led research has found. Minor audiovisual glitches during video calls negatively affect judgments in hiring decisions, and reduce trust in a telehealth provider.
And in a hugely consequential realm - parole hearings for incarcerated individuals - glitches were associated with a 12-percentage-point lower likelihood of someone being granted criminal parole.
"The best feature of video calling is the fact that you basically feel like you're together," said Jacqueline Rifkin, assistant professor of marketing and management communications at the Samuel Curtis Johnson Graduate School of Management, in the Cornell SC Johnson College of Business.
"And so when there's a glitch, you're right in that danger zone where it's almost perfect, but not quite - what has become known as the 'uncanny valley.' It triggers this switch in your brain where things feel just a little bit creepy," she said.
Rifkin is co-first author of "Video Call Glitches Trigger Uncanniness and Harm Consequential Life Outcomes," which published Dec. 3 in Nature. The other first author is Melanie Brucks, assistant professor of marketing at Columbia Business School; also contributing was Jeff Johnson, professor of marketing in the Bloch School of Management at the University of Missouri, Kansas City (UMKC).
The group's study combined analysis of previously held video conferences and experiments they themselves conducted, and all reached the same conclusion: Face-to-face video calls that contained minor glitches produced negative outcomes.
In the first study, Rifkin and the team analyzed the CANDOR (Conversation: A Naturalistic Dataset of Online Recordings) Corpus, an archival database of more than 1,600 "get-to-know-you" video calls conducted from January to November of 2020 to advance the study of human conversation. After the conversations, participants took a survey, which included questions regarding the level of connection they'd reached with their partner, as well as whether they'd experienced technical difficulties during the conversation.
Rifkin and her team then watched the videos where glitches were reported, to determine the type of glitch (audio or video) and whether they were experienced by one or both partners. Analysis revealed that social connection was measurably poorer between participants who'd experienced glitches during their call, regardless of the type of glitch and whether one or both partners had experienced them.
All other experiments confirmed that the presence of glitches interfered with the illusion of in-person conversation that face-to-face video conferencing creates. The feeling of uncanniness, Rifkin said, is hard to shake once it's present.
"We tried a lot of different interventions, but we basically struggled to overcome it."
The study regarding parole hearings was particularly notable. Using transcript data from the state of Kentucky's virtual parole hearings, from January through April of 2021, the researchers developed a "glitch dictionary" that helped them determine which virtual hearings contained audio or video disruption.
Glitches were detected in 32.6% of parole hearings, and played an outsize role in outcomes: Offenders in glitchy calls were granted parole 48% of the time, compared to 60% of those in non-glitchy calls. Controlling for characteristics of the offender or the crime made no difference.
"That was when we started feeling like, wow, there's really something quite important to say here," Rifkin said.
A technology that's supposed to make it easier to bring people together, Rifkin said, also has this potentially negative side effect.
"It's supposed to be the great equalizer, it's going to get everybody on the same page - and in a lot of ways, that's true," she said. "But then there's this other side: What about the people who don't have great internet service, and what sorts of opportunities are they missing out on? There are these implicit biases that they don't even realize are going on."
This work began more than five years ago, just as the COVID-19 pandemic was taking hold, so Rifkin and her collaborators were not only studying the effects of video-call glitches, but experiencing them, as well.
"We actually met in person for the first time a year ago," said Rikin, an assistant professor at UMKC from 2020-22. "During four of the five years, we only met remotely. We definitely had our fair share of glitchy research meetings, and learned a lot from the experience."
