People gave the highest ratings to stories generated by AI that they were told had been written by a human, and were unable to tell the difference between human- and AI-created stories, according to new research published by Cambridge University Press.
The study, led by researchers at Villanova University, USA, and published in Judgement and Decision Making, adds to the growing body of evidence that many people's ability to spot AI-generated content is limited.
For the study, adult participants aged 18-81 were tested to see whether they could tell the difference between human-written and AI-generated short stories, how they rated the quality and appeal of each, and whether they could spot AI-generated content.
The researchers used three fictional short stories written by human authors published in a reputable literary journal or short story collection, alongside three corresponding short stories generated by ChatGPT1.
In the first experiment, 1,682 participants were asked to read a story and were either correctly or incorrectly told that it was written by a human or AI. They then rated the story's quality and how engaging they found it2.
In the second and third experiments, participants (424 and 481 respectively) received one human-written and one AI-generated story, but were not told the origin of the stories. After reading them, they were asked to identify the author of each story.
Participants were also asked to rate their experience with using AI platforms like Chat GPT, and their experience with fictional literature.
The results showed that:
- AI-generated stories received higher ratings of quality (how well-written the stories were) and absorption (how invested in the story the reader was), than human-written stories.
- AI-generated stories were rated even more highly when participants believed they had been written by a human.
- Participants' ability to identify AI-generated content was just 39.93% (less than chance) in experiment two and 51.97% in experiment three (no different from chance).
Dr Deena Weisberg, senior author of the study from the Department of Psychological & Brain Sciences at Villanova University, said: "In our study, participants gave the highest ratings to stories that they had been told were written by humans but which were actually written by AI.
"I'd argue that this reveals a bias towards narratives written by real people. We assume creative writing requires uniquely human qualities, such as emotional understanding and lived experience, which leads people to underestimate AI's capabilities. I'd say that public assumptions about AI's capabilities are increasingly out of date."
The researchers also found that participants who said they had greater AI literacy were better able to identify AI-written text, but participants who said they had greater expertise in literature were not.
In the second experiment, the findings showed that each one-point increase in participants' self-reported AI expertise score was associated with a 14% increase in the odds of correctly guessing the story's author.
In the third experiment, which used a previously validated measure called the Artificial Intelligence Literacy Scale (AILS), a one-point increase in participants' scores was associated with a 33% increase in the odds of correctly guessing the story's author.
Dr Weisberg said: "One thing to note was that familiarity with AI systems appeared to help people recognise the patterns typical of AI-generated writing, such as em dashes and sentence structures such as 'it's not just X, it's Y'. That suggests that improving AI literacy would be one way to help people to navigate the new AI-enabled world that we're living in."
The study validates previous research that has shown that people often prefer AI-generated writing in various formats, including short stories, poetry and essays.
Speculating on why this might be, Dr Weisberg added: "AI writing tends to be clearer, more direct and easier to process. By contrast, human-written stories, are often more subtle and complex. For example, the AI versions of our stories usually stated their themes explicitly, rather than allowing readers to infer meaning from the characters' words and actions.
"We hear a lot about the crisis in literacy, shrinking attention spans and the influence of platforms such as TikTok. But I don't think we can say that our findings in this study can be entirely explained by contemporary media consumption. It's more that these technologies amplify existing tendencies, rather than creating them.
"Perhaps people generally prefer predictability, because difficult or subtle material requires more brain power."
