When it comes to creative writing, score one for the humans over the machines. For now, anyway.
New research finds that people evaluate creative writing less favorably when they learn it was generated in whole or part by artificial intelligence. And the anti-AI bias is persistent and difficult to reduce, even when steps were taken to lessen the aversion within the experiments.
That strength and consistency of the negative attitude toward AI-generated or assisted writing jumped out at researchers, and they say it poses implications for integrating AI in creative fields. As it stands, the study finds people tend to view the creative works of machines as "relatively inauthentic and therefore less worthy of their appreciation."
The researchers say previous research has offered preliminary evidence that AI disclosure can have negative effects on how people evaluate creative content, but their study builds on it by revealing a "surprising level of robustness" across 16 experiments involving 27,000 participants conducted between March 2023 and June 2024.
"What surprised us most was how incredibly 'sticky' this penalty is," said Justin Berg, the study's co-author and an associate professor of management and organizations at University of Michigan's Ross School of Business.
"We threw everything at it, from changing the story's perspective to humanizing the AI or framing it as a collaboration, and nothing reliably reduced the bias. Across all the experiments, the pattern was clear: If readers believe AI is involved, they view the work as less authentic and enjoy it less, even when the content is identical."
Throughout the study, the researchers asked participants to read and evaluate AI-generated writing samples created using ChatGPT-chosen because it was the most well-known large language model at the time of the initial study. Across all the experiments, AI disclosure decreased evaluations by an average of 6.2%.
Berg and his colleagues, Manav Raj of the University of Pennsylvania's Wharton School of Business and Rob Seamans of New York University's Stern School of Business, note the results reflect attitudes during a period of rapid advancements in AI capabilities and shifting perceptions of its role in creative work. It's an open question-and fertile ground for further study-whether the AI disclosure penalty will persist, diminish or reverse as such content becomes more pervasive.
What does appear clear-at least for now-is the use of AI in creative writing triggers different psychological responses than when the technology is employed in other domains. Understanding that bias is crucial for helping navigate the challenges for those working toward fuller, broader human-AI collaboration.
The findings, published in the Journal of Experimental Psychology, also pose practical implications for creative producers using AI, as the U.S. Congress considers AI disclosure legislation. Mandated disclosure of AI involvement in creative work could usher in negative biases toward such content and potentially affect its reception.
