Even linguistics experts are largely unable to spot the difference between writing created by artificial intelligence or humans, according to a new study co-authored by a University of South Florida assistant professor.
Research just published in the ScienceDirect journal Research Methods in Applied Linguistics revealed that experts from the world's top linguistic journals could differentiate between AI- and human-generated abstracts less than 39 percent of the time.
"We thought if anybody is going to be able to identify human-produced writing, it should be people in linguistics who've spent their careers studying patterns in language and other aspects of human communication," said Matthew Kessler, a scholar in the USF the Department of World Languages.
Working alongside J. Elliott Casal, assistant professor of applied linguistics at The University of Memphis, Kessler tasked 72 experts in linguistics with reviewing a variety of research abstracts to determine whether they were written by AI or humans.
Each expert was asked to examine four writing samples. None correctly identified all four, while 13 percent got them all wrong. Kessler concluded that, based on the findings, professors would be unable to distinguish between a student's own writing or writing generated by an AI-powered language model such as ChatGPT without the help of software that hasn't yet been developed.
Despite the experts' attempts to use rationales to judge the writing samples in the study, such as identifying certain linguistic and stylistic features, they were largely unsuccessful with an overall positive identification rate of 38.9 percent.
"What was more interesting was when we asked them why they decided something was written by AI or a human," Kessler said. "They shared very logical reasons, but again and again, they were not accurate or consistent."
Based on this, Kessler and Casal concluded ChatGPT can write short genres just as well as most humans, if not better in some cases, given that AI typically does not make grammatical errors.
The silver lining for human authors lies in longer forms of writing. "For longer texts, AI has been known to hallucinate and make up content, making it easier to identify that it was generated by AI," Kessler said.
Kessler hopes this study will lead to a bigger conversation to establish the necessary ethics and guidelines surrounding the use of AI in research and education.
