Why do we breeze through some sentences in a book or article but have to reread others to comprehend their meaning?
A team of linguists and data scientists has found a partial answer in AI—some of this processing parallels that of neural-network-based large language models (LLMs). However, other aspects of why we read this way cannot be explained by these technologies, revealing where human and AI language processing diverge and maintaining the mystery of some stages of the reading process.
The new study, by researchers from New York University and the University of Massachusetts Amherst, shows that humans and AI process language in similar ways during the earliest moments of reading: both rely on next-word predictions. However, as reading continues and passages become more complex—often requiring rereading—humans' processing differs from that of AI, which is entirely built on next-word prediction and therefore cannot account for how we navigate these types of passages.
"Language models develop their remarkable language understanding capabilities by being trained to predict the next word in a sentence, which led us to ask whether the same predictive processes that drive these AI systems could also explain how humans comprehend sentences," explains William Timkey, a linguistics doctoral student at NYU and the lead author of the paper, which appears in the journal Proceedings of the National Academy of Sciences (PNAS). "We found that LLMs can explain how long it takes people to recognize words when their eyes move smoothly forward through a text, but they fail to capture the cases where people have difficulty integrating a word into the larger context of a sentence, which is often accompanied by rereading."
The authors note that despite the remaining uncertainty on how humans read—notably the rereading of passages—the findings nonetheless offer a potential roadmap for both improving language learning and addressing reading-related afflictions.
"We now know a little bit better how humans and models are different," says Brian Dillon, a professor of linguistics at the UMass Amherst and the paper's senior author. "That is the first step in understanding how we can close that gap, which we want to do because that could have enormous advantages down the road."
"Our work shows that AI can be very valuable for cognitive science, but it is not enough," adds Tal Linzen, an associate professor of linguistics and data science at NYU and one of the paper's authors. "The human mind does not always work like standard AI systems—for instance, 20 percent of our eye movements when reading are backward, and AI models cannot explain when we decide to do that. We now have our work cut out for us to create computational models that more closely match the human mind and that can help us understand in detail how it operates."
When we see words on a page, we go through mental processes of taking the visual information of the letters, accessing the meaning of the word, and then integrating that with the rest of a sentence. While much of reading is driven by word prediction, less clear are its limits—a question the researchers explored in the PNAS study.
To do so, they deployed LLMs because their predictive-text feature aligns with some theories of how the brain works: the prediction process drives our ability to comprehend sentences.
"LLMs seem to capture some of the properties of language as we understand it—they can generate text fluently and they appear to react in a way that suggests they have some understanding of what's going on," explains Dillon, who directs the Computational Sentence Processing Lab in the Department of Linguistics at UMass Amherst.
"We build a mental representation of what we think a sentence means based on the words on the page, then use that representation to make predictions about the next words, and then update our mental representation when those predictions are wrong," adds Timkey.
In the study, the researchers used eye-tracking technology to analyze 368 adult readers, focusing on how long participants spent reading—and rereading—each word of carefully designed sentences. These included a diverse set of syntactically challenging sentences, known as garden- path sentences—grammatically correct sentences that start in such a way that a reader's initial interpretation will likely be incorrect. For instance, take the sentence "The old man the boat." Readers may initially think the sentence is about an old man, but instead, the sentence means that old people are manning a boat. Such sentences, the authors note, are good candidates for understanding how we process complex passages.
The researchers then compared those eye movements with predictions generated by more than 400 AI language models.
The results showed that AI models' next word predictions can explain the first step of processing each word of a sentence: identifying the word from a sequence of letters. However, they can't explain the next step of integrating that word into the larger meaning of the sentence—a process that is particularly difficult for humans in garden-path sentences, and one which still remains poorly understood.
"The predictability of a word really doesn't even come close to explaining just how much time we spend on difficult words and garden-path sentences," says Timkey. "LLMs were drastically underpredicting the type of difficulty that we experience when reading."
"It's in that second stage of processing—recognizing a word and then integrating it with other words in passages—where we find big gaps between what word predictability can explain and what we need cognitive models to explain," adds Linzen.
The research was supported by grants from the National Science Foundation (BCS-2020914, BCS-2020945, IIS-2504953, and IIS-2504954).
