A new study suggests that traditional learning activities like making notes remain critical for students' reading comprehension and retention, while also suggesting that large language models (LLMs), such as ChatGPT or Microsoft Copilot, could be a useful tool for helping students clarify, explore, and contextualise learning material.
Although many students are already using LLMs, there is a lack of research on the impact of generative AI on the fundamental processes of learning.
The study by Cambridge University Press & Assessment and Microsoft Research, which is published today (Thursday) in the journal Computers & Education , is one of the first randomised classroom experiments to investigate how LLMs affect students' reading comprehension and retention.
The research involved 405 secondary school students aged 14-15 attending seven different schools across England.
The students were asked to study texts covering topics from the UK's national History curriculum: one about apartheid in South Africa, and the other about the Cuban missile crisis.
The students were divided into two groups. One group was asked to study one text with an LLM (ChatGPT 3.5 turbo) and another text by writing notes. The second group was also asked to study one of the texts with the LLM, but, for the other text, they were asked to combine using the LLM and making notes. In all LLM conditions, students were given a brief tutorial and allowed to use the tool however they liked.
Three days later, and without advance warning, the students were asked questions about the two texts designed to see how well they understood and remembered the information. For example: What horrific event happened at the Soweto Youth Uprising in 1976? And: Explain the role of the Soviet Union in the Cuban Missile Crisis.
After both the study and test sessions, students were asked about the task, for example whether they enjoyed it or found it interesting.
The results suggest that either making notes or making notes combined with using an LLM, are better than just using the LLM alone for helping students understand and remember new information. However, students enjoyed using the LLM to engage with and explore relevant topics beyond the text.
The study's first author, Dr Pia Kreijkes, a senior researcher at Cambridge University Press & Assessment, UK, said: "We know that students are using chatbots and other AI tools, including to help them with their schoolwork. However, there has been very limited research on how LLM use influences students' ability to understand and remember information. Our study shows that students enjoyed using AI chatbots but note taking was more effective for learning outcomes
"Our findings can help guide the use of LLMs for learning. In particular, they indicate that students should take notes separately from using LLMs to avoid copying the LLM. They also indicate that students should receive training and guidance on how to use LLMs to support active and constructive learning."
"Teachers could also benefit from their students' use of LLMs. For example, in the future, teachers may be able to leverage insights from students' LLM interactions to understand where support is needed and tailor class materials accordingly."
Dr Jake Hofman Senior Principal Researcher from Microsoft Research said: "I was struck by how many students used the LLM to try to deepen their understanding — asking about historical context, clarifying unfamiliar references, and exploring the significance of key events. Rather than viewing traditional learning techniques, like note-taking, and newer generative-AI approaches as competing alternatives, we should view them as complementary."
