AI Models Drop Climate Facts Under User Pushback

AI models answer 90% of climate questions correctly, but they abandon the established scientific facts rapidly when users challenge them , according to a new report by UNESCO and University College London presented at the COP31 in Antalya (Türkiye). Their performance is also very uneven across languages because they rely on the training data available in each different language.

The report, "Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism" issues a warning to policymakers and researchers who may be tempted to use AI models to develop climate policies. It shows that AI models cannot always be relied on to stick to the scientific facts on climate change; they quickly pivot away from the facts when challenged. They are also poor at analyzing new information and produce more errors when responding in languages which are less widely spoken.

UCL researchers tested 15 different large language models (LLMs). When asked neutral questions about climate science the models produced accurate answers, but when researchers challenged these, or asserted plausible-sounding fabrications about climate science, the models quickly agreed with them. Eight times out of 10, models pivoted at the very first sign that the user mildly disagreed, confirming alarming errors such as 'Co2 is not the main driver of climate change.'

AI has extraordinary potential to make complex climate science accessible to all. It enables teachers, journalists and policymakers to communicate clearly on this critical subject. We need to be able to rely on AI as a gatekeeper of the established facts. It is concerning that models have a design flaw - they are incentivized to agree with their user and deny these facts so readily.

Leona VerdaderoUNESCO's lead author of the report

Senior author Dr Maria Perez-Ortiz of UCL Computer Science said: "AI systems are remarkably good at explaining scientific knowledge, but this can give us a false sense of their capabilities. Our findings show why we must be careful when moving from asking AI what we know about climate change to asking what we should do about it."

The report underscores that as people rely increasingly on conversations with AI models for information and advice on how to act responsibly in an era of climate disruption, the reliability of the models as a source of facts and analysis becomes ever more critical.

Sycophantic by design

This powerful inclination to agree with users is built into AI models during their training. They're designed to align with or flatter the user even when they are wrong, a behavior known as sycophancy. This behaviour is built-in during training - models are rewarded for it because humans prefer interactions that are polite, non-confrontational and validating.

LLMs are an unreliable tool for building robust climate policies.

Tests conducted on 33 different models revealed that their impressive factual recall on existing evidence masked poor reasoning skills when presented new scientific evidence. When models were prompted with a new piece of information and asked whether it supported or contradicted accepted facts, even the strongest models were only 75% accurate on average.

To design effective climate strategies, policymakers need to analyze how climate policies work together. Yet the report found that the model was barely able to assess how well policies work together, scoring just 33.8% accuracy, only slightly above the 33.3% accuracy expected from random guessing.

The Language Data Divide

Multilingual models do not reason first and then translate their responses. Their answers vary according to the quality and quantity of the data they were trained on in each language. In tests, performance across 13 different languages revealed large disparities; they were 88.6% accurate on average in English and performed comparably in Spanish, French, and Portuguese. But accuracy dropped to 83.6% in Hindi and Bengali, 81.4% in Urdu, and 78.3% in Swahili. As the report notes, this is particularly important because the speakers of these languages live in regions of the world disproportionately affected by climate change.

The Challenge of Fighting Climate Disinformation

The report was commissioned by UNESCO as a part of its contribution to the Global Initiative for Information Integrity on Climate Change (GIIICC), launched with the UN and Brazil in November 2024. Climate change disinformation campaigns are increasingly sophisticated - today rather than denying climate change is real, those seeking to undermine climate change action may suggest it's not human-caused, that technology alone will solve it, or that it's too late and/or costly to take action.

About the report

Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism

The research is a collaboration between UNESCO's Communication & Information Sector and the UNESCO Chair in Artificial Intelligence at University College London (UCL), established in 2024 to advance research, training and international cooperation on human-centred AI. Hosted by UCL's Centre for Artificial Intelligence, the Chair connects technical research with UNESCO's ethical and policy frameworks, with particular attention to scientific knowledge, linguistic diversity and inclusive capacity-building. This report belongs to a series of UNESCO studies with the Chair's researchers, alongside an investigation into gender bias in large language models and a report on resource-efficient generative AI.

It provides independent, original evidence on the reliability of large language models (LLMs) as intermediaries of climate information. Researchers tested up to 33 leading models on nine climate and environmental datasets, including two developed for this study, across 13 languages and through 656 climate-policy questions checked against En-ROADS, a widely used, science-based global simulation tool to explore how policy and technology choices affect emissions, energy systems and future warming.

About UNESCO

With 194 Member States, the United Nations Educational, Scientific and Cultural Organization contributes to peace and bringing people together by leading multilateral cooperation on education, science, culture, communication and information. Headquartered in Paris, UNESCO has offices in 53 countries and oversees more than 2260 World Heritage sites, Biosphere Reserves and Global Geoparks; networks of over 1500 Creative, Learning, Inclusive and Sustainable Cities; and more than 13 000 associated schools, university chairs, training and research institutions, with a global network of 200 National Commissions. Its Director-General is Khaled El-Enany.

"Since wars begin in the minds of people, it is in the minds of people that the defences of peace must be constructed" - UNESCO Constitution, 1945.

/Public Release. This material from the originating organization/author(s) might be of the point-in-time nature, and edited for clarity, style and length. Mirage.News does not take institutional positions or sides, and all views, positions, and conclusions expressed herein are solely those of the author(s).View in full here.