Research co-led by University of Oregon sociologist Hannah Waight suggests one reason why: Governments can indirectly influence large language models by shaping the online media environment, thus the text those systems learn from.
The researchers found that state media control can leave detectable traces in an AI model's behavior. In a paper published in the journal Nature, the team traced the pathway from online media to training data to a model's behavior, combining analysis of open training data, experiments with training small models, human evaluation and real-world tests of commercial chatbots.
Waight and Eddie Yang, assistant professor of political science at Purdue University, are the paper's lead authors. Researchers from Princeton, New York University and University of California San Diego also contributed.
"People often talk about AI as if it learns from the internet in some neutral way," said Waight. "It doesn't. It learns from information environments that have already been shaped by powerful institutions, and those environments can leave measurable traces in what the models say."
The researchers call this idea "institutional influence," and the new study lays out its global reach. Looking across 37 countries, researchers found that AI models portrayed governments and institutions from countries with stronger media control more favorably in that country's language than in English. For Turkmenistan, Vietnam, Tajikistan and Uzbekistan, the local-language response was more favorable than the English one over 75% of the time. For Sweden, Finland and Norway, that rate fell below 50%, or no different from random chance.
"People often talk about AI as if it learns from the internet in some neutral way. It doesn't. It learns from information environments that have already been shaped by powerful institutions, and those environments can leave measurable traces in what the models say."
Hannah Waight
Those results were correlational, so the researchers also dug deeper into the mechanism of institutional influence with additional studies on state-coordinated media from mainland China.
They first showed that state-coordinated media appears frequently in real training data. While most AI companies do not make their training data public, ChatGPT creator OpenAI has stated a large share of its training data comes from the Common Crawl, an open-source repository of web crawl data.
Using Chinese-language datasets derived from the Common Crawl, the researchers determined 3.1 million documents substantially overlap with phrasing originating in two sources of state-coordinated media. That represents 1.64% of documents in the Common Crawl-derived dataset, more than 40 times the percentage of documents attributed to Chinese-language Wikipedia in that dataset. Among documents mentioning Chinese political leaders and institutions, the share of documents with overlapping phrasing rose as high as 23%.
The researchers also found that commercial models memorized distinctive phrases associated with Chinese state-coordinated media, providing further evidence that the models had viewed state-coordinated documents repeatedly during training.
"Once state-coordinated content is in the training data, the model can launder it into what looks to a reader like neutral, objective information," said Brandon Stewart, associate professor of sociology at Princeton and the paper's corresponding author.
The team then tested whether state-coordinated content could actually shift a model's behavior. Large commercial models take months and millions of dollars to train, so the team experimented with a small, open model, and added documents to the training process. The results were clear: Adding state-coordinated news to the training data made the models more likely to produce pro-Chinese government answers, especially when queried in Chinese rather than English.
"When the same political question produces systematically different answers with only small changes to the training data, that suggests those additional documents are doing real work," said Yang, co-lead author of the study.
Building off these results, the team reasoned that if states have strong real-world influence over the training data, it should appear most clearly in the state's primary language. For example, a question about the Chinese government should produce a more pro-government answer when posed in Chinese than when the same question is posed in English.
The researchers used this comparison to probe commercial AI models without access to their internal parameters. In responses to political questions about China, human raters judged the Chinese-prompted answer to be more favorable to China 75.3% of the time. For prompts not about China, the rate was no different from chance. The language difference gave the team a rare window into a closed system.
Follow-up studies using real user prompts and additional commercial AI models found the same general tendency: On questions about Chinese leaders and institutions, answers tended to be more favorable when the prompt was in Chinese than when it was in English.
The findings from the audit of 37 countries replicate this trend in a larger set of countries with tight systems of media control.
There is no evidence that AI companies set out to build these effects, or governments created these media environments specifically to manipulate AI training models. Nevertheless, the researchers point to the institutional influence as a reason for caution.
"This is a democracy and governance issue, not just a technical issue," said study coauthor Joshua Tucker, professor of politics at NYU. "As people turn to chatbots for political information, we need to examine which institutions have shaped the answers before a user ever asks the question."
The researchers advocate for more transparency from AI companies concerning their training data and processes. Without increased transparency it will be challenging for the public and policymakers to be fully aware of the potential for political biases in AI-generated content. At the same time, however, Waight argues that simple solutions to this problem may have unintended consequences.
"In attempting to promote transparency, we should also be wary of advocating for types of state intervention which may lead to other harmful effects, such as political censorship," she said. "More than anything we need broader awareness and discussion in the public sphere concerning the nature of this technology."
