The way people are finding information online could be entering a new phase, as website providers like Cloudflare push back again AI systems that collect and summarise content. RMIT experts say this shift could have significant implications for content creators, website operators and the quality of information available online.
Dr Dana McKay, Associate Professor in Innovative Interactive Technologies, School of Computing Technologies:
"Searching the web is about to look a bit different. We are going to see a major provider of web services (one third of the world's most popular websites) block Google, the search engine most of us routinely use.
"Cloudflare, the provider, isn't doing this wilfully; they are protecting those who literally write the internet. Where once Google linked to the pages people wrote, now they suck up content and package it for us as AI answers, cutting the writers out of the loop.
"Worse, the cost for those producers of Google (and other search engines) ingesting their content has gone up with AI crawlers. This could result in a drop in the quality of AI summaries specifically, and search results more generally, given search engines can't access content anymore.
"How this tension will be resolved isn't clear, but while it is being resolved, we should be especially cautious about the accuracy of AI summaries. If we can't find something we think should be there, it might be worth trying another search engine."
Dana McKay is Associate Dean, Interaction, Technology and Information in the School of Computing Technologies at RMIT University. Dana's research focuses on ensuring advances in digital information technologies make the world a fairer and more equitable place.
Dr Damiano Spina, Senior Lecturer in Interactive Information Retrieval, School of Computing Technologies:
"The way we produce and consume information on the web is changing very rapidly. Although organisations such as the World Wide Web Consortium help us maintain good practices and standards for how we manage and use this valuable shared global resource, the adoption of generative AI is challenging our understanding of how information should be consumed.
"This has direct implications for how information producers - organisations and individuals who create, host or publish content on the web - estimate the resources needed to keep their servers operational.
"It also raises questions about how the costs of producing and maintaining information should be covered, who or what should have access to it, and for what purposes it should be used.
"AI scrapers and AI agents that access and process information force us to rethink how the principles of the web should be maintained. AI systems may retrieve and process large amounts of online content without visiting websites in the same way as human users.
"This could increase server costs without necessarily contributing to the information producers' revenue. We need to consider how web content is accessed, how producers are compensated, and how the openness and accessibility of the Web can be preserved."
Damiano Spina is a Senior Lecturer and former DECRA Fellow in the Interaction, Technology, and Information discipline at the School of Computing Technologies, RMIT University. Damiano's research focuses on the design and evaluation of interactive information access systems, including search engines, conversational agents, and retrieval-augmented generation.
