Dr Heloise Stevance , Schmidt AI in Science Fellow in Oxford University's Physics Department, is a computational astrophysicist developing intelligent recommendation systems for sky surveys. In this article she discusses the ethics of delegating scientific decision making to our computers.
Recently one of the most prestigious artificial intelligence (AI) conferences (NeurIPS) was caught accepting submissions with hallucinated citations . Not a handful either - over 100 instances.
The response form the NeurIPS board is pretty telling of the times we live in: 'Even if 1.1% of the papers have one or more incorrect references due to the use of LLMs, the content of the papers themselves are not necessarily invalidated.' Effectively, a "who cares" attitude, followed by oxymoronic statements such as being 'committed [...] to best ensure scientific rigor.'
This "relaxed" approach to research ethics and what "scientific rigour" means should not be the blueprint for how science at large approaches AI. Neither should it be a sign that we have to choose between AI and our principles. We just have to think carefully about how and where we are using it.
Delegating scientific tasks and decision making to computers is not a new phenomenon, and it isn't necessarily a bad thing. Some tasks can be automated or accelerated to allow for faster, bigger, discoveries . But delegating scientific processes to our machines always comes at a cost, which should be weighed against the benefits.
The decisions we take now will influence the data we create for future generations of scientists. We have a duty of care and a shared responsibility to our peers, present and future.
The decisions we take now will influence the data we create for future generations of scientists. We have a duty of care and a shared responsibility to our peers, present and future.
As a sky survey astronomer who uses computers to sift through vast quantities of data, I have spent a long time considering this issue. My goal as a scientist is to uncover new knowledge whilst upholding the standards of the scientific method: Reproducibility, Falsifiability, and Bias Awareness.
My particular interest is in learning about how distant explosions of stars create new elements in the Universe that can then go on to form planets, people and even smartphones. These explosions are only visible for a few days to a few weeks so we have to find them quickly to gather the data required to do the analysis.
To do this, we have programs such as the ATLAS sky survey which scour the sky over and over, night after night. In essence we play a game of "spot the difference" with the cosmos, comparing reference images to the recent ones, looking for changes and newly born sources of light.
But the sky is big. On a very dark, moonless night, your eyes can see a few thousand stars; the ATLAS sky survey can see a billion bright sources. It would take me an entire year to visually compare the before and after pictures that ATLAS takes on a single night.
In short, the challenges I face in modern astronomy are two-fold and all too common within science:
- Immense quantities of data (volume, speed and/or dimensionality).
- Not enough time.
Whatever task we use AI and machine learning for, the key question we need to be asking is 'How will this influence the legacy and longevity of my findings?'
Even if you're not in the business of tracking explosions that come and go, if you are a researcher, you will have funding application deadlines, conferences to prepare for, a contract to renew, etc. The time pressure in science is felt across the board.
And so we delegate to our computers. But whatever task we use AI and machine learning for, the key question we need to be asking is 'How will this influence the legacy and longevity of my findings?'
With this in mind, there are three basic principles I stick to when considering whether it is wise or right to delegate a task or decision to a tool.
1. Software is only open if the underlying data is open
Beware "open-washing": models are not reproducible without training data. The word "open" was originally used in the tech industry to designate open source software, or "free" software - software granting everyone the freedom to see, use, modify, and redistribute. Perfect for reproducibility and knowledge sharing.
Unfortunately, just because you see the word "open" nowadays does not necessarily mean these standards are upheld. Even if you can reproduce the result a model gives you, that is not enough for long term scientific reproducibility. Someone other than the model builder must be able to reproduce and understand the training of the model itself. But if the underlying data and training algorithms are not accessible, this is impossible to do.
And if, like me, you are the person releasing models, take the time to release documented data alongside the code, for example on Zenodo .
2. Use the simplest tool that works
Someone other than the model builder must be able to reproduce and understand the training of the model itself. But if the underlying data and training algorithms are not accessible, this is impossible to do.
There is sometimes a pressure to use the newest, "most-advanced", tool or model. We can presume that if we want to make state-of-the-art science, we have to use the most innovative, state-of-the-art models. But this is not necessarily the case. Instead, I advocate starting with the simplest solution possible. If the simple model works, stop now! If it doesn't, we analyse the specific failures to point us in the direction of the next approach to try.
Finding a simple solution to a complex problem is extremely valuable, because it means that my colleagues, present and future, will have an easier time understanding my models. The amount of highly specific knowledge required to understand all the caveats of a method is sometimes called intellectual debt. Just like technical debt, it prevents reproducibility. In a company it means lost revenue; in science it means lost knowledge and reduced scientific rigour.
Using the simplest solution possible also helps to maintain sovereignty. Let's take an extreme example: imagine I use a third-party AI agent as a research assistant. What happens when the company that provides those agents decides to increase the subscription price several fold because they're going public and need to turn in a profit? Can my research grant take the hit? Or what if they decide the model version I was using is no longer available, but the new version behaves differently and/or returns different results? What if the company folds and the service vanishes? I could lose days, weeks, even years of (publicly-funded) research.
3. Be sceptical of what you don't understand
It can be tempting to prompt your way to apply a complex, 'smart-sounding' solution that looks like it works. But science isn't about trying to make something work and stopping there - that's called confirmation bias. Science is about seeing something work and asking 'why?'
Large Language Models, like ChatGPT or Claude, have lowered the barrier to entry for non-specialists to code complex tools and algorithms. It can be tempting to prompt your way to apply a complex, "smart-sounding" solution that looks like it works. But science isn't about trying to make something work and stopping there (that's called confirmation bias). Science is about seeing something work and asking "why?", then trying to make it fail to see where it breaks down (and reporting this along with the successes).
I'll confess that I am not immune to the "Fear Of Missing Out (FOMO)." Following ethical research principals may be the right thing to do, but it doesn't stop the anxiety - will I get left behind if I take too long to test my tools or if I don't create an army of AI agents to read the literature and write papers for me?
The key is remembering that I became a scientist to understand the natural world. Writing papers and grants is a means to an end, not the goal itself. Ultimately, AI can help me do the science but it can't understand it for me. If I can't reproduce my results, you might as well call me an Astrologer.
