A government-developed AI tool has been used for the first time to review public responses to a consultation - helping save time and improve efficiency.
- AI technology, 'Consult', built by the UK government as part of the Humphrey suite has been used to speed up analysis of what the public and experts told the Scottish Government in a recent consultation
- Nearly identical results were found by AI after expert review, ranking themes that were most important for policymakers to take on board
- While currently in trial with more development taking place, AI will analyse other consultations responses in a bid to save officials from 75,000 days of manual analysis every year, which costs £20m in staffing costs, helping to create a more agile, effective state refocused on delivering Plan for Change
A new AI tool has summarised what the public have told the government in response to a consultation for the first time - providing nearly identical results to officials.
The tool, called 'Consult', was first used on a live consultation by the Scottish Government when it was seeking views on how to regulate non-surgical cosmetic procedures - like lip fillers and laser hair removal - as use of the treatments has risen.
The tool now set to be used across departments in a bid to cut down the millions of pounds spent on the current process, which often includes outsourcing analysis to expensive contractors - helping to build a productive and agile state to deliver the Plan for Change .
Reviewing comments from over 2,000 consultation responses using generative AI, Consult identified key themes that feedback fell into across each of six qualitative questions. These themes were checked and refined by experts in the Scottish Government, the AI tool then sorted individual responses into themes and gave officials more time to delve into the detail and evaluate the policy implications of feedback received.
As this was the first time Consult was used on a live consultation, experts at the Scottish Government manually reviewed every response too. Identifying what an individual response is saying, and putting it in a 'theme' is subjective, humans don't always agree. When we compare Consult to the human reviewer, we see they agree the majority of the time - with differences in view having a negligible impact on how themes were ranked overall.
'Consult' is part of 'Humphrey', a bundle of AI tools designed to speed up the work of civil servants and cut back time spent on admin, and money spent on contractors. It forms part of the government's plan to make better use of technology across public services, in a bid to target the £45 billion in productivity savings that it offers while creating a more agile state that can more effectively deliver the Plan for Change.
Technology Secretary Peter Kyle said:
No one should be wasting time on something AI can do quicker and better, let alone wasting millions of taxpayer pounds on outsourcing such work to contractors.
After demonstrating such promising results, Humphrey will help us cut the costs of governing and make it easier to collect and comprehensively review what experts and the public are telling us on a range of crucial issues.
The Scottish Government has taken a bold first step. Very soon, I'll be using Consult, within Humphrey, in my own department and others in Whitehall will be using it too - speeding up our work to deliver the Plan for Change.
The Scottish Government's Public Health Minister Jenni Minto said:
Using the tool was very beneficial in helping the Scottish Government understand more quickly what people wanted us to hear and our respondents' range of views. Officials were reassured through the process that the AI was doing a good job, supporting us to undertake the analysis that will inform our next steps.
Using this tool has allowed the Scottish Government to move more quickly to a focus on the policy questions and dive into the detail of the evidence we've been presented with, while remaining confident that we have heard the strong views expressed by respondents.
While these early results are promising, 'Consult' is currently in trial. More evaluation covering the accuracy and efficiency of the tool will take place to ensure it's working properly ahead of final rollout decisions.
Across the 500 consultations the government runs annually, the tool could help save officials from around 75,000 days of analysis every year, which costs the government £20 million in staffing costs.
In doing this, the technology will help create a more agile state that can more easily respond to new challenges and effectively deliver the Plan for Change.
Officials who worked with Consult from the Scottish Government on this first live test commented that they were "pleasantly surprised" that AI analysis provided a "useful starting point" in its initial analysis, with others noting that it ultimately "saved [them] a heck of a lot of time" and allowed them to "get to the analysis and draw out what's needed next".
They also added that the use of Consult "takes away the bias and makes it more consistent", by removing opportunities for individual analysts to "project their own preconceived ideas".
With some consultations receiving tens or hundreds of thousands of responses, and given the strong levels of accuracy demonstrated in early tests, Consult will soon be used on major consultations without officials manually reviewing every response individually.
That said, Consult has been designed to keep the experts in the loop throughout. Officials will always review the themes and how responses are sorted into them through an interactive dashboard that will allow them to filter and search for insights.
Notes
The response to the Scottish Government consultation will be published before the end of June. The consultation will inform the content of a Non-Surgical Cosmetic Procedures Bill that was announced on 6 May.
The first live evaluation of Consult shows that it secured an F1 score (a common measure of alignment for AI tools) of 0.76, widely considered 'good' when evaluating the performance of AI tools.
The full evaluation, published today, can be found here . We expect further testing and evaluation of the tool to happen in coming months, ahead of any decisions about wider rollout.
DSIT
