The academic team from Imperial contributed to the development by Thomson Reuters of a frontier AI model named Thomson that has been found to rival the performance of the big foundation models, even though its $40 million development cost was just a fraction of the billions invested in building the leading models.
The technology company, which specialises in supporting professions such as law and accountancy, says the approach could allow organisations to build and deploy frontier AI systems without the resources of the major frontier labs.
This would help meet a well-recognised need for governments and companies around the world to build their own highly capable 'sovereign' AI systems that would help ensure vital interests are not in the hands of external organisations.
Thomson Reuters developed Thomson by building on an open-source model named Qwen, aided by techniques it developed in partnership with Imperial.
While open-source models have been available to build on for a few years, it has been an open question until now whether organisations other than the biggest AI companies can substantially improve them while retaining their general capabilities. The research has provided evidence that they can: Drawing on the multinational's proprietary data and its deep domain expertise in fields such as law, Thomson is particularly adept at specialist professional tasks but also performs well as a general-purpose large language model.
Dr Jonathan Richard Schwarz , Head of AI Research at Thomson Reuters and Associate Director of the Thomson Reuters–Imperial Frontier AI Research Lab at Imperial College London, said: "Building a sovereign AI originally meant training your model from scratch. With a billion dollars, that might just produce something that is competitive. But our alternative approach changes that."
Addressing technical challenges
The model is based in substantial part on work at the Thomson Reuters–Imperial Frontier AI Research Lab, a recently launched partnership between the two organisations, which identified solutions to some of the key obstacles to successfully improving on open-source foundation models.
One of the biggest obstacles is catastrophic forgetting, a tendency of foundation models to lose many of their existing skills in the process of acquiring new ones.
A team from the lab led by Dr Schwarz tested several different approaches to preventing this from happening. One of the most successful, they found, was to train the model intensively enough to substantially improve its performance on new tasks, and then to blend the weights (numerical parameters that determine how a model behaves) from the newly trained model with those from the original to yield a model that combines old and new skill sets.
To judge how well each approach worked, the researchers used an evaluation methodology developed at their lab known as CapTrack, a compute-efficient evaluation method that tracks whether a model can perform a task, whether it will do so by default, and how it carries it out, helping researchers spot divergence in general capabilities during training.
The team also addressed the challenge of re-training a foundation model to adhere to ethical principles such as safety and impartiality, drawing on the Public AI Constitution , an open-source governance framework devised by experts from several countries and designed to reflect universal values such as the Universal Declaration of Human Rights.
To this end, they identified patterns in the open-source model's internal activations associated with unwanted behaviour and used an optimisation algorithm to find weight adjustments that weakened those patterns while preserving its useful abilities. They then reinforced desired behaviours by training the model to favour responses that followed the AI Constitution.
The research is described in a new technical report, available as a PDF download published by Thomson Reuters.
The power of convergence science
Professor Alessandra Russo, Director of the Thomson Reuters–Imperial Frontier AI Research Lab and Convening Co-Director of the School of Convergence Science at Imperial College London, said: "To harness the benefits of AI we need more than ever-larger models and ever-greater computing power – we need to combine technical expertise with deep domain expertise and a clear understanding of the real-world contexts in which the technologies will operate. This early result of our work with Thomson Reuters demonstrates the genuinely useful innovations that academic and industry experts achieve when they bring these qualities together in a partnership. It is exactly the kind of collaboration that Imperial's School of Convergence Science is designed to enable."
Imperial's School of Convergence Science is an initiative of the university's Science for Humanity strategy, facilitating research that addresses societal challenges of significant scale that defy conventional approaches through the deep integration of disciplines and cross-sector partners.
The Thomson Reuters–Imperial Frontier AI Research Lab is the first strategic partnership within the School of Convergence Science and aims to tackle foundational challenges in AI safety, reliability and societal impact.
