EPFL researchers have developed new software - now spun-off into a start-up - that eliminates the need for data to be sent to third-party cloud services when AI is used to complete a task. This could challenge the business model of Big Tech.
The use of artificial intelligence for everyday tasks has grown rapidly in the past three years. AI models are increasingly useful to process sensitive data such as patient records, customer requests or any work-related confidential documents.
Each time AI is asked to perform a task, the query starts locally on a user's computer and is then sent into the 'cloud' where the AI generates an answer using powerful hardware, a process known as inference. The answer is finally sent back to the user's local computer.
Huge data processing capabilities are nowadays used to do this, requiring enormous data centers which are also needed to train AI models, such as ChatGPT, Gemini and Claude. This means that both inference and training are currently controlled almost exclusively by Big Tech.
Plug and Play: distributed AI made simple
Now, EPFL researchers Gauthier Voron, Geovani Rizk and Rachid Guerraoui, from the Distributed Computing Laboratory (DCL) in the School of Computer and Communication Sciences, have released new software that allows users to download open-source AI models and use them locally, with no need for the cloud to answer questions or complete tasks.
The new software, called Anyway Systems coordinates and combines distributed machines on a local network into an on-premise cluster. It uses robust self-stabilization techniques to optimize the usage of underlying local hardware, contradicting the common belief that huge data centers are needed to deploy AI models.
It can be installed in just half an hour on a network of local machines with no data leaving the network, guaranteeing privacy and sovereignty. A very big AI model like GPT-120B, the latest and largest open model from OpenAI, can be downloaded and deployed on Anyway Systems in a few minutes, requiring no more than 4 machines with 1 commodity GPU each (costing around 2300 CHF each), instead of an expensive specialized rack enclosure (costing around 100,000 CHF), which until now has been assumed necessary to run an AI model.
"For years people have believed that it's not possible to have large language models and AI tools without huge resources, and that data privacy, sovereignty and sustainability were just victims of this, but this is not the case and smarter, frugal approaches are possible," said Professor Rachid Guerraoui, head of the DCL.
Privacy, sovereignty and sustainability
When a user's data is sent to the cloud, there are crucial questions around security and privacy, particularly whether that data is used to further train or improve AI models. As well, the reliance on large, global cloud providers for AI services raises questions around AI sovereignty because it transfers control over critical national assets-data, algorithms, and infrastructure-from a domestic entity to transnational corporations.
Moreover, the immense computing power needed to answer AI queries in the cloud - it is estimated that inference accounts for 80 to 90% of AI-related computing power - is driving the rapid expansion of the huge data centers for AI that consume enormous amounts of energy and water.
"Anyway Systems shines on inference but it could also help reduce the resources needed for training," explained Guerraoui. Pilot testing has shown that when a model is downloaded and run on scattered local machines instead of a huge cloud, we may lose a bit of latency - that is time to respond to a prompt - but not any accuracy."
From blockchain to AI?
"We say that Anyway Systems is simple, scalable and safe," continues Guerraoui. Earlier variants of the Anyway algorithm were developed by the DCL many years ago where researchers have long been focused on distributed computing, fault tolerance, optimization and privacy.
The DCL's earlier algorithms were existing solutions for other challenges for technologies such as blockchain and cryptocurrency. Three years ago, Guerraoui and his colleagues had the idea to apply self-stabilization techniques to AI, finding an almost perfect fit.
"As a lab we might be unique in working on robust distributed computing and machine learning together from both a theoretical and a practical perspective and we turned our attention to using self-stabilization techniques for AI. They worked! We thought, let's optimize and optimize and they worked even better, the result is almost too good to be true," Guerraoui said.
Looking ahead - your own AI at home
Anyway Systems was recently chosen as one of six inaugural grantees of the Startup Launchpad AI Track - powered by UBS, Switzerland's first grant program dedicated to AI. Selected from over 50 proposals, these projects are receiving funding and tailored support to accelerate their journey from prototype to market readiness.
The software has gone beyond the prototype phase and is now being tested in companies and administrations across Switzerland, including at EPFL. Early users are currently evaluating any trade-offs in terms of speed, accuracy and quality.
"Anyway Systems represents an interesting and appealing technology that optimizes resource usage while ensuring data security and sovereignty and could be an AI gamechanger," said Professor David Atienza, associate vice-president of research centers and technology platforms at EPFL. "Its sustainable approach aligns perfectly with the needs of EPFL's advanced computing platforms and will play a pivotal role in shaping the trajectory of future AI development at EPFL to consume fewer resources with the new deployment of LLM models such as Apertus."
For now, Anyway Systems won't work on a single desktop or laptop at home but the history of computing shows that optimization often occurs quickly.
"Your phone contains crazy amounts of information that would have been unimaginable a few years ago and now you do everything on it. It can beat all the best 100 chess champions at the same time, whereas the computer that was needed to beat Kasparov was enormous. History tells us this is the way things go. What we're saying is that we will be able to do everything locally in terms of AI. We could download our open-source AI of choice, contextualize it to our needs, and we, not big-tech, could be the master of all the pieces," concluded Guerraoui.
Q&A
What is the difference between Anyway Systems and Google AI Edge?
Google AI Edge is meant to be run on mobile phones for very specific and small Google made models with each user running a model constrained by the phone's capacity. There is no distributed computing to enable the deployment of the same large and powerful AI models that are shared by many users of the same organization in a scalable and fault-tolerant manner. The Anyway System can handle hundreds of billion parameters with just a few GPUs.
What is the difference between Anyway Systems and other solutions that allow people to run local LLMs such as Llama or msty.ai?
Most of these approaches help deploy a model on a single machine, which is a single source of failures. To deploy the most powerful models they need to buy and invest in very costly machines, basically the same kind you would find in a data center. Hence, if you have a single commodity machine, the cited solutions are useful for deploying small models. If you have several commodity machines, you can't combine them efficiently with the cited solutions to deploy a large model, and even if you could, it would require a team to manage and maintain the system. The Anyway System does this transparently, robustly and automatically. The fact that a machine fails, or leaves or joins the network is transparent in Anyway Systems, modulo a small change in latency (the time to respond to a request).
AI models are constantly being improved and fed, how are these improvements reflected locally?
As the Anyway System allows any open-source AI model to be deployed locally, feeding it with local and sensitive data becomes completely safe and acceptable, handing control back to the user.
Risk Warning: Cryptocurrency is a unregulated virtual notoriously volatile instrument with a high level of risk. Any news, opinions, research, data, or other information contained within this website is provided for news reporting purposes as general market commentary and does not constitute investment or trading advice.
