Three heads are better than one.
Versions of this proverb are found worldwide and throughout history. Yet, in the race to achieve artificial general intelligence, engineers have centralized AI development and training to accelerate progress, even at the risk of single-point failures and data privacy violations. Alternatively, decentralized frameworks have struggled to match the robustness of centralized systems - what if one of the heads has bad intentions?
Presented at ICML 2026, one of the most important annual conferences on machine learning, researchers from the Machine Learning and Data Science Unit at the Okinawa Institute of Science and Technology (OIST) have unveiled a federated learning (FL) architecture that solves the long-standing conflict between robustness and efficiency. By mathematically proving the underlying principles, the team has raised the bar across the rapidly evolving, often proprietary AI development scene.
"Our model offers a genuine path forward for FL that is both communication-efficient and robust against bad actors. Previous iterations have strived to be either: by giving the system the ability to remember past input from client-side devices, our solution is both," says Kaoru Otsuka, study first author and PhD student in the research unit.
Central point of failure, slow model development, or treacherous generals?
In contrast to centralized AI development systems, FL anonymizes data and reduces the risk of leaks from a single point of failure. Here, AI models are developed on a central server, but training is distributed across individual client devices. This ensures that model training on sensitive data occurs exclusively at the local device rather than sending it to a central location. Each client, in turn, sends numerical outputs - also called gradients - back to the server to update global model parameters and improve overall performance.
A widespread example of FL is mobile keyboard prediction, where a local instance of the model on each device predicts the next word in a draft text message, trained on each user's text messages. Rather than exporting text messages wholesale, each device sends only the computed gradients, which are aggregated on the central server to train the algorithm's overall performance.
However, FL is vulnerable to the so-called Byzantine Generals Problem, a hypothetical scenario named after a dilemma that could have been faced by allied generals in the Byzantine Empire, which illustrates the importance of robustness against outliers. The problem states that to achieve success on the battlefield, all allied generals in an army must either order an attack or a full retreat in unison; if any general, either treacherously or incompetently, issues an order that runs counter to the others, the entire army faces a catastrophic defeat. In FL, poor gradients from just one of the client devices can severely degrade overall model performance or expose serious data vulnerabilities.
To protect against Byzantine clients, current-generation FL networks often aggregate all gradients before updating global parameters to smooth out outliers. But the communication time of this approach scales dramatically with the number of client devices, drastically slowing down model development. The commonly used alternative, known as partial participation, instead averages across small client samples at short intervals. But since poisoned gradients can make up a significant fraction of a sample, this regime is not robust.
Storing memories to maintain speed and safety
To resolve this tension between efficiency and robustness, the researchers developed a novel solution that makes partial participation robust to Byzantine clients. Otsuka explains: "Our algorithm remembers past client gradients and includes them in future aggregations. Imagine a partial participation system with 10 clients. To reduce communication costs, only three clients are sampled per round. If two of those are Byzantine, their updates can disproportionally influence the aggregated result. Instead, we store the most recent gradients from all ten clients and aggregate the fresh updates together with stored memories of all client gradients that were not sampled this round. By continuously updating and retrieving these cached memories, FL models can reduce the influence of malicious signals while maintaining the speed of partial participation."
Professor Makoto Yamada, head of the unit, points to the potential of the algorithm. "The market is evolving at an extremely rapid pace, and new algorithms tend to be either secret or empirically shown to work only in specific settings. As agentic LLMs and other machine learning applications spread to more aspects of our daily lives, we hope that proving the mathematical validity of this communication-efficient, Byzantine-robust algorithm can raise the bar for safe federated learning across the field."
