In VR School, Fish Teach Robots

Scientists use virtual reality for fish to teach robots how to swarm

To the point

• Innovative method: A team of biologists and robotic engineers have developed a virtual reality system for fish to decipher how they school
• Discovering nature's algorithm: They uncovered the natural 'control law' that is used by zebrafish to coordinate behavior with others, a behavioral algorithm that has been tuned over millennia to facilitate effective collective motion.
• Implications for robotics: They tested the natural control law in groups of robotic cars, drones and watercraft, demonstrating its potential for the control autonomous vehicles in the future.

Fish are masters of coordinated motion. Schools of fish have no leader, yet individuals manage to stay in formation, avoid collisions, and respond with liquid flexibility to changes in their environment. Reproducing this combination of robustness and flexibility has been a long-standing challenge for human engineered systems like robots. Now, using virtual reality for freely-moving fish, a research team based in Konstanz has taken an important step towards that goal.

"Our work illustrates that solutions evolved by nature over millennia can inspire robust and efficient control laws in engineered systems," said first author Liang Li from the University of Konstanz. Co-author Máté Nagy from Eötvös University underscores this: "The discovery opens up exciting possibilities for future applications in robotics and autonomous vehicle design."

Deciphering nature's hidden algorithm

Using a virtual reality (VR) setup that mimics natural schooling, researchers placed individual juvenile zebrafish into networked arenas where each fish could freely interact with 'holographic' virtual conspecifics. Each virtual fish was a projection of a real fish, meaning that fish could swim and interact together in the same virtual world. The fully immersive 3D environment lets researchers precisely manipulate visual stimuli and record how the fish respond. This high level of control allowed the scientists to isolate exactly which cues the fish were using to guide their interactions with other fish. In other words, they could reverse engineer the behavior of schooling in zebrafish to understand how fish solve the complex problem of coordinating their motion.

The solution, they discovered, was a simple and robust law based only on the perceived position, not the speed, of their neighbors to regulate their following behavior.

"We were surprised by how little information the fish need to effectively coordinate movements within a school," says Iain Couzin, senior author on the study and Director of the Max Planck Institute of Animal Behavior and Speaker at the Excellence Cluster Collective Behaviour. "They use local rules that are cognitively minimal, but functionally excellent."

To see just how realistic the control law was, the team tested it with real fish They conducted a VR "Turing" test, based on the concept of testing whether people can tell if they are interacting with a real human or with artificial intelligence. In the aquatic Turing test, a real fish would swim with a virtual fish that switched between being real and being controlled by the algorithm they discovered. The real fish couldn't tell the difference. They behaved the same whether interacting with a real conspecific or the virtual follower governed by the algorithm.

From fish to machines

To test the broader utility of their discovery, the team embedded it in swarms of robotic cars, drones, and boats. The robots were tasked with following a moving target using either parameters from the zebrafish algorithm or from a state-of-the-art method used in autonomous vehicles called Model Predictive Controller (MPC). Across all tests, the natural control law that fish have evolved delivered performance that was nearly indistinguishable from MPC in terms of accuracy and energy consumption-but at a fraction of the complexity.

Oliver Deussen, a co-author on the study and Professor in computer science at the University of Konstanz and Speaker at the Excellence Cluster Collective Behaviour: "This work highlights the reciprocal relationship between robotics and biology - using robotics to explore biological mechanisms, which in turn can inspire new and effective robotic control strategies."