Have you ever seen birds flying across the sky in shifting, mesmerizing patterns? Or ants using their own bodies to form a living bridge that other ants can walk across?
In nature, organisms we normally think of as simple can solve remarkably complex tasks when they act together. As a group, they can, for example, travel distances that would be impossible for a single individual.
Researchers can recreate this kind of behavior in mathematical models and algorithms to better understand how nature works. But these models can also inspire new kinds of artificial intelligence (AI) systems - including systems for making music.
"With the AI boom of the last few years, many people have become interested in how humans and machines can create music together. One way to explore this is to look at how we, as humans, can collaborate with an ensemble of AI agents," says Pedro Pablo Lucas Bravo.
How can humans and AI agents play together?
Photo: Annica Thomsson
He is a PhD candidate at the RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion and, part of University of Oslo and draws on insights from animal behavior in his research.
His project has two main parts, he explains.
"First, I study how a swarm of AI agents behaves when they are involved in creating music. What do they do together, and how do they shape the music?"
The second part is to study how humans can collaborate with these agents:
"If we can establish a good form of collaboration, I think it could open exciting possibilities for music production and performance."
Musical AI that doesn't need big data
Today, AI is already widely used in music production - both as a creative assistant and as a tool for speeding up routine tasks. Devices like "Suno" allow users to generate music, including vocals and instrumentation, given text or audio prompts.
When AI systems generate realistic-sounding music, they typically rely on huge datasets: millions of audio samples or many thousands of hours of music.
Working with a swarm of AI agents is different. These systems are built on very simple rules and often require little data - sometimes almost no data at all, Kyrre Glette, a professor at RITMO and Lucas Bravo's supervisor explains.
"It provides a different way of expressing yourself creatively. You're working with a model based on micro-rules, and together they create a fascinating, dynamic system," he says.
The basic idea is to start with a type of biological behavior and translate it into a set of rules. Usually, all the agents follow the same simple rule set - for instance, one inspired by how ants move and coordinate. But when you put many agents together, their interactions can lead to unpredictable yet rich musical behavior, like complex musical patterns in melodies or harmonies.
Photo: Pedro Pablo Lucas Bravo
Adds a live experience to AI music generation
So-called swarm systems, or multi-agent systems, are already used in areas such as robot control and computer animation. However, relatively few people have explored their potential in music.
In his PhD, Lucas Bravo has designed and adapted swarm systems specifically built for interaction with human performers. He calls these systems "Human-Swarm Interactive Music Systems" (HS-IMSs), and they build on earlier research in both swarm intelligence and computer music.
He explains that the systems can provide a "live" experience, a co-creation between humans and AI, in music improvisation.
"The swarm produces its own music, and the user tries to use the swarm's interactive capabilities to shape a different intention than the autonomous behavior. That is why, in this interactive swarm, there is a trade-off between control and autonomy, since the user might not have full control over everything happening," he says.
Using his hands to shape the sound
Photo: Pedro Pablo Lucas Bravo
To study how agents behave, Lucas Bravo works in what's known as "mixed reality". He wears an XR headset ("extended reality") that lets him see both the physical room he's in and the virtual agents.
The agents appear as floating building blocks arranged in different formations around him. He can use his hands to move these blocks closer together, push them apart, or rearrange them in space.
From the outside, it looks as though he's just gesturing randomly in the air. But in fact, he's improvising music. By manipulating one agent, he explores how the others respond. How does the overall sound change? Which musical textures emerge? How much control does he have over the swarm?
If one drum fails, the music goes on
"The agents know which group they belong to, and they know they must move if one of the others is moved. But we can't predict exactly what the result will be," says Lucas Bravo.
One key advantage of swarm systems is their robustness. If one agent - for example, a virtual drum - fails or is removed, the others continue to do their job. The overall system keeps functioning.
For now, he uses these systems for musical improvisation. Over time, however, the goal is that they can also be used as tools for exploring musical ideas and composing new pieces, and perhaps even enabling new forms of live performance.
"The potential of these systems lies in balancing their chaotic tendencies with good control mechanisms," Lucas Bravo explains. "The agents have a behavior of their own, but if we can find a sweet spot where humans have just the right amount of control over them, we can create exciting musical experiences."
