How does brain generate language, and what can a songbird teach us about it?
To understand how humans and other animals learn, organize, and produce complex vocalizations, neuroscientists have long studied canary song birds, who have incredibly detailed and structured songs that they can re-learn over time. For decades, scientists have manually decoded canary songs syllable by syllable, but Tim Gardner, an associate professor of bioengineering at the Knight Campus, believed a computer could do it better. New work from his lab, published in Patterns, introduces an AI tool called TweetyBERT that automatically parses birdsong, freeing researchers from hundreds of hours of tedious notation - and potentially offering a window into how the brain generates language.
What makes canaries especially valuable as a scientific model is that they learn complex courtship songs as juveniles, partly shaped by instinct and partly acquired by listening to and imitating other birds. They can then re-learn and reshape those songs as adults throughout the breeding season. That ongoing plasticity makes them a rare window into how a brain acquires and reorganizes complex learned behavior and language.
Canary songs are made of short individual syllables, each lasting a fraction of a second, that are grouped into longer repeated sequences and strung together into full songs drawn from 30 to 40 distinct syllables.
To analyze canary song, researchers must identify each syllable, mark when it occurs, and understand how it fits into the larger sequence around it. That means annotating recordings - listening, labeling, checking and re-checking - often down to the millisecond. This requires significant time and expertise from the researchers to ensure everything is captured correctly. Gardner had long suspected this was a problem that machine learning could assist with, giving researchers back valuable time, and standardizing the process.
George Vengrovski, a graduate student in the Gardner lab, began by looking at tools already being used to analyze human speech. He focused on a tool called BERT, which was one of the early breakthrough language models that helped launch the modern deep-learning era and influenced the broader wave of AI progress that followed. BERT works by learning the structure of language through a kind of fill-in-the-blank training. The tool was fed vast amounts of text with portions randomly hidden, and learned to predict what was missing. In doing so, it developed a rich sense of how words and phrases relate to one another.
Vengrovski thought this concept could translate to identifying syllables in birdsong. But canaries sing fast - significantly faster than humans speak - and with frequency patterns that don't map cleanly onto the human language models. So Vengrovski built TweetyBERT, a custom platform that could handle the acoustic structure of bird songs, now detailed in the journal Patterns.
To create TweetyBERT, Vengrovski first converted songs from three individual canaries into spectrograms, a visual representation of the birdsong audio recording that captures both pitch and loudness over time (shown above). He then trained a network on these spectrograms using an approach employed by the BERT platform known as masking. By hiding a portion of the spectrogram and asking the network to predict what should be there, the process trains the tool to learn the underlying structure through repetition.
After training, TweetyBERT gained the ability to automatically segment a new canary recording into syllables, labeling each distinct unit the way a human annotator would, but in a fraction of the time. For example, human annotation might take over a minute per song, while TweetyBERT processes many annotations per second. When scaled to the thousands of songs a single bird might produce, multiplied by the 10+ birds in a study, the time savings are significant.
When Vengrovski compared TweetyBERT's annotations to those produced by expert researchers, they were very similar in accuracy, suggesting the network was classifying syllables in a similar manner to a researcher in the lab.
Your browser does not support this video
Here, different colors represent unique individual syllables of the canary song, shown on a UMAP plot. These syllables are separated across the plot, indicating they are unique and identifiable, and the TweetyBERT model can group them as such.
"What's amazing is that we never explicitly tell TweetyBERT what a canary syllable is. Just from having TweetyBERT train on predicting spectrograms, it spontaneously developed its own internal representations of syllables and song structure. We were then able to harness those representations for practical tasks, like fully automated song labeling and tracking how songs change across seasons," says Vengrovski.
Going forward, Gardner's team is interested in understanding how TweetyBERT would perform in different, and perhaps messier, acoustic environments, with the goal of one day applying the tool to recordings outside of a lab.
Wild bird populations are under mounting pressure from human infrastructure and an increasingly noisy world, traffic, construction and other byproducts of urban sprawl can all alter bird vocalizations, sometimes in ways that disrupt communication and even mating.
The Gardner lab is already testing TweetyBERT as a passive acoustic monitoring tool, tracking vocal changes in wild populations of birds that might signal stress, displacement, or adaptation in real time.
Researchers are already applying tools like this to other vocal animals, including dolphins and sperm whales.
"We built this for canaries, in a controlled environment. But the underlying approach isn't specific to canaries," Gardner says. "The world is full of birds (and other animals) whose vocal behavior we're barely tracking. With some modifications that we are already doing, the applications of TweetyBERT starts to look much broader."
This work was supported by the National Institutes of Health (NIH).
