More than two decades after scientists first sequenced the entire human genome - all 3 billion "letters," or base pairs, of DNA code - the meaning of much of this code remains a mystery.
While an estimated 1 to 2% of human DNA codes for proteins, the rest is a mix of "junk DNA" - evolutionary holdovers that no longer code for anything - and regulatory elements that control when, where and how strongly genes are expressed. These non-coding regions of the genome could hold the key to understanding a variety of inherited traits, including those that lead to diseases such as cancer, heart disease and autism. But first, scientists have to understand how variants in this DNA contribute to the multitude of traits that make each of us unique.
Researchers at UC Berkeley have created a new genomic language AI model, called GPN-Star, that far outpaces its competitors at identifying the most important genetic variants that contribute to inherited traits, including those that lead to disease. It is also far more computationally efficient than larger models, requiring only a fraction of the time and computing resources to train.
"Our model excels in making predictions about the pathogenicity of genetic variants, and identifying functional versus non-functional elements in the genome," said study senior author Yun Song, a professor of computer science and statistics at Berkeley and an investigator at the Innovative Genomics Institute.
Along with the study, the researchers have published genome-wide predictions from their model, which highlight genetic variants that are likely to have the most influence on inherited traits. Biologists can use these annotations to identify relevant genes and regulatory elements for further study.
"We hope our work will help drive biological discovery," Song said. "People have developed really creative tools for assaying the impact of genetic variants, but they cannot experimentally test every single variant in the genome. We believe our predictions will help to prioritize the experiments that could have the greatest impact on human health."
Song is also director of the Berkeley Center for Computational Biology and co-director of the recently announced UC Berkeley-UCSF Bakar Computational Biomedicine Initiative. The study, funded in part by the National Institutes of Health, was published today (Sept. 9) in the journal Nature.
Glenn Ramit/UC Berkeley
The grammar of the genome
Genomic language models work a little like chatbots for DNA, but instead of being trained on natural language, they are trained on vast troves of DNA sequences. These models' advanced pattern recognition skills can identify repeating patterns and sequences much faster than any human, allowing them to identify important elements of a genome that might otherwise be impossible to recognize.
"Mathematically, a DNA sequence is just a string of letters - A, C, G and T. We don't know a priori which parts of the genome are functional elements, and a very small percentage of the genome is functional," Song said. "By training a DNA language model on a lot of different sequences, the model can recognize certain patterns that occur in the genomes. People have been using this to learn what we call the 'grammar' of the genome."
Most genomic language models, including the massive Evo 2 model published earlier this year, are trained on sequences from entire unaligned genomes, which can range in size from humans all the way down to single-celled organisms. However, this approach can be extremely computationally demanding. The Evo 2 model, which can generate entire genomes from scratch, was trained on the genomes of more than 100,000 species across all domains of life, and required 2,000 powerful NVIDIA computer processors and months to train.
To train the GPN-Star model, Song and his team used data from whole-genome alignments (WGAs) rather than individual unaligned genomes. WGAs use specialized algorithms to relate the genomes of hundreds of different species to that of a single species, highlighting similarities and differences in the code. For example, in a human-anchored WGA, the genomes of other species are compared to the human genome, revealing where the code has been conserved over the course of evolution and where it has changed.
Because WGAs do the work of identifying conserved areas of code, GPN-Star takes much less time and computing power to train than models that use unaligned genomes. It can be trained in just days, or even hours using only a handful of processors. It is also less likely to be confounded by the plethora of junk DNA that is found in the genomes of most species.
Our approach is that we should use these biological insights to improve the model, rather than hoping that the model will figure out what's important by itself.
Professor Yun Song
"We tried to help the model learn by curating data that's more likely to harbor functional elements," Song said. "Our approach is that we should use these biological insights to improve the model, rather than hoping that the model will figure out what's important by itself."
In the new study, the team trained the model on three different human-anchored WGAs, as well as WGAs for mice, fruit flies, chickens, C. elegans (roundworms) and A. thaliana (a type of plant). Each of the human-anchored WGAs included genomes from a different combination of other species, representing different evolutionary timescales: One included the genomes of other primates, one included the genomes of mammals, and the final included the genomes of other vertebrates.
"We found that models trained at different evolutionary time scales were actually optimized for interpreting different kinds of genetic variants," said study co-first author Chengzhong Ye, a graduate student in statistics at UC Berkeley. "This actually makes sense in terms of evolutionary biology, because some genomic elements evolve much faster than others."
For instance, they found that the model trained on data from longer evolutionary timescales was better at predicting the impact of rare genetic variants in proteins, which tend to evolve very slowly and are conserved across many species. However, the model trained on data from shorter evolutionary timescales was better at predicting the impact of genetic variants on complex traits like schizophrenia risk. These complex traits have been linked to as many as 10,000 different genetic mutations, many of which are in non-coding regions of the genome.
"For complex traits, we were surprised and pleased to see that training a model that's specific to primate genomes - which are more relevant to recent human evolution - really helped us make better predictions," Song said.
Because the model requires minimal resources to train, the researchers hope that it will be easy for other teams around the world to modify, adapt and improve upon their work, accelerating our understanding of genetics in humans and other species.
"We're making great progress," said study co-first author Gonzalo Benegas. "But the more people that can work with these models, the better they will get."
Additional co-authors of the study include Carlos Albors, Canal Li, Sebastian Prillo of Berkeley; Peter Fields of Jackson Laboratory; and Brian Clarke of the German Cancer Research Center in Heidelberg.
This research was supported in part by National Institutes of Health grants R35-GM134922, R35- GM161566 and 3P40-OD011102-24S1 7772, and by the UC National Laboratory Fees Research Program of the University of California Office of the President (UC AI Science at Scale Grant L26CR10102). The Chan Zuckerberg Initiative provided GPU resources (through the "Accelerating and Scaling Biological Sciences with AI" program) to generate genome-wide predictions from the GPN-Star models.
