In a cramped, windowless room on the University of California, Berkeley campus, two bespoke microscopes - each a Swiss Army knife for high-resolution imaging - operate around the clock gathering data that will help train a game-changing technology for the field of biology: AI.
The identical microscopes, described this week in the journal Nature Methods, squeeze a dozen types of high-powered microscopes into a single machine, from standard phase contrast to the latest lattice light-sheet technology - easily switchable with the push of a button. Called MOSAIC (Multimodal Optical Scope with Adaptive Imaging Correction), it has already been recreated in more than a dozen labs worldwide thanks to preprints and elaborate assembly instructions disseminated over the past six years.
At UC Berkeley, it is one in a lineup of improved imaging technologies that could forever alter the field of biology, the researchers say. The microscopes can track over seconds, hours or days the development of live specimens, ranging from molecules and cells to entire embryos, gathering huge amounts of data that will allow biologists to track cells as they move through tissue, the evolution of internal cellular structures and even the shuttling of proteins and other molecules within the cell.
All this data - measured in petabytes, the equivalent of about 500 billion pages of text - requires the analytic ability of a large "vision" language model (LVLM), like ChatGPT. Building an LVLM or AI that can deal with petabytes of imaging data is now one of the main focuses of a team of microscopists, physicists, biologists and computer scientists in Berkeley's Advanced Bioimaging Center, which hopes to create a first-of-its-kind Cell Observatory.
"Life has to be studied in living tissue, holistically, and over fast timescales and for long periods of time," said Eric Betzig, a Berkeley professor of molecular and cell biology and of physics who won the 2014 Nobel Prize in Chemistry for the development of super-resolution fluorescence microscopy - a version of which is now incorporated into MOSAIC. "You can't study something as complex as a cell or organism just by looking at the parts individually - there are something like 40 million protein molecules alone of 20,000 different types. With our microscopes, we can image everything from single molecules to whole organisms at high resolution, following as many players as we can to understand natural physiological interactions in the cell."
Betzig, a Howard Hughes Medical Institute investigator, refers to the imaging data as five-dimensional, or 5D: three spatial dimensions, plus time and color. The color comes from fluorescent labels that allow scientists to track multiple subcellular structures simultaneously - organelles, membranes, the cytoskeleton and more - as they migrate, change shape, divide and interact over time.
"We are the world's best at collecting data at 5D, and have been for a decade," he said. "But we don't know how to interpret the data at scale; we can't think in petabytes and we don't see in 5D. That's why we're developing a 5D AI - it's a sherpa to guide us."
MOSAIC's development was led by Srigokul "Gokul" Upadhyayula, an assistant professor in residence in molecular and cell biology who had earlier worked with Betzig on other high-resolution techniques, the adaptive optical lattice light-sheet microscope, and the expansion lattice light-sheet microscope - both now part of MOSAIC.
"Biology is entering an era in which the data are too complex and too large to interpret by human inspection alone," he said. "A biologist may understand the biological question deeply, but still lack the computational tools and infrastructure needed to process, analyze and quantify what they are seeing. We need to build a mind that can reason natively with 3D movies of living biological systems and let us query those dynamics through language - akin to a ChatGPT for biology."
One remarkable video they captured shows a zebrafish regrowing its tail fin. Although the movie itself spans only 12 hours, it took months of preparation, processing and visualization before they could fully understand what it showed. The video revealed tiny events inside living tissue that are normally very hard to see: cells near the wound releasing small communication packets, microscopic fibers beneath the skin shifting as the tissue repaired itself, two repair cells fusing together and a red blood cell briefly getting trapped as new blood vessels were remodeled.
An AI assistant would not only help assemble these data-intensive movies, but help biologists home in on the specific activities they're interested in.
"There's so much information in these large movies, across scales, about how cells are behaving in the organism and the tissue and at the subcellular level, it can be difficult even for a very well-trained biologist to understand or digest," said Ian Swinburne, a Berkeley assistant professor of molecular and cell biology who works with Upadhyayula's team to study how cells engulf other cells, such as the macrophages that clean up dead cells in a wound. "AI can help us interface with the data and ask or answer questions more easily. Like, 'How many macrophages are crawling into my tissue during an infection?' or 'Can I predict when a cell's going to start leaving its organ?' That happens in development but also in cancer during metastasis."
"The impact of MOSAIC will be minimal until we build an AI model to be able to deal with the data that comes out of those systems; we basically have a gold mine, but we have no ability to get the gold out," Upadhyayula said. "The primary output of our Cell Observatory Initiative will be an AI mind that's able to be our scientific partner in extracting these observations."
A 'Swiss Army Knife' microscope
MOSAIC relies on several advances, Upadhyayula said: fluorescent molecules that allow biologists to mark specific cellular structures and molecular activities in living cells; fast, gentle light-sheet imaging that captures those dynamics with minimal stress or damage to the cells; high-speed data transfer and computing infrastructure capable of moving and processing massive imaging datasets; and new computational tools, including AI models, to help interpret the resulting 3D movies of living systems.
He and his colleagues in the Advanced Bioimaging Center combine these to create one-off high-resolution microscopes, such as the super-resolution microscope, which won Betzig the Nobel Prize. That innovation involved using a laser to stimulate fluorescent tags, which allowed researchers to image individual molecules in a cell and superimpose them into high-resolution images.
Betzig subsequently developed a faster but gentler technique for hi-res cell imaging, lattice light-sheet microscopy (LLSM), which reduces cellular damage by spreading the laser energy across a thin sheet to more gently illuminate a transparent specimen one thin slice at a time. The fluorescing markers are captured in real time and assembled into a 3D video.
MOSAIC combines these and other high-resolution imaging techniques into a single machine that can quickly transition from one imaging mode to another, repositioning many of the lenses that shape the light. To sharpen images, it uses adaptive optical elements, such as a deformable mirror controlled by 69 tiny motors that make minute adjustments to correct for blurring caused by aberrations in the living tissue itself.
Among the available modes are the latest versions of light-sheet and super-resolution microscopy, as well as multi-photon and label-free imaging. Across the various modes, MOSAIC is able to capture subcellular dynamics in cultured cells and live multicellular organisms, map nanoscale features across millimeter-scale expanded tissues and image the neural architecture in the brains of live mice.
Movies make a difference
The researchers emphasized the importance of video in understanding biological interactions, and the need to see them with sufficient fidelity.
"The name of the game is to keep the organism, the sample, as physiologically happy as possible," Upadhyayula said. "Which means using the lowest light dose we can to keep it from deep-frying while getting the information we need. The consequence is that the image gets noisy. When we watch a noisy movie over time, our minds naturally filter out some of the noise and focus on the underlying structures."
But getting AI to interpret 5D images is significantly harder than conventional image recognition.
"The current vision models are not built to reason over three dimensions, time and molecular identity or color, and that's what we want to build," he said.
Swinburne and Dave Matus, a researcher working with Betzig and Upadhyayula on the Cell Observatory Initiative, are now helping develop new labeling reagents that highlight subsets of the thousands of components for AI to recognize. While thoroughly impressed with MOSAIC's ability, Swinburne admits that the videos are so good that it's hard to focus on just one thing.
Fu, Liu, Milkie, Ruan et al, Nature Methods
"There's so much information in these movies," he said. "We come in with maybe a hypothesis about the process we think we're studying and then we get distracted by something we've never seen before. Probably every movie has something new that we acquire just because the quality is so high, the spatial and time resolution so much better than what we're used to."
The four first authors of the Nature Methods paper are Gaoxiang Liu and Xiongtao Ruan of UC Berkeley and Tian-Ming Fu and Daniel Milkie at the Janelia Research Campus of the Howard Hughes Medical Institute (HHMI) in Virginia. Upaphyayula, Betzig and Wesley Legant of the University of North Carolina at Chapel Hill are senior authors of the paper. The work was funded in part by HHMI, Philomathia Foundation, Biohub, the Sloan Foundation and Berkeley Lab. Betzig is a HHMI investigator and Upadhyayula is a Biohub San Francisco investigator.
