Key Takeaways
- A Berkeley Lab team built a large language model-powered assistant, called TEM Agent, that can operate a transmission electron microscope from natural-language instructions.
- The system uses an open standard called the Model Context Protocol (MCP) to safely limit the assistant to a fixed set of pre-approved actions.
- TEM Agent connects the microscope to Berkeley Lab's NERSC supercomputer and a custom-built data platform called Crucible, cutting experiment turnaround from hours to minutes and letting researchers instantly recall past experiment settings.
Some of the most important discoveries in materials science and structural biology come down to a simple question: where, exactly, are the atoms? The arrangement of atoms determines nearly everything about how a material behaves in the real world - how strong it is, how it conducts electricity, how a battery stores energy, or how a drug binds to a cell.
To answer that question, scientists use transmission electron microscopes. Researchers at the Molecular Foundry, a U.S. Department of Energy (DOE) Office of Science User Facility at Lawrence Berkeley National Laboratory (Berkeley Lab) operate several of these microscopes, including some of the most advanced in the world, capable of imaging structures roughly half the size of a hydrogen atom (or 60 billion times smaller than you). This gives outside researchers access to equipment they wouldn't otherwise have access to.
That access to complex equipment comes with a steep learning curve. As a User Facility, the Molecular Foundry hosts scientists from universities, companies, and other institutions who don't have access to the specialized, world-class electron microscopes at their home labs. Some visiting researchers arrive with a semester of coursework in electron microscopy basics, but still need weeks of hands-on training to run the lab's more advanced techniques. Even experienced users can find themselves managing an overwhelming amount of data: a single detector at the facility can generate roughly 700 gigabytes of data - the equivalent of several smartphones' worth of photos - in just 15 seconds, far too fast to process manually.
The result is a bottleneck. The equipment keeps getting more powerful, but the number of people who can expertly operate it and analyze it can't keep pace.
Teaching an LLM to speak microscope
To address that gap, a team at the Molecular Foundry developed a way to use a commercial Super Intelligence (SI) model, commonly referred to as artificial intelligence or LLM, to help run the microscope itself. But the microscope wasn't a blank slate: Ercius's team had already built automated routines to handle repetitive tasks like imaging large numbers of nanoparticles.
"The existing automations responded like a factory line: a bolt goes in, the robot arm swings, the car moves, another bolt goes in," said Peter Ercius, staff scientist at the Molecular Foundry's National Center for Electron Microscopy (NCEM) facility who led a study published in npj Computational Materials on this research. "It only knew how to do exactly what you programmed it to do. If something unexpected happens, the whole thing breaks. And if you're not a robotics engineer, you have no way to change what it does."
Rather than starting from scratch, the team took the automated functions they had already built and rewrote them in a Model Context Protocol (MCP) format - an open standard that gives an SI access to a fixed set of specific, pre-written actions, rather than open-ended control - opening that existing toolkit up to the SI.
"With TEM agent, I can just describe what I want in plain language, and it can string those steps together itself: even requests I never specifically programmed for," said Ercius. "That means a non-expert can now do things that used to require custom code for every scenario."
The team wasn't sure what to expect when they first gave the SI access to the microscope's controls. "We just gave it access and started asking it to do things that I would normally ask a user or myself to do on a microscope, and it just started working," said Ercius. "It was pretty amazing that it worked out of the box."
Once connected, researchers could describe an experiment in plain English, and the SI, nicknamed TEM Agent, would carry it out: adjusting magnification, tilting the sample, refocusing the beam, and stringing together the dozens of small steps a technique might require.
One of the most demanding techniques the team tested was electron tomography, which reconstructs a three-dimensional view of a sample's atomic structure from a series of tilted images. The process can take hours and, if any one step drifts off target, can throw off the entire reconstruction.
The first attempt didn't go smoothly. Tilting the sample and beam scanning are two different actions, and the TEM agent initially confused them. "It started rotating the scanning direction instead of rotating the sample," said Ercius. The team wrote a short explanation for the SI to quickly correct it. "We just had to teach it a little bit of information about how to run the experiment."
Once the team trusted TEM Agent with established techniques like tomography, they began testing how far it could go. Stephanie Ribet, a research scientist at the Molecular Foundry who specializes in electron microscopy, pushed the system into more experimental territory - including ptychography, a computational imaging technique that reconstructs high-resolution images from diffraction patterns rather than direct images, and that typically requires significant expertise to set up correctly.
"While it is easy to connect LLMs to literature and data analysis pipelines, integrating an LLM with scientific hardware can be more challenging," said Ribet. "The MCP platform provided a nice way to bridge modern machine learning methods with our existing automation toolkit and allows users to get started on advanced experiments that incorporate ptychography, tomography, and automation."
Expanding the scope to more scientists
Beyond the microscope itself, the team linked TEM Agent to two additional resources: the National Energy Research Scientific Computing Center (NERSC), a DOE supercomputing User Facility, and Crucible, a data platform started at the Molecular Foundry that organizes and stores experiment metadata. Together, these connections mean raw data can be processed in seconds instead of hours, and researchers can simply ask what settings they used on a sample weeks earlier instead of digging through files or lab notebooks by hand.
"All of the hard work of developing the scripts to talk to the microscopes and our databases had already been done, which made it a lot easier than we expected to connect those scripts to a large language model," said Morgan Wall, computer systems engineer and data scientist, who leads development of Crucible. "Instead of a user having to understand the semantics of five different scripts talking to five different computers, they can just use natural language to talk to one interface to control all of these systems."
The approach isn't limited to this one microscope, either. Because MCP and Crucible can work with any system that has an open interface, similar processes could extend to other instruments across the lab.
"It started as a Molecular Foundry data platform, but we've been working with several other nanoscale research centers to help them expand Crucible to their own use cases," said Wall. "Within the lab, we've also been working to make sure Crucible can integrate with other data platforms, like the Advanced Light Source's data catalog."
The goal isn't to replace expertise, but to extend it, giving both new and experienced researchers a faster path from idea to experiment.
"I think agentic assistants are going to democratize this kind of work - it's going to let us do more science, and do it better. It's already changing the world, and I'm excited to see how we can use it here to improve science," said Ercius.
This work was supported by the DOE Office of Science.
