In the IntraSurge project, researchers from TU/e are working together with physician-researchers from UMC Utrecht on a 'Google Maps' for the surgeon. By adding real-time information to the images of the surgical robot, new surgeons can be trained faster in robot-assisted surgery; they know where they are in the body, and they can see which steps remain in the operation. This is still only possible in research, not reality, but the first results are very promising.
In patients with esophageal cancer, the stomach and part of the esophagus sometimes have to be removed. Nowadays, this is increasingly done with robot-assisted surgery. The surgeon can work more accurately, and it has been shown that patients recover faster and experience fewer complications afterwards.
UMC Utrecht surgeon Jelle Ruurda is an expert in this procedure. He and his team treat many patients in the Netherlands and also receive more complex operation referrals from colleagues.Additionally, Ruurda trains surgeons to perform the procedure with the Da Vinci surgical robot.
But there is room for improvement. The training requires a long time and a lot of practice, even for experienced regular surgeons. That is why he set up the IntraSurge research project together with researchers from TU/e.
The goal is to create an automated navigation & orientation system for surgeons during surgery. You could call it a sort of Google Maps for the surgeon to help guide their steps during the procedure.
Augmented Reality
Postdoc researcher Yasmina Al Khalil has been working on the IntraSurge project in the Department of Biomedical Engineering since 2023, together with PhD students Yiping Li and Ronald de Jong . She talks enthusiastically about their work. "I always wanted to use my knowledge as an electrical engineer to improve healthcare. When I was alerted to this project, it was really a dream come true."
It is helpful to know how the surgeon performs such a robot-assisted operation. He is not standing at the operating table but is sitting a few meters apart, looking at a 3D screen and holding the controls. He controls the robot's surgical tools with his hands and feet.
"We are working on real-time augmented reality for surgeons. The first step is to clearly indicate in the image during the operations where all the important anatomical regions, tissues, and organs are. Think of the pericardium, lungs, large veins and arteries, and nerve bundles, for instance," Al Khalil explains.
Augmented reality adds an additional layer of information to the images the surgeon sees. That is the first step for the route planner: recognizing streets, houses, and cycle paths.
"The first examples show that we are already quite successful in 2D. However, we are far from satisfied yet, and we think that there is still much room for improvement. But the surgeons in training, whom we involve in our research, are already very enthusiastic about our work."
From 2D to 3D
The next step will be to add this information to the 3D images that the surgeons are working with. "During the operation, the surgeons need a 3D image to estimate the depth, which is also what the Da Vinci operation robot offers. We are already working on ways to provide those images with additional information seamlessly," says Al Khalil.
"We also want to add information that the surgeon can't see. With our augmented reality, we can show the parts of the body that are invisible, such as deeper layers of tissue or organs hidden from view by fatty tissue or blood. We need a CT scan of the patient in advance, so that we know their anatomy. By incorporating information about organs or arteries that are hidden from view, we can provide valuable additional insights."
In addition, the augmented information helps the surgeon to know exactly where he is during the operation. Anyone who has ever played a 3D game knows how quickly you lose your sense of up and down.
"When your image tells you where you are and in which direction you are looking, it makes a huge difference. This makes these types of robot-assisted surgical techniques not only faster to learn but also less tiring for the surgeons, especially with long, complex, and risky procedures," says Al Khalil.
Adding information
In the final step of the project, the researchers intend to add information about the procedure itself. Every surgical procedure consists of several steps, such as entering the body, removing an organ or tumor, neatly suturing, and closing the skin. Of course, this will be done with a much greater level of detail and medical knowledge.
Al Khalil: "Just as it can be nice for a driver to turn on the sat nav on a known route, it can be equally valuable for experienced surgeons to have this type of information while they work. You know where you are, can anticipate the next turn, and know how much of the journey remains."
"That's why Jelle (Ruurda, ed.) likens it to Google Maps. It will provide the same reassurance when you are on the road, but also provide tips and information you need to arrive safely and on time. This will make long operations less exhausting and minimize the risk of mistakes due to fatigue."
Early results
While Al Khalil has been working on the project for almost two years now, PhD researcher Yiping Li has only been working on it for just over a year. Nevertheless, she has already published an article and received first prize at the last SPIE conference on medical imaging .
Ronald de Jong recently started as a PhD student, but in a way, has been working on the project longer than the others, since he was already working on IntraSurge during his master's thesis. Al Khalil is very enthusiastic about their team and the collaboration with UMC Utrecht, as well as within TU/e.
"We get a lot of support from everyone, and that's very nice as young scientists. We also talk often to the surgeons who will be working with our system, and we learn from them how they do their jobs. Fortunately, there are also industrial partners in the consortium. In this way, we hope our work will not end up on a shelf but in the OR. Because that's why we are so passionate about our work."
