Leiden-based researcher Lincen Yang worked with doctors from the LUMC to develop an AI model that helps assess whether a patient can be safely discharged from an intensive care unit. Collaboration and transparency were the keys to success.
The use of artificial intelligence (AI) and algorithms is increasingly coming under scrutiny. After all, it can already do a great deal, and even more is promised. But what exactly lies under the hood of such an AI model? For some applications, that knowledge doesn't matter all that much. But doctors in the intensive care unit (ICU) have to make decisions that can ultimately be a matter of life and death. They want to know exactly how a risk assessment is constructed.
This is relevant, for example, in the case of a model that predicts how likely it is that a patient discharged from intensive care will need to be readmitted.
Making a transparent model
Computer scientists and medical professionals at Leiden University have developed a model that does what clinicians want: it provides an accurate risk assessment and makes it clear on the basis of which criteria it was calculated. Postdoctoral researcher Lincen Yang is one of the researchers. He saw an opportunity to test in practice a model he had already investigated in 2023 with Professor of Explainable Machine Learning Matthijs van Leeuwen.
'It wasn't so much a question of whether my model was more accurate than other models. The key question was: is the outcome logical and well-founded?'
In short, the model is based on unordered rules. An algorithm can learn a set of rules and use them to make predictions. Usually, these rules are assigned a specific order, so that one rule becomes a prerequisite for applying another. This makes the models difficult for end-users to interpret, as each decision depends on a previous one. The unordered rules are straightforward and therefore easier to understand.
Not replacing, but supporting doctors
'The problem for doctors in intensive care is that most models come in two variants. They are very good at identifying patients as high-risk, but without explaining why. Or they do provide an explanation, but according to the doctors, it is incorrect.' Yang suspected that his model could offer a solution. To do so, he had to take it from the safe lab environment into clinical practice.
Yang: 'It wasn't so much a question of whether my model was more accurate than other models. The key question was: is the outcome logical and well-founded?' Yang and his colleagues weren't looking for a model that would take over the doctors' work, but for one that would support them.
'Our algorithm has been trained using ten years' worth of data on ICU patients. It is impossible for a doctor to remember and assess all that data. But an algorithm does not see the individual patient. Thanks to their years of experience, doctors can base their judgement on what they see at the bedside too.'
Mutual understanding between disciplines
'At first, I thought: this is easy. My model, their data. Done. But interdisciplinary research also requires other things. Such as mutual understanding. The ability to work with each other's systems. And that takes time.' Yang noticed that everyone took that time. 'I think it's brilliant to see that researchers in Leiden share the same values. They work meticulously and carefully, because this is a critical field.'
'Transparency of algorithms is important not only for users, but also for the government and society.'
The model that emerged from the collaboration between LIACS and LUMC is proving to work. It provides ICU doctors with a risk assessment they can follow, enabling them to make their own decision on whether or not to discharge a patient from the ward.
'It's important that users understand what's happening. That every step can be traced back. In the past, factories had machines that only the technicians understood. They knew which buttons to press to change the output. In this model, I wanted to do away with those "magic parameters". Transparency of algorithms is important not only for users, but also for the government and society.'
