Trials are essential to ruling out what doesn't work, as well as identifying what does
Last year, we ran a randomised controlled trial working with frontline workers in the homelessness sector, which aimed to increase their wellbeing. The project, which began at the What Works Centre for Wellbeing (before it closed down), was conducted in partnership with the Centre for Homelessness Impact, and found that an intervention which had previously been shown to work well at improving wellbeing and reducing burnout for 911 call operators in the US, did not have an impact in this context.
Last week, an academic article, criticising the use of randomised controlled trials in general, chose to take issue with our study in particular, arguing that it took a "methodology-first" approach to research. Did we select our intervention purely because it was amenable to a randomised trial, the paper asked?
The question is itself an interesting one. Do we choose what to do based on the methodology we want to use, rather than choosing the intervention that is most likely to succeed? Is the tail of methodology wagging the dog of intervention selection? This is not dissimilar to the argument made against nudging by two prominent behavioural scientists, Nick Chater and George Loewenstein, in a paper published in 2023.
The answer to this question is slightly complicated. First, because I am a trialist, I am attracted to the types of questions that are amenable to being answered with trials. In particular, I am very interested in the question "If I do this, what will happen to that", which is fundamentally a question of causal inference, which is a question that trials are well placed to try and answer (other methodologies are available). In similar vein, my colleagues who have pursued other methodological specialisms are often interested in the questions that can be answered by those methods. So, in that sense, yes, I was attracted by this intervention because it was amendable to trialing.
The story doesn't stop there, however. There are a huge number of interventions that are amenable to trialing, and which I have also suggested. Both in children's social care, and in housing, I have proposed trials that would test giving workers more pay, or reducing their caseload, in order to see what would happen. The reason these interventions weren't taken forward was not because they weren't amenable to trial (I think they are), but because they are expensive and hard.
In fact, if you're in favour of more radical solutions to workplace wellbeing interventions, I'd argue you should be pleased that we ran this trial. We now know that this doesn't work, and the case for more intensive interventions to support people has been made stronger. If you think this evidence didn't need producing, I'd encourage you to look at the constant proliferation of light-touch interventions in the real world.
This is where trials are useful - they sort the wheat from the chaff of lots of interventions. Earlier in my career, my former doctoral student Bibi Groot and I ran a large-scale trial with an independent evaluator to test an intervention which had shown promise in early pilots. The trial found no effect of the intervention on grades - it was a failure. Our qualitative research (the basis of Bibi's PhD) was overwhelmingly positive, and so too was the qualitative element of the trial - as it too often is. If we'd just listened to the qualitative research, we'd definitely have learned a lot, but perhaps we'd still be running an intervention that doesn't actually make a difference. By knowing that this isn't effective, we can move on - perhaps to bolder, more powerful interventions, as we've since written about.
There are times when a trial isn't appropriate - when you're at an early stage, or trying to answer a question other than "What happens to outcomes as a consequence of me doing this". Some reforms - and excellent ones at that - cannot be trialled. We shouldn't shy away from those just because we can't run an RCT, but we should still ask questions about effectiveness, and try to answer them - if only so that we can find things don't work and move on to something else.
