Should we use black box models in medicine?

The other day I had an interesting conversation with a researcher who had developed a complex deep learning model to extract medical information from videos, with the aim of deploying this model within a clinical context. Although the model seemed to work well, they were concerned that it amounted to a black box, and that this gave them little basis for trust in how (or even whether) it really worked.

Before I discuss the roles of such models in medicine, I’d first like to say that it’s refreshing to see such paranoia. As I keep saying in my posts, there’s a lot that can go wrong when developing ML models, and a lot of these wrong things can lead to models that appear to work well but in practice do not. So starting with a healthy sense of paranoia is a good thing, and I wish it was more common.

But the question here is: should we be using black boxes in medicine? Such questions have been plaguing machine learning ever since black boxes starting getting good at predicting, classifying and segmenting things. For instance, Cynthia Rudin wrote an influential article in Nature way back in 2019. Her view was that black box models should be avoided like the plague when it comes to high stakes decision making. At the time I pretty much agreed with her. But now we’re a few years on, deep learning has progressed immeasurably, and black boxes have got better and better at solving challenging problems. I’m no longer convinced that we can avoid them.

Something I still strongly agree with Cynthia about is that we shouldn’t be using explainable AI as a crutch. These techniques aim to explain the machinations of black box models, but in reality they don’t. So long as the explanation is much simpler than the model it’s explaining, then at best the insights it gives will be incomplete, and at worst they could be misleading. For example, an explainability analysis of a deep learning vision model may suggest it’s looking at sensible regions of a medical image, but who knows what it’s doing with the information within those regions — it could be something completely crazy.

Not all black boxes are the same. In particular, it’s important to consider the context within which a model is developed and used. Bad things tend to happen when the person developing the model didn’t talk to the person who’s going to use it. Much better things happen when the latter person — the clinician, or potentially even the patient, in this case — is involved with model development from the beginning. For example, collecting data that is correct and unbiased is instrumental to developing a model that people can have confidence in. It’s not the only thing of course, and you still have to look out for all the other modelling pitfalls that litter the path, but it’s a good place to start. In short, if we have confidence in our machine learning processes — both from a practitioner’s and end user’s perspective — then we’ll also gain more confidence in our models, even if we have no idea how they actually work.

The use case is also important. Using a black box model to replace a clinician still raises red flags. Using a black box model to supplement a clinician’s decision making, on the other hand, is generally more palatable. For instance, a black box model can be used for diagnostic support to highlight cases that a clinician should look at more closely. If it agrees with the clinician, then the model adds confidence to the clinician’s diagnosis; if it disagrees, then this should prompt the clinician to find out why. Explainable AI has a potentially useful role to play here too, pointing the clinician towards diagnostic features they might consider in their own ruminations.

Yet we’re increasingly entering a world where black box machine learning models can do things that humans are unable to do. Things like diagnosing complex conditions that medical researchers don’t yet understand. In this kind of situation, we have a choice of using the black box or doing nothing, and I’m guessing that most people would lean towards the former than the latter. Beyond the benefits of doing something rather than nothing, this also reflects a wider acceptance of black box models into out lives. For example, many people seem to have no problem trusting the outputs of models like ChatGPT, even if these outputs are sometimes wrong.

A concrete example of all this is the uptake of AI services for managing and treating mental health conditions, in the form of online counselling services that use fine-trained large language models. These are very much black boxes, even more so than earlier generations of deep learning, since their capabilities are not directly trained into the model, but built on top of an opaque model of natural language. In the UK, these services are actively being used by the National Health Service to supplement an insufficient supply of human mental health practitioners. So, for many patients, it is already a case of being treated by a black box or receiving no treatment.

So, it seems we’ve already reached a point where we have to accept that black box models have a role in medicine, and this role is only likely to grow. And my feeling is that the benefits probably outweigh the risks. But having said that, I still think it’s important that black box models are only used when necessary. ML practitioners often see deep learning as the go-to approach, but for many problems, it’s still possible to build interpretable models that have competitive performance. And for these kind of problems, it’s not unusual for deep learning models to perform worse than simpler and more interpretable models, particularly when data is scarce.

Before I go, it’s worth noting one more argument for using black boxes. This states that decisions made by complex black box models are akin to those made by trained clinicians — both are based on accumulated knowledge, and in both cases it may not be possible to determine the logic behind the decision. Loosely speaking, they’re both experts that follow their gut instincts. However, personally I don’t think machine learning has got anywhere near the point where it’s reasonable to equate the two. ML models still suffer from a host of well-known problems, including overfitting and poor robustness. Whilst humans may be imperfect, they are still a lot more capable than black box models. So, no need for clinicians to make their retirement plans just yet.