Accuracy versus Transparency in Pharmacoeconomic Modelling

Abstract

As modellers push to make their models more accurate, the ability of others to understand the models can decrease, causing the models to lose transparency. When this type of conflict between accuracy and transparency occurs, the question arises, “Where do we want to operate on that spectrum?” This paper argues that in such cases we should give absolute priority to accuracy: push for whatever degree of accuracy is needed to answer the question being asked, try to maximise transparency within that constraint, and find other ways to replace what we wanted to get from transparency. There are several reasons.

The fundamental purpose of a model is to help us get the right answer to a question and, by any measure, the expected value of a model is proportional to its accuracy. Ironically, we use transparency as a way to judge accuracy. But transparency is not a very powerful or useful way to do this. It rarely enables us to actually replicate the model’s results and, even if we could, replication would not tell us the model’s accuracy. Transparency rarely provides even face validity; from the content expert’s perspective, the simplifications that modellers have to make usually raise more questions than they answer. Transparency does enable modellers to alert users to weaknesses in their models, but that can be achieved simply by listing the model’s limitations and does not get us any closer to real accuracy. Sensitivity analysis tests the importance of uncertainty about the variables in a model, but does not tell us about the variables that were omitted or the structure of the model. What people really want to know is whether a model actually works. Transparency by itself can’t answer this; only demonstrations that the model accurately calculates or predicts real events can. Rigorous simulations of clinical trials are a good place to start. This is the type of empirical validation we need to provide if the potential of mathematical models in pharmacoeconomics is to be fully achieved.