Developing a scoring system to quality assess economic evaluations

Abstract

The number of economic evaluations is constantly increasing. The need to establish a framework with which to assess the validity of the studies has led to the development of checklists and scoring systems to compare the quality of different studies and ensure that decisions made by decision makers and researchers are based on solid evidence. The most prominent of these checklists is that produced by Drummond et al. Their checklist aims to answer two important questions: is the methodology employed in the study appropriate and are the results valid? In this paper three methods using the checklist of Drummond et al. are developed to assess the quality of a random sample of 50 papers selected from the NHS Economic Evaluation Database. Method 1, a direct application of the checklist calculates an average score with each of the ten items weighted equally, this method proves to be good at identifying low quality studies but is a rather a blunt tool for differentiating between high-quality studies. Method 2, also using an additive score, introduces a hierarchy of the effectiveness data and also adds a new item with respect to the transferability of the results. With this method the number of papers scoring top marks decreases dramatically in comparison to method 1. Method 3 involves a multiplicative rather than additive scoring system, which is better at distinguishing between good quality studies but appears to reduce discrimination between poor-quality studies. Overall, the ten-point checklist of Drummond et al. is a useful and quick tool with which to assess the quality of economic evaluations and to enable decision makers and researchers to focus upon the most relevant studies.