Assessing and communicating data quality in policy-relevant research

Abstract

The quality of scientific information in policy-relevant fields of research is difficult to assess, and quality control in these important areas is correspondingly difficult to maintain. Frequently there are insufficient high-quality measurements for the presentation of the statistical uncertainty in the numerical estimates that are crucial to policy decisions. We propose and develop a grading system for numerical estimates that can deal with the full range of data quality—from statistically valid estimates to informed guesses. By analyzing the underlying quality of numerical estimates, summarized as spread and grade, we are able to provide simple rules whereby input data can be coded for quality, and these codings carried through arithmetical calculations for assessing the quality of model results. For this we use the NUSAP (numeral, unit, spread, assessment, pedigree) notational system. It allows the more quantitative and the more qualitative aspects of data uncertainty to be managed separately. By way of example, we apply the system to an ecosystem valuation study that used several different models and data of widely varying quality to arrive at a single estimate of the economic value of wetlands. The NUSAP approach illustrates the major sources of uncertainty in this study and can guide new research aimed at the improvement of the quality of outputs and the efficiency of the procedures.