Empirical Investigation of Ranking vs Best–Worst Scaling Generated Preferences for Attributes of Quality of Life: One and the Same or Differentiable?
The objective of this study was to investigate the degree of inconsistency in quality-of-life attribute preference orderings generated via successive best–worst scaling (a form of ranking whereby the respondent chooses the best and worst attributes from a choice set, these attributes are then eliminated and the best and worst attributes from the reduced choice set are then chosen and this process is continued until all presented attributes are eliminated) and conventional ranking methods (whereby the respondent chooses the best, second best and third best from a choice set until all presented attributes are eliminated).
An on-line survey was developed for administration to two general population samples comprising younger people (aged 18–64 years) and older people (aged 65 years and above). Data were analysed in STATA through an empirical examination of the relative level of choice inconsistency (randomness in responses or the variability in choice outcomes not explained by attributes and their associated preference weights) for successive best–worst in comparison with the conventional ranking method for the younger and older person samples.
For the younger person sample, both methods were found to be similarly consistent. In contrast, for the older person sample, ranking performed relatively worse than best–worst scaling with more inconsistent responses (tau = 0.515, p < 0.01).
These findings lend some support to the hypothesis initially propagated by the developers of best–worst scaling that it is a comparatively easier choice task for respondents to undertake than a traditional ranking task.
The authors thank Dr. Nicolas Krucien, Dr. Yuanyuan Gu and Associate Prof. Gang Chen for their helpful comments on a previous version of this paper.
JR, BK and EL conceived the study. All authors designed the study. JR and EL oversaw the data analysis. BK and CH performed the data analysis. JR led and BK, EL and CH contributed to drafting the article. All authors read and approved the final article.
Compliance with Ethical Standards
This study was funded in part by an Australian Research Council Linkage Grant (LP110200079).
Conflict of interest
Julie Ratcliffe, Billingsley Kaambwa, Claire Hutchinson and Emily Lancsar have no conflicts of interest that are directly relevant to the content of this article.
This study was approved by the Flinders University Social and Behavioural Research Ethics Committee (Project No.: 6682).
- 2.Brazier J, Ratcliffe J, Salomon J, Tsuchiya A. Measuring and valuing health benefits for economic evaluation. 2nd ed. Oxford: Oxford University Press; 2017.Google Scholar
- 22.Netten A, Burge P, Malley J, et al. Outcomes of social care for adults: developing a preference-weighted measure. Health Technol Assess. 2001;16:1–165.Google Scholar
- 23.Pink B. Information paper: an introduction to Socio-Economic Indexes for Areas (SEIFA). Cat no. 2039.0. Canberra (ACT): Australian Bureau of Statistics; 2006.Google Scholar
- 24.Pink B. Socio-Economic Indexes for Areas (SEIFA)-Technical Paper. Cat no. 2039.0.55.001. Canberra (ACT: Australian Bureau of Statistics; 2006A.Google Scholar
- 30.Greene W. Econometric analysis. 7th ed. Upper Saddle River: Prentice Hall; 2007.Google Scholar
- 33.Australian Bureau of Statistics. Household use of information technology, Australia, 2016–17. Canberra (ACT): Australian Bureau of Statistics; 2018.Google Scholar