Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics
The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) modelling taskforce suggests decision models should be thoroughly reported and transparent. However, the level of transparency and indeed how transparency should be assessed are yet to be defined. One way may be to attempt to replicate the model and its outputs. The ability to replicate a decision model could demonstrate adequate reporting transparency. This review aims to explore published definitions of replication success across all scientific disciplines and to consider how such a definition should be tailored for use in health economic models. A literature review was conducted to identify published definitions of a ‘successful replication’. Using these as a foundation, several definitions of replication success were constructed, to be applicable to replications of economic decision models, with the associated strengths and weaknesses of such definitions discussed. A substantial body of literature discussing replicability was found; however, relatively few studies, ten, explicitly defined a successful replication. These definitions varied from subjective assessments to expecting exactly the same results to be reproduced. Whilst the definitions that have been found may help to construct a definition specific to health economics, no definition was found that completely encompassed the unique requirements for decision models. Replication is widely discussed in other scientific disciplines; however, as of yet, there is no consensus on how replicable models should be within health economics or what constitutes a successful replication. Replication studies can demonstrate how transparently a model is reported, identify potential calculation errors and inform future reporting practices. It may therefore be a useful adjunct to other transparency or quality measures.
The authors would like to thank the delegates of the Bristol Health Economists’ Study Group (2018), where an earlier version of this paper was discussed.
Compliance with Ethical Standards
Conflict of interest
The authors, Emma McManus, David Turner, and Tracey Sach, have no conflicts of interest to declare.
Funding for this project was received from CLAHRC East of England (Project HE7: Dec 2015). This paper was written whilst also being funded by Professor Tracey Sach’s Career Development Fellowship (CDF award-2014-07-006) supported by the National Institute for Health Research. The views expressed in this paper are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care.
- 10.Duvendack M, Palmer-Jones RW, Reed WR. Replications in economics: a progress report. Econ J Watch. 2015;12(2):164–91.Google Scholar
- 12.Chang AC, Li P. Is economics research replicable? Sixty published papers from thirteen journals say ‘usually not’. Finance and Economics Discussion Series 2015-083, 2015.Google Scholar
- 17.International Initiative for Impact Evaluation. About 3ie. 2017; Available from: http://www.3ieimpact.org/en/about/.
- 18.Berkeley Initiative for Transparency in the Social Sciences. 2017; Available from: http://www.bitss.org/. Cited 2 Nov 2017.
- 19.The Replication Network. Furthering the practice of replication in economics. 2017; Available from: https://replicationnetwork.com/. Cited 2 Nov 2017.
- 20.Pashler H et al. PsychFileDrawer: archive of replication attempts in experimental psychology. Available from: http://www.psychfiledrawer.org/faq.php. Cited 29 April 2019.
- 33.Philips Z et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess 2004;8.Google Scholar
- 34.The Reward Alliance. The REWARD statement. 2016; Available from: http://researchwaste.net/reward-statement/. Cited 19 Apr 2016.
- 35.Equator Network. EQUATOR Network: what we do and how we are organised. 2016; Available from: http://www.equator-network.org/about-us/. Cited 19 Apr 2016.
- 40.Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international Conference on Evaluation and Assessment in Software Engineering. 2014. ACM.Google Scholar
- 41.Open Science Collaboration. Estimating the reproducibility of psychological science. Science 2015;349(6251):aac4716.Google Scholar
- 43.García FM. Do Kenyan teenagers respond to HIV risk information? A procedural replication of Dupas (2011). 2014.Google Scholar
- 44.Chang AC. A replication recipe: list your ingredients before you start cooking. 2017, Economics Discussion Papers.Google Scholar
- 45.Hardwicke TE, Mathur MB, Frank MC. Pre-registration: evaluation of the open data policy at Cognition. 2017. Available from: https://osf.io/q4qy3/. Cited 3 July 2018.
- 46.Höffler JH. ReplicationWiki - Improving transparency in the social sciences. 2018; http://replication.uni-goettingen.de/wiki/index.php/Main_Page. Cited 3 July 2018.
- 47.Brunner J, Schimmack U. How replicable is psychology? A comparison of four methods of estimating replicability on the basis of test statistics in original studies. 2016.Google Scholar
- 49.Cova F et al. Estimating the reproducibility of experimental philosophy. OpenCova, Florian et al.“Estimating the Reproducibility of Experimental Philosophy”. PsyArXiv, 2018. p. 21.Google Scholar
- 52.Palmer AJ et al. Computer modeling of diabetes and its transparency: a report on the Eighth Mount Hood Challenge. Value in Health, 2018.Google Scholar
- 53.McManus E et al. The barriers and facilitators to model replication within health economics. Value in Health, 2019.Google Scholar
- 55.McManus E, Sach TJP. Comment on “replicating health economic models: firm foundations or a house of cards?” 2017;35(11):1187–1188.Google Scholar
- 61.American Economic Association. Data availability policy. 2018; https://www.aeaweb.org/journals/policies/data-availability-policy. Cited 10 May 2018.
- 63.Sampson CJ. Call for a model registry. 2012; Available from: https://aheblog.com/2012/10/19/call-for-a-model-registry/. Cited 11 May 2018.