Skip to main content
SpringerLink
Log in

Triangulation of multiple meaningful change thresholds for patient-reported outcome scores

  • Review
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

The notion of what constitutes meaningful differences or changes in patient-reported outcome scores is represented by meaningful change thresholds (MCTs). Applying multiple methods to estimate MCTs inevitably results in a range of estimates; however, a single estimate or small range is sought in practice to enable consistent interpretation of scores. While current recommendations for triangulation are appropriate in principle, the vital step of moving from all estimates to a value or small range lacks clarity and is subjective in nature. This article aims to review current triangulation approaches and provide more robust recommendations than what is currently available.

Methods

Current approaches to perform triangulation are described and discussed. Anchor-based estimates are focussed upon due to their recognition as the most valid and developed approach. Recommendations for triangulation are provided.

Results

A correlation-weighted average of MCT estimates is recommended to triangulate multiple MCT estimates derived from a single study into a single value, where increased weighting is given to stronger anchor measures. The choice of method to triangulate estimates from several published studies is highly dependent on the availability of information within the publications. MCTs designed for between-group differences, within-group changes, and within-individual changes should be considered separately.

Conclusion

The recommendations within this article provide a reliable and transparent approach to triangulation when a single value is sought, based on meta-analytic approaches. This approach is preferable to a simple mean of estimates where all are weighted equally, or through ‘eyeballing’ plotted estimates which is unreliable. We encourage researchers to adopt these methods, but to remain aware of the limitations within each method and further nuances in study design that result in heterogeneity. Sensitivity analyses with a range of plausible values are encouraged; however, the recommendations provide a suitable starting value for inferences. Unresolved issues in triangulation, requiring further exploration, are highlighted.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. King, M. T. (2011). A point of minimal important difference (MID): A critique of terminology and methods. Expert Review of Pharmacoeconomics & Outcomes Research, 11(2), 171–184.

    Article  Google Scholar 

  2. Bell, M. L., Dhillon, H. M., Bray, V. J., & Vardy, J. L. (2018). Important differences and meaningful changes for the Functional Assessment of Cancer Therapy-Cognitive Function (FACT-Cog). Journal of Patient-Reported Outcomes, 2(1), 48. https://doi.org/10.1186/s41687-018-0071-4

    Article  PubMed Central  Google Scholar 

  3. King, M. T., Dueck, A. C., & Revicki, D. A. (2019). Can methods developed for interpreting group-level patient-reported outcome data be applied to individual patient management? Medical Care, 57(Suppl 5 1), S38–S45. https://doi.org/10.1097/mlr.0000000000001111

    Article  PubMed  PubMed Central  Google Scholar 

  4. Cocks, K., King, M. T., Velikova, G., de Castro, G., Jr., Martyn St-James, M., Fayers, P. M., et al. (2012). Evidence-based guidelines for interpreting change scores for the European Organisation for the Research and Treatment of Cancer Quality of Life Questionnaire Core 30. European Journal of Cancer, 48(11), 1713–1721. https://doi.org/10.1016/j.ejca.2012.02.059

    Article  CAS  PubMed  Google Scholar 

  5. Musoro, J. Z., Bottomley, A., Coens, C., Eggermont, A. M., King, M. T., Cocks, K., et al. (2018). Interpreting European Organisation for Research and Treatment for Cancer Quality of life Questionnaire core 30 scores as minimally importantly different for patients with malignant melanoma. European Journal of Cancer, 104, 169–181. https://doi.org/10.1016/j.ejca.2018.09.005

    Article  PubMed  Google Scholar 

  6. Musoro, J. Z., Coens, C., Fiteni, F., Katarzyna, P., Cardoso, F., Russell, N. S., et al. (2019). Minimally important differences for interpreting EORTC QLQ-C30 scores in patients with advanced breast cancer. JNCI Cancer Spectrum, 3(3), pkz037. https://doi.org/10.1093/jncics/pkz037

    Article  PubMed  PubMed Central  Google Scholar 

  7. United States Food and Drug Administration. (2009). Patient-reported outcome measures: Use in medical product development to support labeling claims. Guidance for Industry. https://www.fda.gov/media/77832/download. Accessed 26 July 2021.

  8. Globe, G., Wiklund, I., Mattera, M., Zhang, H., & Revicki, D. A. (2019). Evaluating minimal important differences and responder definitions for the asthma symptom diary in patients with moderate to severe asthma. Journal of Patient-Reported Outcomes, 3(1), 22. https://doi.org/10.1186/s41687-019-0109-2

    Article  PubMed  PubMed Central  Google Scholar 

  9. Coon, C. D., & Cappelleri, J. C. (2016). Interpreting change in scores on patient-reported outcome instruments. Therapeutic Innovation & Regulatory Science, 50(1), 22–29.

    Article  Google Scholar 

  10. Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11.

    Article  Google Scholar 

  11. Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), 102–109.

    Article  PubMed  Google Scholar 

  12. Staunton, H., Willgoss, T., Nelsen, L., Burbridge, C., Sully, K., Rofail, D., et al. (2019). An overview of using qualitative techniques to explore and define estimates of clinically important change on clinical outcome assessments. Journal of Patient-Reported Outcomes, 3(1), 1–10.

    Article  Google Scholar 

  13. Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33–41. https://doi.org/10.1111/j.1745-3992.1991.tb00195.x

    Article  Google Scholar 

  14. Lord, F. M., Novick, M. R., & Birnbaum, A. (1968). Statistical theories of mental test scores. Addison-Wesley.

    Google Scholar 

  15. Hays, R. D., Farivar, S. S., & Liu, H. (2005). Approaches and recommendations for estimating minimally important differences for health-related quality of life measures. COPD, 2(1), 63–67. https://doi.org/10.1081/copd-200050663

    Article  PubMed  Google Scholar 

  16. Terwee, C. B., Roorda, L. D., Dekker, J., Bierma-Zeinstra, S. M., Peat, G., Jordan, K. P., et al. (2010). Mind the MIC: Large variation among populations and methods. Journal of Clinical Epidemiology, 63(5), 524–534.

    Article  PubMed  Google Scholar 

  17. Gerlinger, C., & Schmelter, T. (2011). Determining the non-inferiority margin for patient reported outcomes. Pharmaceutical Statistics, 10(5), 410–413. https://doi.org/10.1002/pst.507

    Article  PubMed  Google Scholar 

  18. de Vet, H. C. W., Ostelo, R. W. J. G., Terwee, C. B., van der Roer, N., Knol, D. L., Beckerman, H., et al. (2007). Minimally important change determined by a visual method integrating an anchor-based and a distribution-based approach. Quality of Life Research, 16(1), 131–142. https://doi.org/10.1007/s11136-006-9109-9

    Article  PubMed  Google Scholar 

  19. Hays, R. D., & Woolley, J. M. (2000). The concept of clinically meaningful difference in health-related quality-of-life research. How meaningful is it? PharmacoEconomics, 18(5), 419–423. https://doi.org/10.2165/00019053-200018050-00001

    Article  CAS  PubMed  Google Scholar 

  20. Santanello, N. C., Zhang, J., Seidenberg, B., Reiss, T. F., & Barber, B. L. (1999). What are minimal important changes for asthma measures in a clinical trial? European Respiratory Journal, 14(1), 23–27. https://doi.org/10.1034/j.1399-3003.1999.14a06.x

    Article  CAS  PubMed  Google Scholar 

  21. Wang, Y.-C., Hart, D. L., Stratford, P. W., & Mioduski, J. E. (2011). Baseline dependency of minimal clinically important improvement. Physical Therapy, 91(5), 675–688. https://doi.org/10.2522/ptj.20100229

    Article  PubMed  Google Scholar 

  22. Halme, A. S., Fritel, X., Benedetti, A., Eng, K., & Tannenbaum, C. (2015). Implications of the minimal clinically important difference for health-related quality-of-life outcomes: A comparison of sample size requirements for an incontinence treatment trial. Value in Health, 18(2), 292–298. https://doi.org/10.1016/j.jval.2014.11.004

    Article  PubMed  Google Scholar 

  23. European Medicines Agency. (2016). Guideline on multiplicity issues in clinical trials—draft. https://www.ema.europa.eu/en/documents/scientific-guideline/draft-guideline-multiplicity-issues-clinical-trials_en.pdf. Accessed 26 July 2021.

  24. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. (1998). ICH Topic E9 Statistical Principles for Clinical Trials. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-9-statistical-principles-clinical-trials-step-5_en.pdf. Accessed 26 July 2021.

  25. Schwind, J., Learman, K., O’Halloran, B., Showalter, C., & Cook, C. (2013). Different minimally important clinical difference (MCID) scores lead to different clinical prediction rules for the Oswestry disability index for the same sample of patients. Journal of Manual & Manipulative Therapy, 21(2), 71–78. https://doi.org/10.1179/2042618613Y.0000000028

    Article  Google Scholar 

  26. Sloan, J. A., Cella, D., & Hays, R. D. (2005). Clinical significance of patient-reported questionnaire data: Another step toward consensus. Journal of Clinical Epidemiology, 58(12), 1217–1219. https://doi.org/10.1016/j.jclinepi.2005.07.009

    Article  PubMed  Google Scholar 

  27. Wyrwich, K. W., Metz, S. M., Kroenke, K., Tierney, W. M., Babu, A. N., & Wolinsky, F. D. (2007). Triangulating patient and clinician perspectives on clinically important differences in health-related quality of life among patients with heart disease. Health Services research, 42(6p1), 2257–2274.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Myles, P. S., Myles, D. B., Galagher, W., Chew, C., MacDonald, N., & Dennis, A. (2016). Minimal clinically important difference for three quality of recovery scales. Anesthesiology, 125(1), 39–45. https://doi.org/10.1097/aln.0000000000001158

    Article  PubMed  Google Scholar 

  29. Ousmen, A., Touraine, C., Deliu, N., Cottone, F., Bonnetain, F., Efficace, F., et al. (2018). Distribution- and anchor-based methods to determine the minimally important difference on patient-reported outcome questionnaires in oncology: A structured review. Health and Quality of Life Outcomes, 16(1), 228. https://doi.org/10.1186/s12955-018-1055-z

    Article  PubMed  PubMed Central  Google Scholar 

  30. Griffiths, P., Williams, A., Brohan, E., & Cocks, K. (2019). Understanding the role of anchor correlations in the calculation of meaningful change thresholds for health-related quality of life research. Value in Health, 22, S826. https://doi.org/10.1016/j.jval.2019.09.2266

    Article  Google Scholar 

  31. Devji, T., Carrasco-Labra, A., Qasim, A., Phillips, M., Johnston, B. C., Devasenapathy, N., et al. (2020). Evaluating the credibility of anchor based estimates of minimal important differences for patient reported outcomes: Instrument development and reliability study. BMJ, 369, m1714. https://doi.org/10.1136/bmj.m1714

    Article  PubMed  PubMed Central  Google Scholar 

  32. Harper, A., Trennery, C., Sully, K., & Trigg, A. (2018). Triangulating estimates of meaningful change or difference in patient-reported outcomes: Application of a correlation-based weighting procedure. Quality of Life Research, 27, S17.

    Google Scholar 

  33. Sully, K., Trigg, A., Bonner, N., Moreno-Koehler, A., Trennery, C., Shah, N., et al. (2019). Estimation of minimally important differences and responder definitions for EORTC QLQ-MY20 scores in multiple myeloma patients. European Journal of Haematology, 103(5), 500–509.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Gatz, D. F., & Smith, L. (1995). The standard error of a weighted mean concentration—I. Bootstrapping vs other methods. Atmospheric Environment, 29(11), 1185–1193. https://doi.org/10.1016/1352-2310(94)00210-C

    Article  CAS  Google Scholar 

  35. Schisterman, E. F., & Perkins, N. (2007). Confidence intervals for the youden index and corresponding optimal cut-point. Communications in Statistics Simulation and Computation, 36(3), 549–563. https://doi.org/10.1080/03610910701212181

    Article  Google Scholar 

  36. Fisher, R. A. (1934). Statistical methods for research workers (5th ed.). Springer.

    Google Scholar 

  37. Jacobs, P., & Viechtbauer, W. (2017). Estimation of the biserial correlation and its sampling variance for use in meta-analysis. Research Synthesis Methods, 8(2), 161–180.

    Article  PubMed  Google Scholar 

  38. Myers, L., & Sirois, M. J. (2004). Spearman correlation coefficients, differences between. Encyclopedia of Statistical Sciences. https://doi.org/10.1002/0471667196.ess5050.pub2

    Article  Google Scholar 

  39. Pustejovsky, J. E. (2014). Converting from d to r to z when the design uses extreme groups, dichotomization, or experimental control. Psychological Methods, 19(1), 92–112. https://doi.org/10.1037/a0033788

    Article  PubMed  Google Scholar 

  40. Lee, A. C., Driban, J. B., Price, L. L., Harvey, W. F., Rodday, A. M., & Wang, C. (2017). Responsiveness and minimally important differences for 4 patient-reported outcomes measurement information system short forms: Physical function, pain interference, depression, and anxiety in knee osteoarthritis. The Journal of Pain, 18(9), 1096–1110. https://doi.org/10.1016/j.jpain.2017.05.001

    Article  PubMed  PubMed Central  Google Scholar 

  41. Devji, T., Guyatt, G. H., Lytvyn, L., Brignardello-Petersen, R., Foroutan, F., Sadeghirad, B., et al. (2017). Application of minimal important differences in degenerative knee disease outcomes: A systematic review and case study to inform BMJ Rapid Recommendations. British Medical Journal Open, 7(5), e015587. https://doi.org/10.1136/bmjopen-2016-015587

    Article  Google Scholar 

  42. Hao, Q., Devji, T., Zeraatkar, D., Wang, Y., Qasim, A., Siemieniuk, R. A. C., et al. (2019). Minimal important differences for improvement in shoulder condition patient-reported outcomes: A systematic review to inform a BMJ Rapid Recommendation. British Medical Journal Open, 9(2), e028777. https://doi.org/10.1136/bmjopen-2018-028777

    Article  Google Scholar 

  43. Johnston, B. C., Ebrahim, S., Carrasco-Labra, A., Furukawa, T. A., Patrick, D. L., Crawford, M. W., et al. (2015). Minimally important difference estimates and methods: A protocol. British Medical Journal Open, 5(10), e007953. https://doi.org/10.1136/bmjopen-2015-007953

    Article  Google Scholar 

  44. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley.

    Book  Google Scholar 

  45. Olsen, M. F., Bjerre, E., Hansen, M. D., Tendal, B., Hilden, J., & Hróbjartsson, A. (2018). Minimum clinically important differences in chronic pain vary considerably by baseline pain and methodological factors: Systematic review of empirical studies. Journal of Clinical Epidemiology, 101, 87–106. https://doi.org/10.1016/j.jclinepi.2018.05.007

    Article  PubMed  Google Scholar 

  46. Olsen, M. F., Bjerre, E., Hansen, M. D., Hilden, J., Landler, N. E., Tendal, B., et al. (2017). Pain relief that matters to patients: Systematic review of empirical studies assessing the minimum clinically important difference in acute pain. BMC Medicine, 15(1), 35. https://doi.org/10.1186/s12916-016-0775-3

    Article  PubMed  PubMed Central  Google Scholar 

  47. Ebrahim, S., Vercammen, K., Sivanand, A., Guyatt, G. H., Carrasco-Labra, A., Fernandes, R. M., et al. (2017). Minimally important differences in patient or proxy-reported outcome studies relevant to children: A systematic review. Pediatrics, 139(3), e20160833. https://doi.org/10.1542/peds.2016-0833

    Article  PubMed  Google Scholar 

  48. Chung, J. K., Kannappan, P. L., Ng, C. T., & Sahoo, P. K. (1989). Measures of distance between probability distributions. Journal of Mathematical Analysis and Applications, 138(1), 280–292. https://doi.org/10.1016/0022-247X(89)90335-1

    Article  Google Scholar 

  49. Cocks, K., & Buchanan, J. (2015). Defining responders on the European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire (30-item core module)(QLQ-C30) subscales. Quality of Life Research, 24, 125.

    Google Scholar 

  50. Molenaar, P. C. M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement Interdisciplinary Research and Perspectives, 2(4), 201–218. https://doi.org/10.1207/s15366359mea0204_1

    Article  Google Scholar 

  51. Vanier, A., Sébille, V., Blanchin, M., & Hardouin, J.-B. (2021). The minimal perceived change: A formal model of the responder definition according to the patient’s meaning of change for patient-reported outcome data analysis and interpretation. BMC Medical Research Methodology, 21(1), 128. https://doi.org/10.1186/s12874-021-01307-9

    Article  PubMed  PubMed Central  Google Scholar 

  52. Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018). Lack of group-to-individual generalizability is a threat to human subjects research. Proceedings of the National Academy of Sciences USA, 115(27), E6106. https://doi.org/10.1073/pnas.1711978115

    Article  CAS  Google Scholar 

  53. McLeod, L. D., Coon, C. D., Martin, S. A., Fehnel, S. E., & Hays, R. D. (2011). Interpreting patient-reported outcome results: US FDA guidance and emerging methods. Expert Review of Pharmacoeconomics & Outcomes Research, 11(2), 163–169. https://doi.org/10.1586/erp.11.12

    Article  Google Scholar 

  54. Nixon, A., Doll, H., Kerr, C., Burge, R., & Naegeli, A. N. (2016). Interpreting change from patient reported outcome (PRO) endpoints: Patient global ratings of concept versus patient global ratings of change, a case study among osteoporosis patients. Health and Quality of Life Outcomes, 14(1), 25. https://doi.org/10.1186/s12955-016-0427-5

    Article  PubMed  PubMed Central  Google Scholar 

  55. Demidenko, E. (2007). Sample size determination for logistic regression revisited. Statistics in Medicine, 26(18), 3385–3397. https://doi.org/10.1002/sim.2771

    Article  PubMed  Google Scholar 

  56. Avery, K. N. L., Richards, H. S., Portal, A., Reed, T., Harding, R., Carter, R., et al. (2019). Developing a real-time electronic symptom monitoring system for patients after discharge following cancer-related surgery. BMC Cancer, 19(1), 463. https://doi.org/10.1186/s12885-019-5657-6

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Tahira Devji for running searches using the PROMID database (www.promid.org) and reviewing an earlier draft of this manuscript. The authors also thank Kim Cocks for reviewing the draft manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Adelphi Values, Adelphi Mill, Bollington, Cheshire, SK10 5JB, UK

    Andrew Trigg

  2. IQVIA, Reading, Berkshire, UK

    Pip Griffiths

Authors
  1. Andrew Trigg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pip Griffiths
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AT: Study conception, review of articles, development of recommendations and manuscript writing. PG: Review of articles, development of recommendations and manuscript writing.

Corresponding author

Correspondence to Andrew Trigg.

Ethics declarations

Conflict of interest

AT is an employee of Adelphi Values, a health outcomes research company which provides consultancy services to numerous pharmaceutical companies. PG is an employee of IQVIA, a health outcomes research company which provides consultancy services to numerous pharmaceutical companies. PG was an employee of Adelphi Values during the initial development of this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 232 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Trigg, A., Griffiths, P. Triangulation of multiple meaningful change thresholds for patient-reported outcome scores. Qual Life Res 30, 2755–2764 (2021). https://doi.org/10.1007/s11136-021-02957-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-021-02957-4

Keywords

Search

Navigation