Skip to main content

Increasing respondent engagement in composite time trade-off tasks by imposing three minimum trade-offs to improve data quality

Abstract

Background

Web-based surveys are increasingly utilized for health valuation studies but may be more prone to lack of engagement and, therefore, poor data validity. The objective of this study was to evaluate the effect of imposed engagement (i.e., at least three trade-offs) in the composite time trade-off (cTTO) task.

Methods

The EQ-5D-5L valuation study protocol and study design were adapted for online, unsupervised completion in two arms: base case and engagement. Validity of preferences was assessed using the prevalence of inconsistent valuations and expected patterns of TTO values. Respondent task engagement was measured using time per task. Value sets were generated using linear regression with a random intercept (RILR).

Results

The base case (n = 501) and engagement arms (n = 504) clustered at different TTO values: [base case] 0, 1; [engagement] -0.5, 0.45, 0.6. Mean TTO values were lower for the engagement arm. Engagement respondents did not spend more time per TTO task: [base case] 63.3 s (SD 77.9 s); [engagement] 64.7 s (SD 73.3 s); p = 0.36. No significant difference was found between arms for prevalence of respondents with at least one inconsistent TTO value: [base case] 61.1%; [engagement] 63.5%; p = 0.43. Both value sets had significant intercepts far from 1: [base case] 0.846; [engagement] 0.783. The relative importance of the EQ-5D dimensions also differed between arms.

Conclusions

Both online arms had poor quality data. A minimum trade-off threshold did not improve engagement nor face validity of the data, indicating that modifications to the number of iterations are insufficient alone to improve data quality/validity of online TTO studies.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. 1.

    Whitehead, S.J., Ali, S.: Health outcomes in economic evaluation: the QALY and utilities. Br. Med. Bull. 96, 5–21 (2010). https://doi.org/10.1093/bmb/ldq033

    Article  PubMed  Google Scholar 

  2. 2.

    Oppe, M., Devlin, N.J., van Hout, B., Krabbe, P.F., de Charro, F.: A program of methodological research to arrive at the new international EQ-5D-5L valuation protocol. Val. Health. 17(4), 445–453 (2014). https://doi.org/10.1016/j.jval.2014.04.002

    Article  Google Scholar 

  3. 3.

    Boye, K.S., Matza, L.S., Feeny, D.H., Johnston, J.A., Bowman, L., Jordan, J.B.: Challenges to time trade-off utility assessment methods: when should you consider alternative approaches? Expert. Rev. Pharmacoecon. Outcomes. Res. 14(3), 437–450 (2014). https://doi.org/10.1586/14737167.2014.912562

    Article  PubMed  Google Scholar 

  4. 4.

    Xie, F., Pullenayegum, E., Pickard, A.S., Ramos Goni, J.M., Jo, M.W., Igarashi, A.: Transforming latent utilities to health utilities: east does not meet west. Health. Econ. 26(12), 1524–1533 (2017). https://doi.org/10.1002/hec.3444

    Article  PubMed  Google Scholar 

  5. 5.

    Hadi, M., Swinburn, P., Nalysnyk, L., Hamed, A., Mehta, A.: A health state utility valuation study to assess the impact of treatment mode of administration in Gaucher disease. Orphanet. J. Rare. Dis. 13(1), 159 (2018). https://doi.org/10.1186/s13023-018-0903-6

    Article  PubMed  PubMed Central  Google Scholar 

  6. 6.

    King, M.T., Viney, R., Simon Pickard, A., Rowen, D., Aaronson, N.K., Brazier, J.E., Cella, D., Costa, D.S.J., Fayers, P.M., Kemmler, G., McTaggart-Cowen, H., Mercieca-Bebber, R., Peacock, S., Street, D.J., Young, T.A., Norman, R., Consortium, M.A.: Australian utility weights for the EORTC QLU-C10D, a multi-attribute utility instrument derived from the cancer-specific quality of life questionnaire, EORTC QLQ-C30. Pharmacoeconomics. (2017). https://doi.org/10.1007/s40273-017-0582-5

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Norman, R., King, M.T., Clarke, D., Viney, R., Cronin, P., Street, D.: Does mode of administration matter? Comparison of online and face-to-face administration of a time trade-off task. Qual. Life. Res. 19(4), 499–508 (2010). https://doi.org/10.1007/s11136-010-9609-5

    Article  PubMed  Google Scholar 

  8. 8.

    Edelaar-Peeters, Y., Stiggelbout, A.M., Van Den Hout, W.B.: Qualitative and quantitative analysis of interviewer help answering the time tradeoff. Med. Decis. Making. 34(5), 655–665 (2014). https://doi.org/10.1177/0272989X14524989

    Article  PubMed  Google Scholar 

  9. 9.

    Maniaci, M.R., Rogge, R.D.: Caring about carelessness: participant inattention and its effects on research. J. Res. Pers. 48, 61–83 (2014). https://doi.org/10.1016/j.jrp.2013.09.008

    Article  Google Scholar 

  10. 10.

    Paas, L.J., Morren, M.: PLease do not answer if you are reading this: respondent attention in online panels. Mark. Lett. 29(1), 13–21 (2018). https://doi.org/10.1007/s11002-018-9448-7

    Article  Google Scholar 

  11. 11.

    Bansback, N., Tsuchiya, A., Brazier, J., Anis, A.: Canadian valuation of EQ-5D health states: preliminary value set and considerations for future valuation studies. PLoS ONE 7(2), e31115 (2012). https://doi.org/10.1371/journal.pone.0031115

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Jiang, R., Shaw, J.W., Pickard, A.S.: PS2-37 Comparison of online and face-to-face valuation of the EQ-5D-5L using composite time trade-off and discrete choice tasks. In: 40th Annual Meeting of the Society for Medical Decision Making, Montréal, Québec, Canada, vol. 39, pp. E1–E343. SAGE Publications Inc STM (2018). https://doi.org/10.1177/0272989X18821191

  13. 13.

    Jiang, R., Muhlbacher, A., Shaw, J.W., Lee, T.A., Walton, S.M., Pickard, A.S. Comparing online and face-to-face data quality and preferences in a health valuation study. In: 10th Meeting “International Academy of Health Preference Research”, Basel, Switzerland, vol. 12, pp. 429–435 (2019). https://doi.org/10.1007/s40271-019-00368-9

  14. 14.

    Oppe, M.: Personal Communication. In. (2018)

  15. 15.

    Lenert, L.A., Cher, D.J., Goldstein, M.K., Bergen, M.R., Garber, A.: The effect of search procedures on utility elicitations. Med. Decis. Making. 18(1), 76–83 (1998). https://doi.org/10.1177/0272989X9801800115

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    van Reenen, M., Janssen, B.: EQ-5D-5L user guide-basic information on how to use the EQ-5D-5 L instrument. EuroQol Group, Rotterdam (2013)

    Google Scholar 

  17. 17.

    Versteegh, M.M., Attema, A.E., Oppe, M., Devlin, N.J., Stolk, E.A.: Time to tweak the TTO: results from a comparison of alternative specifications of the TTO. Eur. J. Health. Econ. 14(Suppl 1), S43–51 (2013). https://doi.org/10.1007/s10198-013-0507-y

    Article  PubMed  Google Scholar 

  18. 18.

    Devlin, N., Buckingham, K., Shah, K., Tsuchiya, A., Tilling, C., Wilkinson, G., van Hout, B.: A comparison of alternative variants of the lead and lag time TTO. Health. Econ. 22(5), 517–532 (2013). https://doi.org/10.1002/hec.2819

    Article  PubMed  Google Scholar 

  19. 19.

    Pickard, A.S., Law, E.H., Jiang, R., Pullenayegum, E., Shaw, J.W., Xie, F., Oppe, M., Boye, K.S., Chapman, R.H., Gong, C.L., Balch, A., Busschbach, J.J.V.: United states valuation of EQ-5D-5L health states using an international protocol. Val. Health. (2019). https://doi.org/10.1016/j.jval.2019.02.009

    Article  Google Scholar 

  20. 20.

    Stolk, E., Ludwig, K., Rand, K., van Hout, B., Ramos-Goni, J.M.: Overview, update, and lessons learned from the international EQ-5D-5L valuation work: version 2 of the EQ-5D-5L valuation protocol. Val. Health. 22(1), 23–30 (2019). https://doi.org/10.1016/j.jval.2018.05.010

    Article  Google Scholar 

  21. 21.

    Oppe, M., Rand-Hendriksen, K., Shah, K., Ramos-Goni, J.M., Luo, N.: EuroQol protocols for time trade-off valuation of health outcomes. Pharmacoeconomics. 34(10), 993–1004 (2016). https://doi.org/10.1007/s40273-016-0404-1

    Article  PubMed  PubMed Central  Google Scholar 

  22. 22.

    Luo, N., Li, M., Stolk, E.A., Devlin, N.J.: The effects of lead time and visual aids in TTO valuation: a study of the EQ-VT framework. Eur. J. Health. Econ. 14(Suppl 1), S15–24 (2013). https://doi.org/10.1007/s10198-013-0504-1

    Article  PubMed  Google Scholar 

  23. 23.

    Oppe, M., van Hout, B.: The “power” of eliciting EQ-5D-5L values: the experimental design of the EQ-VT (2017). https://euroqol.org/wp-content/uploads/2016/10/EuroQol-Working-Paper-Series-Manuscript-17003-Mark-Oppe.pdf

  24. 24.

    Pew Research Center: Internet/Broadband Factsheet. https://www.pewinternet.org/fact-sheet/internet-broadband/ (2018). Accessed June 6 2019

  25. 25.

    Anderson, M., Perrin, A., Jiang, J., Kumar, M.: 10% of Americans don’t use the internet. Who are they? (2018). http://www.pewresearch.org/fact-tank/2018/03/05/some-americans-dont-use-the-internet-who-are-they/

  26. 26.

    Hanmer, J., Hays, R.D., Fryback, D.G.: Mode of administration is important in US national estimates of health-related quality of life. Med. Care. 45(12), 1171–1179 (2007). https://doi.org/10.1097/MLR.0b013e3181354828

    Article  PubMed  Google Scholar 

  27. 27.

    Duffy, B., Smith, K., Terhanian, G., Bremer, J.: Comparing data from online and face-to-face surveys. Int. J. Mark. Res. 47(6), 615–639 (2005). https://doi.org/10.1177/147078530504700602

    Article  Google Scholar 

  28. 28.

    Lenert, L.A., Sturley, A., Rupnow, M.: Toward improved methods for measurement of utility: automated repair of errors in elicitations. Med. Decis. Making. 23(1), 67–75 (2003). https://doi.org/10.1177/0272989X02239649

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Funding for data collection was supported by Bristol-Myers Squibb. R Jiang was supported by fellowships from the PhRMA Foundation and a Dean’s Scholar Award from the University of Illinois at Chicago during her PhD, during the time of study conduct and analysis.

Author information

Affiliations

Authors

Corresponding author

Correspondence to A. Simon Pickard.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables

Table 5 Table 4 with incremental dummies

5 and

Table 6 Dominated health state pairs within each TTO block

6 and Figs. 

Fig. 5
figure5

composite time trade-off task: conventional TTO for better-than-dead preference elicitation

5,

Fig. 6
figure6

Composite time trade-off task: lead-time TTO for worse-than-dead preference elicitation

6, and

Fig. 7
figure7

Planned TTO routing adapted from (Stolk 2019)

7.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jiang, R., Kohlmann, T., Lee, T.A. et al. Increasing respondent engagement in composite time trade-off tasks by imposing three minimum trade-offs to improve data quality. Eur J Health Econ 22, 17–33 (2021). https://doi.org/10.1007/s10198-020-01224-6

Download citation