Skip to main content

Advertisement

SpringerLink
Log in

Modernizing quality of life assessment: development of a multidimensional computerized adaptive questionnaire for patients with schizophrenia

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Objective

Quality of life (QoL) is still assessed using paper-based and fixed-length questionnaires, which is one reason why QoL measurements have not been routinely implemented in clinical practice. Providing new QoL measures that combine computer technology with modern measurement theory may enhance their clinical use. The aim of this study was to develop a QoL multidimensional computerized adaptive test (MCAT), the SQoL-MCAT, from the fixed-length SQoL questionnaire for patients with schizophrenia.

Methods

In this multicentre cross-sectional study, we collected sociodemographic information, clinical characteristics (i.e., duration of illness, the PANSS, and the Calgary Depression Scale), and quality of life (i.e., SQoL). The development of the SQoL-CAT was divided into three stages: (1) multidimensional item response theory (MIRT) analysis, (2) multidimensional computerized adaptive test (MCAT) simulations with analyses of accuracy and precision, and (3) external validity.

Results

Five hundred and seventeen patients participated in this study. The MIRT analysis found that all items displayed good fit with the multidimensional graded response model, with satisfactory reliability for each dimension. The SQoL-MCAT was 39% shorter than the fixed-length SQoL questionnaire and had satisfactory accuracy (levels of correlation >0.9) and precision (standard error of measurement <0.55 and root mean square error <0.3). External validity was confirmed via correlations between the SQoL-MCAT dimension scores and symptomatology scores.

Conclusion

The SQoL-MCAT is the first computerized adaptive QoL questionnaire for patients with schizophrenia. Tailored for patient characteristics and significantly shorter than the paper-based version, the SQoL-MCAT may improve the feasibility of assessing QoL in clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Awad, A. G., Voruganti, L. N., & Heslegrave, R. J. (1997). Measuring quality of life in patients with schizophrenia. PharmacoEconomics, 11(1), 32–47.

    Article  CAS  PubMed  Google Scholar 

  2. Bobes, J., García-Portilla, P., Sáiz, P. A., Bascarán, T., & Bousoño, M. (oct 2005). Quality of life measures in schizophrenia. European Psychiatry, 20(Suppl 3), S313–S317.

    Article  PubMed  Google Scholar 

  3. Boyer, L., Baumstarck, K., Boucekine, M., Blanc, J., Lançon, C., & Auquier, P. (2013). Measuring quality of life in patients with schizophrenia: An overview. Expert Review of Pharmacoeconomics & Outcomes Research, 13(3), 343–349.

    Article  Google Scholar 

  4. Greenhalgh, J., Long, A. F., & Flynn, R. (1982). The use of patient reported outcome measures in routine clinical practice: Lack of impact or lack of theory? Social Science & Medicine, 60(4), 833–843.

    Article  Google Scholar 

  5. Morris, J., Perez, D., & McNoe, B. (1998). The use of quality of life data in clinical practice. Quality of Life Research, 7(1), 85–91.

    Article  CAS  PubMed  Google Scholar 

  6. Awad, A. G., & Voruganti, L. N. P. (2012). Measuring quality of life in patients with schizophrenia: An update. PharmacoEconomics, 30(3), 183–195.

    Article  PubMed  Google Scholar 

  7. Halyard, M. Y., Frost, M. H., Dueck, A., & Sloan, J. A. (2006). Is the use of QOL data really any different than other medical testing? Current Problems in Cancer, 30(6), 261–271.

    Article  PubMed  Google Scholar 

  8. Halyard, M. Y., Frost, M. H., & Dueck, A. (2006). Integrating QOL assessments for clinical and research purposes. Current Problems in Cancer, 30(6), 319–330.

    Article  PubMed  Google Scholar 

  9. Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., & Reeve, B., et al. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11.

    Article  PubMed Central  PubMed  Google Scholar 

  10. Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., & Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194.

    Article  PubMed Central  PubMed  Google Scholar 

  11. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists (1 ed). Mahwah, NJ: Psychology Press.

    Google Scholar 

  12. Fayers, P., & Machin, D. (2007). Quality of life: The assessment, analysis and interpretation of patient-reported outcomes (2e éd). New York: Wiley.

    Book  Google Scholar 

  13. Weiss. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37, 70–84.

    Article  Google Scholar 

  14. Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., & Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5 Suppl 1), S22–S31.

    Article  PubMed  Google Scholar 

  15. Hill, C. D., Edwards, M. C., Thissen, D., Langer, M. M., Wirth, R. J., & Burwinkle, T. M., et al. (2007). Practical issues in the application of item response theory: A demonstration using items from the pediatric quality of life inventory (PedsQL) 4.0 generic core scales. Medical Care, 45(5 Suppl 1), S39–S47.

    Article  PubMed  Google Scholar 

  16. Gardner, W., Kelleher, K. J., & Pajer, K. A. (2002). Multidimensional adaptive testing for mental health problems in primary care. Medical Care, 40(9), 812–823.

    Article  PubMed  Google Scholar 

  17. Petersen, M. A., Groenvold, M., Aaronson, N., Fayers, P., Sprangers, M., Bjorner, J. B. (2006). Multidimensional computerized adaptive testing of the EORTC QLQ-C30: Basic developments and evaluations. Quality Life Research, 15(3), 315–329.

    Article  Google Scholar 

  18. Haley, S. M., Ni, P., Dumas, H. M., Fragala-Pinkham, M. A., Hambleton, R. K., & Montpetit, K., et al. (2009) Measuring global physical health in children with cerebral palsy: Illustration of a multidimensional bi-factor model and computerized adaptive testing. Quality Life Research, 18(3), 359–370.

    Article  Google Scholar 

  19. Auquier, P., Simeoni, M. C., Sapin, C., Reine, G., Aghababian, V., & Cramer, J., et al. (2003). Development and validation of a patient-based health-related quality of life questionnaire in schizophrenia: The S-QoL. Schizophrenia Research, 63(1–2), 137–149.

    Article  CAS  PubMed  Google Scholar 

  20. Baumstarck, K., Boyer, L., Boucekine, M., Aghababian, V., Parola, N., & Lançon, C., et al. Self-reported quality of life measure is reliable and valid in adult patients suffering from schizophrenia with executive impairment. Schizophrenia Research, 147(1), 58–67.

  21. Baumstarck, K., Boucekine, M., Boyer, L., Aghababian, V., Parola, N., & Reuter, F., et al. (2014). Quantification of relevance of quality of life assessment for patients with cognitive impairment: The suitability indices. BMC Neurology, 14(1), 78.

    Article  PubMed Central  PubMed  Google Scholar 

  22. Auquier, P., Tinland, A., Fortanier, C., Loundou, A., Baumstarck, K., & Lancon, C., et al. (2013). Toward meeting the needs of homeless people with schizophrenia: The validity of quality of life measurement. PloS ONE, 8(10), e79677.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  23. Boyer, L., Lançon, C., Baumstarck, K., Parola, N., Berbis, J., & Auquier, P. (2013). Evaluating the impact of a quality of life assessment with feedback to clinicians in patients with schizophrenia: Randomised controlled trial. The British Journal of Psychiatry, 202, 447–453.

    Article  PubMed  Google Scholar 

  24. Catherine, F.-A., Boyer, L., Jonathan, W., Jean-Philippe, R., Raphaelle, R., Elisabeth, S., et al. (2015). Neural substrate of quality of life in patients with schizophrenia: A magnetisation transfer imaging study. Scientific Reports, 5, 17650.

    CAS  PubMed Central  PubMed  Google Scholar 

  25. Boyer, L., Richieri, R., Faget, C., Padovani, R., Vaillant, F., Mundler, O., et al. (2012). Functional involvement of superior temporal sulcus in quality of life of patients with schizophrenia. Psychiatry Research, 202(2), 155–160.

    Article  PubMed  Google Scholar 

  26. McKenna, S. P. (1997) Measuring quality of life in schizophrenia. European Psychiatry, 12(Suppl 3), 267s–274s.

    Article  PubMed  Google Scholar 

  27. Calman, K. C. (1984). Quality of life in cancer patients–an hypothesis. Journal of medical ethics, 10(3), 124–127.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  28. Caqueo-Urízar, A., Boyer, L., Boucekine, M., & Auquier, P. (2014). Spanish cross-cultural adaptation and psychometric properties of the Schizophrenia Quality of Life short-version questionnaire (SQoL18) in 3 middle-income countries: Bolivia, Chile and Peru. Schizophrenia Research, 159(1), 136–143.

    Article  PubMed  Google Scholar 

  29. Chou, C.-Y., Ma, M.-C., Yang, T.-T., Chen-Sea, M.-J. (2011). Psychometric validation of the S-QoL Chinese (Taiwan) version for patients with schizophrenia. Quality Life Research, 20(5), 763–767.

    Article  Google Scholar 

  30. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: American Psychiatric Association.

    Google Scholar 

  31. Loi n° 78–17 du 6 janvier. (1978). relative à l’informatique, aux fichiers et aux libertés.

  32. WMA Declaration of Helsinki - Ethical principles for medical research involving human subjects. (2013). Retrieved March 18, 2016, from http://www.wma.net/fr/30publications/10policies/b3/.

  33. Kay, S. R., Fiszbein, A., & Opler, L. A. (1987). The positive and negative syndrome scale (PANSS) for Schizophrenia. Schizophrenia Bulletin, 13(2), 261–276.

    Article  CAS  PubMed  Google Scholar 

  34. Lançon, C., Reine, G., Llorca, P. M., & Auquier, P. (1999). Validity and reliability of the French-language version of the Positive and Negative Syndrome Scale (PANSS). Acta Psychiatrica Scandinavica, 100(3), 237–243.

    Article  PubMed  Google Scholar 

  35. Addington, D, Addington, J, & Schissel, B. (1990) A depression rating scale for schizophrenics. Schizophrenia Research, 3(4), 247–251.

    Article  CAS  PubMed  Google Scholar 

  36. Michel, P., Baumstarck, K., Ghattas, B., Pelletier, J., Loundou, A., & Boucekine, M., et al. A multidimensional computerized adaptive short-form quality of life questionnaire developed and validated for multiple sclerosis. The MusiQoL-MCAT. Medicine (Baltimore). 2016.

  37. Fayers, P. M., & Hand, D. J.. (1997). Factor analysis, causal indicators and quality of life. Quality Life Research, 6(2), 139–150.

    Article  CAS  Google Scholar 

  38. Costa, D. S. J. (2015). Reflective, causal, and composite indicators of quality of life: A conceptual or an empirical distinction? Quality Life Research, 24(9), 2057–2065.

    Article  Google Scholar 

  39. Muthén, L., & Muthén, B. (2012). MPlus user’s guide (7th ed). Los Angeles: Muthén & Muthén.

    Google Scholar 

  40. Steiger, J. H. (2007). Understanding the limitations of global fit assessment in structural equation modeling. Personality and Individual Differences, 42(5), 893–898.

    Article  Google Scholar 

  41. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Iowa City: Psychometric Society.

    Google Scholar 

  42. Samejima, F. (1974). Normal ogive model on the continuous response level in the multidimensional latent space. Psychometrika, 39(1), 111–121.

    Article  Google Scholar 

  43. Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23.

    Article  CAS  Google Scholar 

  44. Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38(9 Suppl), II28-II42.

    PubMed Central  Google Scholar 

  45. Cai L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335.

    Article  Google Scholar 

  46. Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1–29.

    Article  Google Scholar 

  47. Kang, T., & Chen, T. T. (2008). Performance of the generalized S-X2 item fit index for polytomous IRT models. Journal of Educational Measurement, 45(4), 391–406.

    Article  Google Scholar 

  48. Raju, N. S., Price, L. R., Oshima, T. C., & Nering, M. L. (2007). Standardized conditional SEM: A case for conditional reliability. Applied Psychological Measurement, 31(3), 169–80.

    Article  Google Scholar 

  49. Wang, T, & Others, A. (1996). Conditional standard errors, reliability and decision consistency of performance levels using polytomous IRT. Retrieved February 18, 2015, from http://eric.ed.gov/?id=ED401323.

  50. Wright, B. D., & Linacre, J. M. (1994) Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

    Google Scholar 

  51. Molenaar, I. W. (1991). A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items. Kwant Methoden, 12(37), 97–117.

    Google Scholar 

  52. van der Ark, L. A. (2007). Mokken scale analysis in R. Journal of Statistical Software, 39(8), 1–30.

    Google Scholar 

  53. Stochl, J., Jones, P. B., & Croudace, T. J. (2012). Mokken scale analysis of mental health and well-being questionnaire item responses: A non-parametric IRT method in empirical research for applied health researchers. BMC Medical Research Methodology, 12, 74.

    Article  PubMed Central  PubMed  Google Scholar 

  54. Siu, C. O., Harvey, P. D., Agid, O., Waye, M., Brambilla, C., & Choi, W.-K., et al. (2015). Insight and subjective measures of quality of life in chronic schizophrenia. Schizophrenia Research: Cognition, 2(3), 127–132.

    Article  Google Scholar 

  55. Boyer, L., Cermolacce, M., Dassa, D., Fernandez, J., Boucekine, M., & Richieri, R., et al. (2012). Neurocognition, insight and medication nonadherence in schizophrenia: A structural equation modeling approach. PloS ONE, 7(10), e47655.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  56. Choi, S. W., Gibbons, L. E., & Crane, P. K. (2011). Lordif: An R Package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software, 39(8), 1–30.

    Article  PubMed Central  PubMed  Google Scholar 

  57. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and likert-type (ordinal) item scores. Ottawa, Directorate of Human Resources Research and Evaluation, Department of National Defense.

  58. Anatchkova, M. D., Saris-Baglama, R. N., Kosinski, M., & Bjorner, J. B. (2009). Development and preliminary testing of a computerized adaptive assessment of chronic pain. The Journal of Pain, 10(9), 932–943.

    Article  PubMed Central  PubMed  Google Scholar 

  59. Huisman, M., & Molenaar, I. W. (2001). Imputation of missing scale data with item response models. In: Boomsma, A., van Duijn, M. A. J. & Snijders, T. A. B. (Eds.) Essays on item response theory, pp. 221–244. New York: Springer. (Lecture Notes in Statistics). Retrieved September 20, 2013, from http://link.springer.com/chapter/10.1007/978-1-4613-0169-1_13.

  60. Peyre, H., Leplège, A., & Coste, J. (2010). Missing data methods for dealing with missing items in quality of life questionnaires. A comparison by simulation of personal mean score, full information maximum likelihood, multiple imputation, and hot deck techniques applied to the SF-36 in the French 2003 decennial health survey. Quality of Life Research, 20(2), 287–300.

    Article  PubMed  Google Scholar 

  61. Mulder, J, & van der Linden, W. J.. (2009). Multidimensional adaptive testing with Kullback–Leibler information item selection. In: van der Linden, W. J. & Glas, C. A. W. (Eds.) Elements of adaptive testing, pp. 77–101. New York: Springer. (Statistics for Social and Behavioral Sciences). Retrieved January 28, 2015, from http://link.springer.com/chapter/10.1007/978-0-387-85461-8_4.

  62. Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33–41.

    Article  Google Scholar 

  63. Babcock, B., & Weiss, D. J. (2013). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement? Journal of Computerized Adaptive Testing, 1, 1–18.

    Google Scholar 

  64. Choi, S. W., Reise, S. P., Pilkonis, P. A., Hays, R. D., & Cella, D. (2010). Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research, 19(1), 125–136.

    Article  PubMed  Google Scholar 

  65. R Core Team. (2016). R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/.

  66. Alessandrini, M., Lançon, C., Fond, G., Faget-Agius, C., Richieri, R., & Faugere, M., et al. (2016). A structural equation modelling approach to explore the determinants of quality of life in schizophrenia. Schizophrenia Research, 171(1–3), 27–34.

    Article  PubMed  Google Scholar 

  67. Ruggeri, M., Nosè, M., Bonetto, C., Cristofalo, D., Lasalvia, A., Salvi, G., et al. (2005). Changes and predictors of change in objective and subjective quality of life: Multiwave follow-up study in community psychiatric practice. The British Journal of Psychiatry, 187, 121–130.

    Article  PubMed  Google Scholar 

  68. Reine, G., Simeoni, M.-C., Auquier, P., Loundou, A., Aghababian, V., & Lancon, C. (2005). Assessing health-related quality of life in patients suffering from schizophrenia: A comparison of instruments. European Psychiatry, 20(7), 510–519.

    Article  PubMed  Google Scholar 

  69. Kemmler, G., Holzner, B., Neudorfer, C., Meise, U., & Hinterhuber, H. (1997). General life satisfaction and domain-specific quality of life in chronic schizophrenic patients. Quality of Life Research, 6(3), 265–273.

    Article  CAS  PubMed  Google Scholar 

  70. Folsom, D. P., Depp, C., Palmer, B. W., Mausbach, B. T., Golshan, S., & Fellows, I., et al. (2009). Physical and mental health-related quality of life among older people with schizophrenia. Schizophrenia Research, 108(1–3), 207–213.

    Article  PubMed Central  PubMed  Google Scholar 

  71. Smits, N. (2016). On the effect of adding clinical samples to validation studies of patient-reported outcome item banks: A simulation study. Quality of Life Research, 25(7), 1635–1644.

    Article  PubMed  Google Scholar 

  72. Yao, L. (2014). Multidimensional CAT item selection methods for domain scores and composite scores with item exposure control and content constraints. Journal of Educational Measurement, 51(1), 18–38.

    Article  Google Scholar 

  73. Petersen, M. A., Giesinger, J. M., Holzner, B., Arraras, J. I., Conroy, T., & Gamper, E.-M., et al. (2013). Psychometric evaluation of the EORTC computerized adaptive test (CAT) fatigue item pool. Quality of Life Research, 22(9), 2443–2454.

    Article  PubMed  Google Scholar 

  74. Zhang, J. (2012). Calibration of response data using MIRT models with simple and mixed structures. Applied Psychological Measurement, 36(5), 375–398.

    Article  Google Scholar 

  75. Baumstarck, K., Boyer, L., Boucekine, M., Aghababian, V., Parola, N., & Lançon, C., et al. (2013). Self-reported quality of life measure is reliable and valid in adult patients suffering from schizophrenia with executive impairment. Schizophrenia Research, 147(1), 58–67.

    Article  PubMed  Google Scholar 

  76. Girard, V., Tinland, A., Boucekine, M., Loubière, S., Lancon, C., & Boyer, L., et al. (2016). Validity of a common quality of life measurement in homeless individuals with bipolar disorder and schizophrenia. Journal of Affective Disorders, 204, 131–137.

    Article  CAS  PubMed  Google Scholar 

  77. Woods, C. M. (2014). Estimating the latent density in unidimensional IRT to permit non-normality. In Handbook of item response theory modeling: Applications to typical performance assessment. New York: Routledge.

    Google Scholar 

  78. Green, B. F., Bock, R. D., Humphreys, L. G., Linn, R. L., & Reckase, M. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21(4), 347–360.

    Article  Google Scholar 

  79. Nunnally, J. C, & Bernstein, I. H. (1994) Psychometric theory, p. 736 (3rd ed). New York: McGraw-Hill.

    Google Scholar 

  80. Carpenter, J. R., & Kenward, M. G. (2008). Missing data in randomised controlled trials—a practical guide. Birmingham: The National Institute for Health Research. RM03/JH17/MK.

    Google Scholar 

  81. Fielding, S., Fayers, P. M., & Ramsay, C. R. (2009). Investigating the missing data mechanism in quality of life outcomes: A comparison of approaches. Health and Quality of Life Outcomes, 7, 57.

    Article  PubMed Central  PubMed  Google Scholar 

  82. Nortvedt, M. W., Riise, T., Myhr, K. M., & Nyland, H. I. (2000). Quality of life as a predictor for change in disability in MS. Neurology, 55(1), 51–54.

    Article  CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Aix-Marseille University, EA 3279 – Public Health, Chronic Diseases and Quality of Life - Research Unit, 13005, Marseille, France

    Pierre Michel, Karine Baumstarck, Christophe Lancon, Badih Ghattas, Anderson Loundou, Pascal Auquier & Laurent Boyer

  2. Aix-Marseille University – I2M UMR 7373 – Mathematics Institute of Marseille, 13009, Marseille, France

    Pierre Michel & Badih Ghattas

Authors
  1. Pierre Michel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karine Baumstarck
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christophe Lancon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Badih Ghattas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anderson Loundou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pascal Auquier
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Laurent Boyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pierre Michel.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 14 KB)

Supplementary material 2 (DOCX 17 KB)

Appendix

Appendix

See Table 5.

Table 5 Differential item functioning according to gender, presence of paranoid symptoms, and insight
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Michel, P., Baumstarck, K., Lancon, C. et al. Modernizing quality of life assessment: development of a multidimensional computerized adaptive questionnaire for patients with schizophrenia. Qual Life Res 27, 1041–1054 (2018). https://doi.org/10.1007/s11136-017-1553-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-017-1553-1

Keywords

Search

Navigation