Quality of Life Research

, Volume 27, Issue 11, pp 3003–3012 | Cite as

The reliability of end of day and ecological momentary assessments of pain and pain interference in individuals with spinal cord injury

  • Noelle E. CarlozziEmail author
  • Stephen Schilling
  • Jenna Freedman
  • Claire Z. Kalpakjian
  • Anna L. Kratz



This study investigated the most efficient means of measuring pain intensity and pain interference comparing ecological momentary assessment (EMA) to end of day (EOD) data, with the highest level of measurement reliability as examined in individuals with spinal cord injury.


EMA (five times throughout the day) and EOD ratings of pain and pain interference were collected over a 7-day period. Multilevel models were used to examine the reliability for both EOD and EMA assessments in order to determine the amount of variability in these assessments over the course of a week or the day, and a multilevel version of the Spearman–Brown Prophecy formula was used to estimate values for reliability.


Findings indicate the minimum of number of EOD and EMA assessments needed to achieve different levels of reliability (“adequate” > 0.70, “good” > 0.80 and excellent > 0.90). In addition, the time of day (either morning, midday or evening) did not impact the estimated reliability for the EMA assessments.


These findings can help researchers and clinician balance the cost/benefit tradeoffs of these different types of assessments by providing specific cutoffs for the numbers of each type of assessment that are needed to achieve excellent reliability.


Feasibility Spinal cord injury Ecological momentary assessment Daily diaries Pain Pain interference 



We thank Siera Goodnight, Kristen Pickup, Daniela Ristova-Trendov, Christopher Garbaccio, Jessica Mackelprang-Carter and Angela Garza for collecting and managing these data. A sincere thanks to all of our study participants for their effort in being in this study.


Research reported in this publication was supported by the Craig H. Neilsen Foundation under Award Number 287372 (PI: Kratz). The content is solely the responsibility of the authors and does not necessarily represent the views of the Craig H. Neilsen Foundation. Dr. Kratz was supported during manuscript preparation by a Grant from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (Award Number K01AR064275).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Broderick, J. E., Stone, A. A., Calvanese, P., Schwartz, J. E., & Turk, D. C. (2006). Recalled pain ratings: A complex and poorly defined task. The Journal of Pain, 7(2), 142–149.CrossRefGoogle Scholar
  2. 2.
    Schneider, S., Stone, A. A., Schwartz, J. E., & Broderick, J. E. (2011). Peak and end effects in patients’ daily recall of pain and fatigue: A within-subjects analysis. The Journal of Pain, 12(2), 228–235.CrossRefGoogle Scholar
  3. 3.
    Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32.CrossRefGoogle Scholar
  4. 4.
    Schneider, S., & Stone, A. A. (2016). Ambulatory and diary methods can facilitate the measurement of patient-reported outcomes. Quality of Life Research, 25(3), 497–506.CrossRefGoogle Scholar
  5. 5.
    Heapy, A., Dziura, J., Buta, E., Goulet, J., Kulas, J. F., & Kerns, R. D. (2014). Using multiple daily pain ratings to improve reliability and assay sensitivity: How many is enough? The Journal of Pain, 15(12), 1360–1365.CrossRefGoogle Scholar
  6. 6.
    Broderick, J. E., Schwartz, J. E., Schneider, S., & Stone, A. A. (2009). Can End-of-day reports replace momentary assessment of pain and fatigue? The Journal of Pain, 10(3), 274–281.CrossRefGoogle Scholar
  7. 7.
    Salaffi, F., Sarzi-Puttini, P., & Atzeni, F. (2015). How to measure chronic pain: New concepts. Best Practice & Research: Clinical Rheumatology, 29(1), 164–186.CrossRefGoogle Scholar
  8. 8.
    Perkins, D. O., Wyatt, R. J., & Bartko, J. J. (2000). Penny-wise and pound-foolish: The impact of measurement error on sample size requirements in clinical trials. Biological Psychiatry, 47(8), 762–766.CrossRefGoogle Scholar
  9. 9.
    Stone, A. A., & Broderick, J. E. (2007). Real-time data collection for pain: Appraisal and current status. Pain Medicine, 8(Suppl 3), S85–S93.CrossRefGoogle Scholar
  10. 10.
    Stone, A. A., Broderick, J. E., & Schwartz, J. E. (2010). Validity of average, minimum, and maximum end-of-day recall assessments of pain and fatigue. Contemporary Clinical Trials, 31(5), 483–490.CrossRefGoogle Scholar
  11. 11.
    Kratz, A. L., Kalpakjian, C. Z., & Hanks, R. A. (2017). Are intensive data collection methods in pain research feasible in those with physical disability? A study in persons with chronic pain and spinal cord injury. Quality of Life Research, 26(3), 587–600.CrossRefGoogle Scholar
  12. 12.
    Siddall, P. J., McClelland, J. M., Rutkowski, S. B., & Cousins, M. J. (2003). A longitudinal study of the prevalence and characteristics of pain in the first 5 years following spinal cord injury. Pain, 103(3), 249–257.CrossRefGoogle Scholar
  13. 13.
    Dijkers, M., Bryce, T., & Zanca, J. (2009). Prevalence of chronic pain after traumatic spinal cord injury: A systematic review. Journal of Rehabilitation Research & Development, 46(1), 13–29.CrossRefGoogle Scholar
  14. 14.
    Siddall, P. J., & Loeser, J. D. (2001). Pain following spinal cord injury. Spinal Cord, 39(2), 63–73.CrossRefGoogle Scholar
  15. 15.
    Cardenas, D. D., & Jensen, M. P. (2006). Treatments for chronic pain in persons with spinal cord injury: A survey study. The Journal of Spinal Cord Medicine, 29(2), 109–117.CrossRefGoogle Scholar
  16. 16.
    Budh, C. N., & Osteraker, A. L. (2007). Life satisfaction in individuals with a spinal cord injury and pain. Clinical Rehabilitation, 21(1), 89–96.CrossRefGoogle Scholar
  17. 17.
    Cruz-Almeida, Y., Alameda, G., & Widerstrom-Noga, E. G. (2009). Differentiation between pain-related interference and interference caused by the functional impairments of spinal cord injury. Spinal Cord, 47(5), 390–395.CrossRefGoogle Scholar
  18. 18.
    Ataoglu, E., Tiftik, T., Kara, M., Tunc, H., Ersoz, M., & Akkus, S. (2013). Effects of chronic pain on quality of life and depression in patients with spinal cord injury. Spinal Cord, 51(1), 23–26.CrossRefGoogle Scholar
  19. 19.
    Finnerup, N. B. (2013). Pain in patients with spinal cord injury. Pain, 154(Suppl 1), S71–S76.CrossRefGoogle Scholar
  20. 20.
    Mehta, S., McIntyre, A., Janzen, S., Loh, E., & Teasell, R. (2016). Spinal cord injury rehabilitation evidence T. Systematic review of pharmacologic treatments of pain after spinal cord injury: An update. Archives of Physical Medicine and Rehabilitation, 97(8), 1381–1391 e1381.CrossRefGoogle Scholar
  21. 21.
    Kalpakjian, C. Z., Farrell, D. J., Albright, K. J., Chiodo, A., & Young, E. A. (2009). Association of daily stressors and salivary cortisol in spinal cord injury. Rehabilitation Psychology, 54(3), 288–298.CrossRefGoogle Scholar
  22. 22.
    Kratz, A. L., Ehde, D. M., Bombardier, C. H., Kalpakjian, C. Z., & Hanks, R. A. (2017). Pain acceptance decouples the momentary associations between pain, pain interference, and physical activity in the daily lives of people with chronic pain and spinal cord injury. The Journal of Pain, 18(3), 319–331.CrossRefGoogle Scholar
  23. 23.
    Kirshblum, S. C., Burns, S. P., Biering-Sorensen, F., et al. (2011). International standards for neurological classification of spinal cord injury (revised 2011). The Journal of Spinal Cord Medicine, 34(6), 535–546.CrossRefGoogle Scholar
  24. 24.
    Kirshblum, S. C., Waring, W., Biering-Sorensen, F., et al. (2011). Reference for the 2011 revision of the International Standards for Neurological Classification of Spinal Cord Injury. The Journal of Spinal Cord Medicine, 34(6), 547–554.CrossRefGoogle Scholar
  25. 25.
    Riley, W. T., Rothrock, N., Bruce, B., et al. (2010). Patient-reported outcomes measurement information system (PROMIS) domain names and definitions revisions: Further evaluation of content validity in IRT-derived item banks. Quality of Life Research, 19(9), 1311–1321.CrossRefGoogle Scholar
  26. 26.
    Cella, D., Riley, W., Stone, A., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194.CrossRefGoogle Scholar
  27. 27.
    Czajkowski, S. M., Cella, D., Stone, A. A., Amtmann, D., & Keefe, F. (2010). Patient-Reported Outcomes Measurement Information System (Promis): Using new theory and technology to improve measurement of patient-reported outcomes in clinical research. Annals of Behavioral Medicine, 39, 46–46.Google Scholar
  28. 28.
    Cella, D., Rothrock, N., Choi, S., Lai, J. S., Yount, S., & Gershon, R. (2010). Promis overview: Development of new tools for measuring health-related quality of life and related outcomes in patients with chronic diseases. Annals of Behavioral Medicine, 39, 47–47.Google Scholar
  29. 29.
    Donders, J., Tulsky, D. S., & Zhu, J. (2001). Criterion validity of new WAIS-II subtest scores after traumatic brain injury. Journal of the International Neuropsychological Society, 7(7), 892–898.PubMedGoogle Scholar
  30. 30.
    Tulsky, D. S., Kisala, P. A., Victorson, D., et al. (2015). Methodology for the development and calibration of the SCI-QOL item banks. The Journal of Spinal Cord Medicine, 38(3), 270–287.CrossRefGoogle Scholar
  31. 31.
    Tulsky, D. S., Kisala, P. A., Victorson, D., et al. (2015). Overview of the Spinal Cord Injury–Quality of Life (SCI-QOL) measurement system. The Journal of Spinal Cord Medicine, 38(3), 257–269.CrossRefGoogle Scholar
  32. 32.
    Tulsky, D. S., & Kisala, P. A. (2015). The Spinal Cord Injury–Quality of Life (SCI-QOL) measurement system: Development, psychometrics, and item bank calibration. The Journal of Spinal Cord Medicine, 38(3), 251–256.CrossRefGoogle Scholar
  33. 33.
    Cohen, M. L., Kisala, P. A., Dyson-Hudson, T. A., & Tulsky, D. S. (2017). Measuring pain phenomena after spinal cord injury: Development and psychometric properties of the SCI-QOL Pain Interference and Pain Behavior assessment tools. The Journal of Spinal Cord Medicine, 41(3), 267–280.CrossRefGoogle Scholar
  34. 34.
    Cleeland, C. S. (1989). Measurement of pain by subjective report. In C. R. Chapman & J. D. Loeser (Eds.), Advances in pain research and therapy (Vol. 12, pp. 391–403). New York: Raven Press.Google Scholar
  35. 35.
    Cleeland, C. S., & Ryan, K. M. (1994). Pain assessment: Global use of the Brief Pain Inventory. Annals of the Academy of Medicine (Singapore), 23(2), 129–138.Google Scholar
  36. 36.
    Osborne, T. L., Jensen, M. P., Ehde, D. M., Hanley, M. A., & Kraft, G. (2007). Psychosocial factors associated with pain intensity, pain-related interference, and psychological functioning in persons with multiple sclerosis and pain. Pain, 127(1–2), 52–62.CrossRefGoogle Scholar
  37. 37.
    Osborne, T. L., Raichle, K. A., Jensen, M. P., Ehde, D. M., & Kraft, G. (2006). The reliability and validity of pain interference measures in persons with multiple sclerosis. Journal of Pain and Symptom Management, 32(3), 217–229.CrossRefGoogle Scholar
  38. 38.
    Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177.CrossRefGoogle Scholar
  39. 39.
    Raudenbush, S. W., & Bryk, A. S. (1992). Hierarchical linear models in social and behavioral research: Applications and data analysis methods (2 edn.). ed. Thousand Oaks, CA: Sage Publications.Google Scholar
  40. 40.
    Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edn.). New York: Academic Press.Google Scholar
  41. 41.
    DeVellis, R. (2017). Scale development: Theory and applications (4th edn.). Los Angeles: Sage.Google Scholar
  42. 42.
    Rasbash, J., Steele, F., Browne, W. J., Goldstein, H. (2012). Users Guide to MLwiN, v2.26. Bristol: University of Bristol: Centre for Multilevel Modelling.Google Scholar
  43. 43.
    Nunnally, J., & Bernstein, I. (1994). Psychometric theory. New York: McGraw-Hill.Google Scholar
  44. 44.
    Felix, E. R., Cruz-Almeida, Y., & Widerstrom-Noga, E. G. (2007). Chronic pain after spinal cord injury: What characteristics make some pains more disturbing than others? Journal of Rehabilitation Research & Development, 44(5), 703–715.CrossRefGoogle Scholar
  45. 45.
    Stone, A., Shiffman, S., Atienza, A., & Nebling, L. (Eds.). (2007). The science of real-time data capture: Self-reports in health research. New York: Oxford University.Google Scholar
  46. 46.
    Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2003). Patient compliance with paper and electronic diaries. Controlled Clinical Trials, 24(2), 182–199.CrossRefGoogle Scholar
  47. 47.
    Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002). Patient non-compliance with paper diaries. Brit Med J., 324(7347), 1193–1194.CrossRefGoogle Scholar
  48. 48.
    Stone, A. A., Broderick, J. E., Schwartz, J. E., Shiffman, S., Litcher-Kelly, L., & Calvanese, P. (2003). Intensive momentary reporting of pain with an electronic diary: Reactivity, compliance, and patient satisfaction. Pain, 104(1–2), 343–351.CrossRefGoogle Scholar
  49. 49.
    Kratz, A. L., Davis, M. C., & Zautra, A. J. (2012). Attachment predicts daily catastrophizing and social coping in women with pain. Health Psychology, 31(3), 278–285.CrossRefGoogle Scholar
  50. 50.
    Kratz, A. L., Ansari, S., Duda, M., et al. (2017). Feasibility of the E4 Wristband to Assess Sleep in Adults with and without Fibromyalgia. 36th Annual Scientific Meeting of the American Pain Society; Pittsburgh, PAGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Noelle E. Carlozzi
    • 1
    • 3
    Email author
  • Stephen Schilling
    • 2
  • Jenna Freedman
    • 1
  • Claire Z. Kalpakjian
    • 1
  • Anna L. Kratz
    • 1
  1. 1.Department of Physical Medicine & RehabilitationUniversity of MichiganAnn ArborUSA
  2. 2.Institute for Social ResearchUniversity of MichiganAnn ArborUSA
  3. 3.Department of Physical Medicine & RehabilitationUniversity of MichiganAnn ArborUSA

Personalised recommendations