Skip to main content

The development of a clinical outcomes survey research application: Assessment CenterSM

Quality of Life Research volume 19, pages 677–685 (2010)Cite this article

Abstract

Introduction

The National Institutes of Health sponsored Patient-Reported Outcome Measurement Information System (PROMIS) aimed to create item banks and computerized adaptive tests (CATs) across multiple domains for individuals with a range of chronic diseases.

Purpose

Web-based software was created to enable a researcher to create study-specific Websites that could administer PROMIS CATs and other instruments to research participants or clinical samples. This paper outlines the process used to develop a user-friendly, free, Web-based resource (Assessment CenterSM) for storage, retrieval, organization, sharing, and administration of patient-reported outcomes (PRO) instruments.

Methods

Joint Application Design (JAD) sessions were conducted with representatives from numerous institutions in order to supply a general wish list of features. Use Cases were then written to ensure that end user expectations matched programmer specifications. Program development included daily programmer “scrum” sessions, weekly Usability Acceptability Testing (UAT) and continuous Quality Assurance (QA) activities pre- and post-release.

Results

Assessment Center includes features that promote instrument development including item histories, data management, and storage of statistical analysis results.

Conclusions

This case study of software development highlights the collection and incorporation of user input throughout the development process. Potential future applications of Assessment Center in clinical research are discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

Abbreviations

CAT:

Computerized adaptive test

CTQ:

Critical to quality

IRT:

Item response theory

JAD:

Joint application design

NIH:

National Institutes of Health

NINDS:

National Institutes of Neurological Disorders and Stroke

PRO:

Patient-reported outcomes

PROMIS:

Patient-reported outcome measurement information system

PRS:

Primary research sites

QA:

Quality assurance

SCC:

Statistical coordinating center

UAT:

Usability acceptability testing

References

  1. Zerhouni, E. (2003). Medicine. The NIH roadmap. Science, 302(5642), 63–72.

    Article  CAS  PubMed  Google Scholar 

  2. Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11.

    Article  PubMed  Google Scholar 

  3. DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical Care, 45(5 Suppl 1), S12–S21.

    Article  PubMed  Google Scholar 

  4. Cella, D., Riley, W., Stone, A. A., Rothrock, N., Reeve, B. B., & Yount, S., et al. (2009). Initial item banks and first wave testing of the patient reported outcomes measurement information system (PROMIS) network: 2005–2008. Journal of Clinical Epidemiology, In Press.

  5. Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141.

    Article  PubMed  Google Scholar 

  6. Lai, J. S., Cella, D., Chang, C. H., Bode, R. K., & Heinemann, A. W. (2003). Item banking to improve, shorten and computerize self-reported fatigue: An illustration of steps to create a core item bank from the FACIT-Fatigue Scale. Quality of Life Research, 12(5), 485–501.

    Article  PubMed  Google Scholar 

  7. Davis, K. M., Chang, C. H., Lai, J. S., & Cella, D. (2002). Feasibility and acceptability of computerized adaptive testing (CAT) for fatigue monitoring in clinical practice. Quality of Life Research, 11(7), 134.

    Google Scholar 

  8. Ware, J. E., Jr., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlof, C. G., et al. (2003). Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Quality of Life Research, 12(8), 935–952.

    Article  PubMed  Google Scholar 

  9. O’Carroll, P. W., O’Carroll, P. W., Yasnoff, W. A., Ward, M. E., Ripp, L. H., & Martin, E. L. (2003). Information architecture. In K. J. Hannah & M. J. Ball (Eds.), Public health informatics and information systems (pp. 85–97). New York: Springer.

    Chapter  Google Scholar 

  10. Davidson, E. J. (1999). Joint application design (JAD) in practice. Journal of Systems and Software, 45(3), 215–223.

    Article  Google Scholar 

  11. Schroeder, R. G., Linderman, K., Liedtke, C., & Choo, A. S. (2008). Six sigma: Definition and underlying theory. Journal of Operations Management, 26, 536–554.

    Article  Google Scholar 

  12. Lane, J. P., Usiak, D. J., Stone, V. I., & Scherer, M. J. (1997). The voice of the customer: Consumers define the ideal battery charger. Assistive Technology, 9(2), 130–139.

    CAS  PubMed  Google Scholar 

  13. Sharp, A., & McDermott, P. (2001). Workflow modeling: Tools for process improvement and application development. Boston: Artech House.

    Google Scholar 

  14. Reijers, H. A. (2003). Design and control of workflow processes: Business process management for the service industry. Berlin; New York: Springer.

    Book  Google Scholar 

  15. Bittner, K., & Spence, I. (2003). Use case modeling. Boston: Addison-Wesley.

    Google Scholar 

  16. Kulak, D., & Guiney, E. (2004). Use cases: Requirements in context. Boston, Mass: Addison-Wesley.

    Google Scholar 

  17. Cockburn, A. (2002). Agile software development. Boston: Addison-Wesley.

    Google Scholar 

  18. Highsmith, J. A. (2004). Agile project management: Creating innovative products. Boston: Addison-Wesley.

    Google Scholar 

  19. Jansky, L. J., & Huang, J. C. (2009). A multi-method approach to assess usability and acceptability: A case study of the patient-reported outcomes measurement system (PROMIS) workshop. Social Science Computer Review, 27(2), 267–270.

    Article  Google Scholar 

  20. Harniss, M. K., & Amtmann, D. (2008). Patient reported outcomes measurement information system (PROMIS) network study: Accessibility of the PROMIS computer adaptive testing system. Seattle: University of Washington Center on Outcomes in Rehabilitation Research.

    Google Scholar 

  21. Section 508 of the Rehabilitation Act, as amended by the Workforce Investment Act of 1998 (P.L. 105–220)(1998).

  22. Lie, H. W., & Bos, B. (2005). Cascading style sheets: Designing for the web. Boston: Addison-Wesley Professional.

    Google Scholar 

  23. Meyer, E. A. (2006). Cascading style sheets: The definitive guide. Sebastopol, CA: O’Reilly & Associates.

    Google Scholar 

  24. Garcia, S. F., Cella, D., Clauser, S. B., Flynn, K. E., Lai, J. S., Reeve, B. B., et al. (2007). Standardizing patient-reported outcomes assessment in cancer clinical trials: A patient-reported outcomes measurement information system initiative. Journal of Clinical Oncology, 25(32), 5106–5112.

    Article  PubMed  Google Scholar 

  25. Gershon, R. G. (2007). NIH toolbox: Assessment of neurological and behavioral function. NIH (Contract HHS-N-260-2006 00007-C), from http://www.nihtoolbox.org.

  26. Chang, C. H. (2007). Patient-reported outcomes measurement and management with innovative methodologies and technologies. Quality of Life Research, 16(Suppl 1), 157–166.

    Article  PubMed  Google Scholar 

  27. Davis, K. M., & Cella, D. (2002). Assessing quality of life in oncology clinical practice: A review of barriers and critical success factors. Journal of Clinical Outcomes Management, 9(6), 327–332.

    Google Scholar 

  28. Ruta, D., Coutts, A., Abdalla, M., Masson, J., Russell, E., Brunt, P., et al. (1995). Feasibility of monitoring patient based health outcomes in a routine hospital setting. Quality in Health Care, 4(3), 161–165.

    Article  CAS  PubMed  Google Scholar 

  29. DesRoches, C. M., Campbell, E. G., Rao, S. R., Donelan, K., Ferris, T. G., Jha, A., et al. (2008). Electronic health records in ambulatory care–a national survey of physicians. New England Journal Medicine, 359(1), 50–60.

    Article  CAS  Google Scholar 

  30. Overhage, J. M., Evans, L., & Marchibroda, J. (2005). Communities’ readiness for health information exchange: The National Landscape in 2004. Journal of the American Medical Informatics Association, 12(2), 107–112.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Primary funding for the Assessment Center application has been provided as part of several federally funded projects sponsored by the National Institutes of Health including the Patient-Reported Outcomes Measurement Information System (PROMIS; U01 AR052177), Neurological Quality of Life (Neuro-QOL; HHSN 2652004236-01C), Refining and Standardizing Health Literacy Assessment (RO1 HL081485-03), the NIH Toolbox for the Assessment of Neurological and Behavioral Function (AG-260-06-01) and a recent award as the PROMIS Technology Center (U54AR057943). The authors would like to acknowledge the editorial assistance of Lani Gershon.

Author information

Affiliations

  1. Department of Medical Social Sciences, Northwestern University, 625 North Michigan Avenue, Suite 2700, Chicago, IL, 60611, USA

    Richard Gershon, Nan E. Rothrock & Rachel T. Hanrahan

  2. Westat, Rockville, MD, USA

    Liz J. Jansky

  3. University of Washington, Seattle, WA, USA

    Mark Harniss

  4. National Heart, Lung, and Blood Institute, Bethesda, MD, USA

    William Riley

Authors
  1. Richard Gershon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nan E. Rothrock
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rachel T. Hanrahan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liz J. Jansky
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark Harniss
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. William Riley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Gershon.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Gershon, R., Rothrock, N.E., Hanrahan, R.T. et al. The development of a clinical outcomes survey research application: Assessment CenterSM . Qual Life Res 19, 677–685 (2010). https://doi.org/10.1007/s11136-010-9634-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-010-9634-4

Keywords

  • Software
  • Software design
  • Outcome assessment (health care)
  • Psychometrics
  • Quality of life
  • Health surveys
  • Questionnaires