Abstract
Introduction
The National Institutes of Health sponsored Patient-Reported Outcome Measurement Information System (PROMIS) aimed to create item banks and computerized adaptive tests (CATs) across multiple domains for individuals with a range of chronic diseases.
Purpose
Web-based software was created to enable a researcher to create study-specific Websites that could administer PROMIS CATs and other instruments to research participants or clinical samples. This paper outlines the process used to develop a user-friendly, free, Web-based resource (Assessment CenterSM) for storage, retrieval, organization, sharing, and administration of patient-reported outcomes (PRO) instruments.
Methods
Joint Application Design (JAD) sessions were conducted with representatives from numerous institutions in order to supply a general wish list of features. Use Cases were then written to ensure that end user expectations matched programmer specifications. Program development included daily programmer “scrum” sessions, weekly Usability Acceptability Testing (UAT) and continuous Quality Assurance (QA) activities pre- and post-release.
Results
Assessment Center includes features that promote instrument development including item histories, data management, and storage of statistical analysis results.
Conclusions
This case study of software development highlights the collection and incorporation of user input throughout the development process. Potential future applications of Assessment Center in clinical research are discussed.
Similar content being viewed by others
Abbreviations
- CAT:
-
Computerized adaptive test
- CTQ:
-
Critical to quality
- IRT:
-
Item response theory
- JAD:
-
Joint application design
- NIH:
-
National Institutes of Health
- NINDS:
-
National Institutes of Neurological Disorders and Stroke
- PRO:
-
Patient-reported outcomes
- PROMIS:
-
Patient-reported outcome measurement information system
- PRS:
-
Primary research sites
- QA:
-
Quality assurance
- SCC:
-
Statistical coordinating center
- UAT:
-
Usability acceptability testing
References
Zerhouni, E. (2003). Medicine. The NIH roadmap. Science, 302(5642), 63–72.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11.
DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical Care, 45(5 Suppl 1), S12–S21.
Cella, D., Riley, W., Stone, A. A., Rothrock, N., Reeve, B. B., & Yount, S., et al. (2009). Initial item banks and first wave testing of the patient reported outcomes measurement information system (PROMIS) network: 2005–2008. Journal of Clinical Epidemiology, In Press.
Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141.
Lai, J. S., Cella, D., Chang, C. H., Bode, R. K., & Heinemann, A. W. (2003). Item banking to improve, shorten and computerize self-reported fatigue: An illustration of steps to create a core item bank from the FACIT-Fatigue Scale. Quality of Life Research, 12(5), 485–501.
Davis, K. M., Chang, C. H., Lai, J. S., & Cella, D. (2002). Feasibility and acceptability of computerized adaptive testing (CAT) for fatigue monitoring in clinical practice. Quality of Life Research, 11(7), 134.
Ware, J. E., Jr., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlof, C. G., et al. (2003). Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Quality of Life Research, 12(8), 935–952.
O’Carroll, P. W., O’Carroll, P. W., Yasnoff, W. A., Ward, M. E., Ripp, L. H., & Martin, E. L. (2003). Information architecture. In K. J. Hannah & M. J. Ball (Eds.), Public health informatics and information systems (pp. 85–97). New York: Springer.
Davidson, E. J. (1999). Joint application design (JAD) in practice. Journal of Systems and Software, 45(3), 215–223.
Schroeder, R. G., Linderman, K., Liedtke, C., & Choo, A. S. (2008). Six sigma: Definition and underlying theory. Journal of Operations Management, 26, 536–554.
Lane, J. P., Usiak, D. J., Stone, V. I., & Scherer, M. J. (1997). The voice of the customer: Consumers define the ideal battery charger. Assistive Technology, 9(2), 130–139.
Sharp, A., & McDermott, P. (2001). Workflow modeling: Tools for process improvement and application development. Boston: Artech House.
Reijers, H. A. (2003). Design and control of workflow processes: Business process management for the service industry. Berlin; New York: Springer.
Bittner, K., & Spence, I. (2003). Use case modeling. Boston: Addison-Wesley.
Kulak, D., & Guiney, E. (2004). Use cases: Requirements in context. Boston, Mass: Addison-Wesley.
Cockburn, A. (2002). Agile software development. Boston: Addison-Wesley.
Highsmith, J. A. (2004). Agile project management: Creating innovative products. Boston: Addison-Wesley.
Jansky, L. J., & Huang, J. C. (2009). A multi-method approach to assess usability and acceptability: A case study of the patient-reported outcomes measurement system (PROMIS) workshop. Social Science Computer Review, 27(2), 267–270.
Harniss, M. K., & Amtmann, D. (2008). Patient reported outcomes measurement information system (PROMIS) network study: Accessibility of the PROMIS computer adaptive testing system. Seattle: University of Washington Center on Outcomes in Rehabilitation Research.
Section 508 of the Rehabilitation Act, as amended by the Workforce Investment Act of 1998 (P.L. 105–220)(1998).
Lie, H. W., & Bos, B. (2005). Cascading style sheets: Designing for the web. Boston: Addison-Wesley Professional.
Meyer, E. A. (2006). Cascading style sheets: The definitive guide. Sebastopol, CA: O’Reilly & Associates.
Garcia, S. F., Cella, D., Clauser, S. B., Flynn, K. E., Lai, J. S., Reeve, B. B., et al. (2007). Standardizing patient-reported outcomes assessment in cancer clinical trials: A patient-reported outcomes measurement information system initiative. Journal of Clinical Oncology, 25(32), 5106–5112.
Gershon, R. G. (2007). NIH toolbox: Assessment of neurological and behavioral function. NIH (Contract HHS-N-260-2006 00007-C), from http://www.nihtoolbox.org.
Chang, C. H. (2007). Patient-reported outcomes measurement and management with innovative methodologies and technologies. Quality of Life Research, 16(Suppl 1), 157–166.
Davis, K. M., & Cella, D. (2002). Assessing quality of life in oncology clinical practice: A review of barriers and critical success factors. Journal of Clinical Outcomes Management, 9(6), 327–332.
Ruta, D., Coutts, A., Abdalla, M., Masson, J., Russell, E., Brunt, P., et al. (1995). Feasibility of monitoring patient based health outcomes in a routine hospital setting. Quality in Health Care, 4(3), 161–165.
DesRoches, C. M., Campbell, E. G., Rao, S. R., Donelan, K., Ferris, T. G., Jha, A., et al. (2008). Electronic health records in ambulatory care–a national survey of physicians. New England Journal Medicine, 359(1), 50–60.
Overhage, J. M., Evans, L., & Marchibroda, J. (2005). Communities’ readiness for health information exchange: The National Landscape in 2004. Journal of the American Medical Informatics Association, 12(2), 107–112.
Acknowledgments
Primary funding for the Assessment Center application has been provided as part of several federally funded projects sponsored by the National Institutes of Health including the Patient-Reported Outcomes Measurement Information System (PROMIS; U01 AR052177), Neurological Quality of Life (Neuro-QOL; HHSN 2652004236-01C), Refining and Standardizing Health Literacy Assessment (RO1 HL081485-03), the NIH Toolbox for the Assessment of Neurological and Behavioral Function (AG-260-06-01) and a recent award as the PROMIS Technology Center (U54AR057943). The authors would like to acknowledge the editorial assistance of Lani Gershon.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gershon, R., Rothrock, N.E., Hanrahan, R.T. et al. The development of a clinical outcomes survey research application: Assessment CenterSM . Qual Life Res 19, 677–685 (2010). https://doi.org/10.1007/s11136-010-9634-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-010-9634-4