Quality of Life Research

, Volume 19, Issue 5, pp 677–685 | Cite as

The development of a clinical outcomes survey research application: Assessment CenterSM

  • Richard GershonEmail author
  • Nan E. Rothrock
  • Rachel T. Hanrahan
  • Liz J. Jansky
  • Mark Harniss
  • William Riley



The National Institutes of Health sponsored Patient-Reported Outcome Measurement Information System (PROMIS) aimed to create item banks and computerized adaptive tests (CATs) across multiple domains for individuals with a range of chronic diseases.


Web-based software was created to enable a researcher to create study-specific Websites that could administer PROMIS CATs and other instruments to research participants or clinical samples. This paper outlines the process used to develop a user-friendly, free, Web-based resource (Assessment CenterSM) for storage, retrieval, organization, sharing, and administration of patient-reported outcomes (PRO) instruments.


Joint Application Design (JAD) sessions were conducted with representatives from numerous institutions in order to supply a general wish list of features. Use Cases were then written to ensure that end user expectations matched programmer specifications. Program development included daily programmer “scrum” sessions, weekly Usability Acceptability Testing (UAT) and continuous Quality Assurance (QA) activities pre- and post-release.


Assessment Center includes features that promote instrument development including item histories, data management, and storage of statistical analysis results.


This case study of software development highlights the collection and incorporation of user input throughout the development process. Potential future applications of Assessment Center in clinical research are discussed.


Software Software design Outcome assessment (health care) Psychometrics Quality of life Health surveys Questionnaires 



Computerized adaptive test


Critical to quality


Item response theory


Joint application design


National Institutes of Health


National Institutes of Neurological Disorders and Stroke


Patient-reported outcomes


Patient-reported outcome measurement information system


Primary research sites


Quality assurance


Statistical coordinating center


Usability acceptability testing



Primary funding for the Assessment Center application has been provided as part of several federally funded projects sponsored by the National Institutes of Health including the Patient-Reported Outcomes Measurement Information System (PROMIS; U01 AR052177), Neurological Quality of Life (Neuro-QOL; HHSN 2652004236-01C), Refining and Standardizing Health Literacy Assessment (RO1 HL081485-03), the NIH Toolbox for the Assessment of Neurological and Behavioral Function (AG-260-06-01) and a recent award as the PROMIS Technology Center (U54AR057943). The authors would like to acknowledge the editorial assistance of Lani Gershon.


  1. 1.
    Zerhouni, E. (2003). Medicine. The NIH roadmap. Science, 302(5642), 63–72.CrossRefPubMedGoogle Scholar
  2. 2.
    Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11.CrossRefPubMedGoogle Scholar
  3. 3.
    DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical Care, 45(5 Suppl 1), S12–S21.CrossRefPubMedGoogle Scholar
  4. 4.
    Cella, D., Riley, W., Stone, A. A., Rothrock, N., Reeve, B. B., & Yount, S., et al. (2009). Initial item banks and first wave testing of the patient reported outcomes measurement information system (PROMIS) network: 2005–2008. Journal of Clinical Epidemiology, In Press.Google Scholar
  5. 5.
    Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 133–141.CrossRefPubMedGoogle Scholar
  6. 6.
    Lai, J. S., Cella, D., Chang, C. H., Bode, R. K., & Heinemann, A. W. (2003). Item banking to improve, shorten and computerize self-reported fatigue: An illustration of steps to create a core item bank from the FACIT-Fatigue Scale. Quality of Life Research, 12(5), 485–501.CrossRefPubMedGoogle Scholar
  7. 7.
    Davis, K. M., Chang, C. H., Lai, J. S., & Cella, D. (2002). Feasibility and acceptability of computerized adaptive testing (CAT) for fatigue monitoring in clinical practice. Quality of Life Research, 11(7), 134.Google Scholar
  8. 8.
    Ware, J. E., Jr., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlof, C. G., et al. (2003). Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Quality of Life Research, 12(8), 935–952.CrossRefPubMedGoogle Scholar
  9. 9.
    O’Carroll, P. W., O’Carroll, P. W., Yasnoff, W. A., Ward, M. E., Ripp, L. H., & Martin, E. L. (2003). Information architecture. In K. J. Hannah & M. J. Ball (Eds.), Public health informatics and information systems (pp. 85–97). New York: Springer.CrossRefGoogle Scholar
  10. 10.
    Davidson, E. J. (1999). Joint application design (JAD) in practice. Journal of Systems and Software, 45(3), 215–223.CrossRefGoogle Scholar
  11. 11.
    Schroeder, R. G., Linderman, K., Liedtke, C., & Choo, A. S. (2008). Six sigma: Definition and underlying theory. Journal of Operations Management, 26, 536–554.CrossRefGoogle Scholar
  12. 12.
    Lane, J. P., Usiak, D. J., Stone, V. I., & Scherer, M. J. (1997). The voice of the customer: Consumers define the ideal battery charger. Assistive Technology, 9(2), 130–139.PubMedGoogle Scholar
  13. 13.
    Sharp, A., & McDermott, P. (2001). Workflow modeling: Tools for process improvement and application development. Boston: Artech House.Google Scholar
  14. 14.
    Reijers, H. A. (2003). Design and control of workflow processes: Business process management for the service industry. Berlin; New York: Springer.CrossRefGoogle Scholar
  15. 15.
    Bittner, K., & Spence, I. (2003). Use case modeling. Boston: Addison-Wesley.Google Scholar
  16. 16.
    Kulak, D., & Guiney, E. (2004). Use cases: Requirements in context. Boston, Mass: Addison-Wesley.Google Scholar
  17. 17.
    Cockburn, A. (2002). Agile software development. Boston: Addison-Wesley.Google Scholar
  18. 18.
    Highsmith, J. A. (2004). Agile project management: Creating innovative products. Boston: Addison-Wesley.Google Scholar
  19. 19.
    Jansky, L. J., & Huang, J. C. (2009). A multi-method approach to assess usability and acceptability: A case study of the patient-reported outcomes measurement system (PROMIS) workshop. Social Science Computer Review, 27(2), 267–270.CrossRefGoogle Scholar
  20. 20.
    Harniss, M. K., & Amtmann, D. (2008). Patient reported outcomes measurement information system (PROMIS) network study: Accessibility of the PROMIS computer adaptive testing system. Seattle: University of Washington Center on Outcomes in Rehabilitation Research.Google Scholar
  21. 21.
    Section 508 of the Rehabilitation Act, as amended by the Workforce Investment Act of 1998 (P.L. 105–220)(1998).Google Scholar
  22. 22.
    Lie, H. W., & Bos, B. (2005). Cascading style sheets: Designing for the web. Boston: Addison-Wesley Professional.Google Scholar
  23. 23.
    Meyer, E. A. (2006). Cascading style sheets: The definitive guide. Sebastopol, CA: O’Reilly & Associates.Google Scholar
  24. 24.
    Garcia, S. F., Cella, D., Clauser, S. B., Flynn, K. E., Lai, J. S., Reeve, B. B., et al. (2007). Standardizing patient-reported outcomes assessment in cancer clinical trials: A patient-reported outcomes measurement information system initiative. Journal of Clinical Oncology, 25(32), 5106–5112.CrossRefPubMedGoogle Scholar
  25. 25.
    Gershon, R. G. (2007). NIH toolbox: Assessment of neurological and behavioral function. NIH (Contract HHS-N-260-2006 00007-C), from
  26. 26.
    Chang, C. H. (2007). Patient-reported outcomes measurement and management with innovative methodologies and technologies. Quality of Life Research, 16(Suppl 1), 157–166.CrossRefPubMedGoogle Scholar
  27. 27.
    Davis, K. M., & Cella, D. (2002). Assessing quality of life in oncology clinical practice: A review of barriers and critical success factors. Journal of Clinical Outcomes Management, 9(6), 327–332.Google Scholar
  28. 28.
    Ruta, D., Coutts, A., Abdalla, M., Masson, J., Russell, E., Brunt, P., et al. (1995). Feasibility of monitoring patient based health outcomes in a routine hospital setting. Quality in Health Care, 4(3), 161–165.CrossRefPubMedGoogle Scholar
  29. 29.
    DesRoches, C. M., Campbell, E. G., Rao, S. R., Donelan, K., Ferris, T. G., Jha, A., et al. (2008). Electronic health records in ambulatory care–a national survey of physicians. New England Journal Medicine, 359(1), 50–60.CrossRefGoogle Scholar
  30. 30.
    Overhage, J. M., Evans, L., & Marchibroda, J. (2005). Communities’ readiness for health information exchange: The National Landscape in 2004. Journal of the American Medical Informatics Association, 12(2), 107–112.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Richard Gershon
    • 1
    Email author
  • Nan E. Rothrock
    • 1
  • Rachel T. Hanrahan
    • 1
  • Liz J. Jansky
    • 2
  • Mark Harniss
    • 3
  • William Riley
    • 4
  1. 1.Department of Medical Social SciencesNorthwestern UniversityChicagoUSA
  2. 2.WestatRockvilleUSA
  3. 3.University of WashingtonSeattleUSA
  4. 4.National Heart, Lung, and Blood InstituteBethesdaUSA

Personalised recommendations