Skip to main content
SpringerLink
Log in

Validation of PROMIS® Physical Function Computerized Adaptive Tests for Orthopaedic Foot and Ankle Outcome Research

  • Symposium: ABJS Carl T. Brighton Workshop on Outcome Measures
  • Published:
Clinical Orthopaedics and Related Research®

Abstract

Background

In 2012, the American Orthopaedic Foot & Ankle Society® established a national network for collecting and sharing data on treatment outcomes and improving patient care. One of the network’s initiatives is to explore the use of computerized adaptive tests (CATs) for patient-level outcome reporting.

Questions/purposes

We determined whether the CAT from the NIH Patient Reported Outcome Measurement Information System® (PROMIS®) Physical Function (PF) item bank provides efficient, reliable, valid, precise, and adequately covered point estimates of patients’ physical function.

Methods

After informed consent, 288 patients with a mean age of 51 years (range, 18–81 years) undergoing surgery for common foot and ankle problems completed a web-based questionnaire. Efficiency was determined by time for test administration. Reliability was assessed with person and item reliability estimates. Validity evaluation included content validity from expert review and construct validity measured against the PROMIS® Pain CAT and patient responses based on tradeoff perceptions. Precision was assessed by standard error of measurement (SEM) across patients’ physical function levels. Instrument coverage was based on a person-item map.

Results

Average time of test administration was 47 seconds. Reliability was 0.96 for person and 0.99 for item. Construct validity against the Pain CAT had an r value of −0.657 (p < 0.001). Precision had an SEM of less than 3.3 (equivalent to a Cronbach’s alpha of ≥ 0.90) across a broad range of function. Concerning coverage, the ceiling effect was 0.32% and there was no floor effect.

Conclusions

The PROMIS® PF CAT appears to be an excellent method for measuring outcomes for patients with foot and ankle surgery. Further validation of the PROMIS® item banks may ultimately provide a valid and reliable tool for measuring patient-reported outcomes after injuries and treatment.

Level of Evidence

Level III, diagnostic study. See Instructions for Authors for a complete description of levels of evidence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Amadio PC, Silverstein MD, Ilstrup MD, Schleck CK, Jensen LM. Outcome assessment for carpal tunnel surgery: the relative responsiveness of generic, arthritis-specific, disease-specific, and physical exam measures. J Hand Surg Am. 1996;21:338–346.

    Article  PubMed  CAS  Google Scholar 

  2. Amtmann D, Cook KF, Jensen MP, Chen WH, Choi S, Revicki D, Cella D, Rothrock N, Keefe F, Callahan L. Development of a PROMIS item bank to measure pain interference. Pain. 2010;150:173–182.

    Article  PubMed  Google Scholar 

  3. Bajaj JS, Thacker LR, Wade JB, Sanyal AJ, Heuman DM, Sterling RK, Gibson DP, Stravitz RT, Puri P, Fuchs M, Luketic V, Noble N, White M, Bell D, Revicki DA. PROMIS computerised adaptive tests are dynamic instruments to measure health-related quality of life in patients with cirrhosis. Aliment Pharmacol Ther. 2011;34:1123–1132.

    Article  PubMed  CAS  Google Scholar 

  4. Beaton DE, Hogg-Johnson S, Bombardier C. Evaluating changes in health status: reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders. J Clin Epidemiol. 1997;50:79–93.

    Article  PubMed  CAS  Google Scholar 

  5. Beaton DE, Richards RR. Assessing the reliability and responsiveness of 5 shoulder questionnaires. J Shoulder Elbow Surg. 1998;7:565–572.

    Article  PubMed  CAS  Google Scholar 

  6. Beskin J, Brage M, Guyton G, Saltzman C, Sands A, SooHoo NF, Kadel N, Stroud C, Thordarson D, Sangeorzan B. Reproducibility of the Foot Function Index: a report of the AOFAS outcomes committee. Foot Ankle Int. 2005;26:962–967.

    PubMed  Google Scholar 

  7. Bessette L, Sangha O, Kuntz KM, Keller RB, Lew RA, Fossel AH, Katz JN. Comparative responsiveness of generic vs. disease-specific and weighted vs. unweighted health status measures in carpal tunnel syndrome. Med Care. 1998;36:491–502.

    Article  PubMed  CAS  Google Scholar 

  8. Bond TG, Fox CM. Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Mahwah, NJ: Lawrence Erlbaum; 2001.

    Google Scholar 

  9. Budiman-Mak E, Conrad KJ, Roach KE. The Foot Function Index: a measure of foot pain and disability. J Clin Epidemiol. 1991;44:561–570.

    Article  PubMed  CAS  Google Scholar 

  10. Button G, Pinney S. A meta-analysis of outcome rating scales in foot and ankle surgery: is there a valid, reliable, and responsive system? Foot Ankle Int. 2004;25:521–525.

    PubMed  Google Scholar 

  11. Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, Ader D, Fries JF, Bruce B, Rose M; PROMIS Cooperative Group. The Patient Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45:S3–S11.

    Article  PubMed  Google Scholar 

  12. Davidshofer KR, Murphy CO. Psychological Testing: Principles and Applications. 6th ed. Upper Saddle River, NJ: Pearson/Prentice Hall; 2005.

    Google Scholar 

  13. Glazebrook M, Daniels T, Younger A, Foote CJ, Penner M, Wing K, Lau J, Leighton R, Dunbar M. Comparison of health-related quality of life between patients with end-stage ankle and hip arthrosis. J Bone Joint Surg Am. 2008;90:499–505.

    Article  PubMed  Google Scholar 

  14. Hung M, Carter M, Hayden C, Dzierzon R, Morales J, Snow L, Butler J, Bateman K, Samore M. Psychometric assessment of the patient activation measure short form (PAM-13) in rural settings. Qual Life Res. 2013;22:521–529.

    Article  PubMed  Google Scholar 

  15. Hung M, Clegg DO, Greene T, Saltzman CL. Evaluation of the PROMIS physical function item bank in orthopaedic patients. J Orthop Res. 2011;29:947–953.

    Article  PubMed  Google Scholar 

  16. Hung M, Clegg DO, Greene T, Weir C, Saltzman CL. A lower extremity physical function computerized adaptive testing instrument. Foot Ankle Int. 2012;33:326–335.

    Article  PubMed  Google Scholar 

  17. Hung M, Nickisch F, Beals T, Greene T, Clegg DO, Saltzman CL. A new paradigm for patient-reported outcomes assessment in foot and ankle research: computerized adaptive testing. Foot Ankle Int. 2012;33:621–626.

    Article  PubMed  Google Scholar 

  18. Jette AM, Haley SM, Ni P, Olarsch S, Moed R. Creating a computer adaptive test version of the late-life function and disability instrument. J Gerontol A Biol Sci Med Sci. 2008;63:1246–1256.

    Article  PubMed  Google Scholar 

  19. Katz JN, Gelberman RH, Wright EA, Lew RA, Liang MH. Responsiveness of self-reported and objective measures of disease severity in carpal tunnel syndrome. Med Care. 1994;32:1127–1132.

    Article  PubMed  CAS  Google Scholar 

  20. Kitaoka HB, Alexander IJ, Adelaar RS, Nunley JA, Myerson MS, Sanders M. Clinical rating systems for the ankle-hindfoot, midfoot, hallux, and lesser toes. Foot Ankle Int. 1994;15:349–353.

    Article  PubMed  CAS  Google Scholar 

  21. Kocher MS, Horan MP, Briggs KK, Richardson TR, O’Holleran J, Hawkins RJ. Reliability, validity, and responsiveness of the American Shoulder and Elbow Surgeons Subjective Shoulder Scale in patients with shoulder instability, rotator cuff disease, and glenohumeral arthritis. J Bone Joint Surg Am. 2005;87:2006–2011.

    Article  PubMed  Google Scholar 

  22. Kuyvenhoven MM, Gorter KJ, Zuithoff P, Budiman-Mak E, Conrad KJ, Post MW. The Foot Function Index with Verbal Rating Scales (FFI-5pt): a clinimetric evaluation and comparison with the original FFI. J Rheumatol. 2002;29:1023–1028.

    PubMed  Google Scholar 

  23. Liang MH, Fossel AH, Larson MG. Comparison of five health status instruments for orthopedic evaluation. Med Care. 1990;28:632–642.

    Article  PubMed  CAS  Google Scholar 

  24. Lingard EA, Katz JN, Wright RJ, Wright EA, Sledge CB. Validity and responsiveness of the Knee Society Clinical Rating System in comparison with the SF-36 and WOMAC. J Bone Joint Surg Am. 2001;83:1856–1864.

    PubMed  Google Scholar 

  25. Lohr KN, Aaronson NK, Alonso J, Burnam MA, Patrick DL, Perrin EB, Roberts JS. Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clin Ther. 1996;18:979–992.

    Article  PubMed  CAS  Google Scholar 

  26. Mangione CM, Goldman L, Orav EJ, Marcantonio ER, Pedan A, Ludwig LE, Donaldson MC, Sugarbaker DJ, Poss R, Lee TH. Health-related quality of life after elective surgery: measurement of longitudinal changes. J Gen Intern Med. 1997;12:686–697.

    Article  PubMed  CAS  Google Scholar 

  27. Martin DP, Engelberg R, Agel J, Swiontkowski MF. Comparison of the Musculoskeletal Function Assessment questionnaire with the Short Form-36, the Western Ontario and McMaster Universities Osteoarthritis Index, and the Sickness Impact Profile health-status measures. J Bone Joint Surg Am. 1997;79:1323–1335.

    PubMed  CAS  Google Scholar 

  28. Marx RG, Jones EC, Allen AA, Altchek DW, O’Brien SJ, Roeda SA, Williams RJ, Warren RF, Wickiewicz TZ. Reliability, validity, and responsiveness of four knee outcome scales for athletic patients. J Bone Joint Surg Am. 2001;83:1459–1469.

    PubMed  Google Scholar 

  29. Masters GN. A Rasch model for partial credit scoring. Psychometrika. 1982;47:149–174.

    Article  Google Scholar 

  30. Masters GN. The analysis of partial credit scoring. Applied Measurement in Education. 1988;1:279–297.

    Article  Google Scholar 

  31. McHorney CA, Ware JE, Raczek AE. The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Med Care. 1993;31:247–263.

    Article  PubMed  CAS  Google Scholar 

  32. Naal FD, Impellizzeri FM, Rippstein PF. Which are the most frequently used outcome instruments in studies on total ankle arthroplasty? Clin Orthop Relat Res. 2010;468:815–826.

    Article  PubMed  Google Scholar 

  33. Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Chicago, IL: The University of Chicago Press; 1980.

    Google Scholar 

  34. Rose M, Bjorner JB, Becker J, Fries JF, Ware JE. Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2008;61:17–33.

    Article  PubMed  CAS  Google Scholar 

  35. Saag KG, Saltzman CL, Brown K, Budiman-Mak E. The Foot Function Index for measuring rheumatoid arthritis pain: evaluating side-to-side reliability. Foot Ankle Int. 1996;17:506–510.

    Article  PubMed  CAS  Google Scholar 

  36. Salaffi F, Carotti M, Grassi W. Health-related quality of life in patients with hip or knee osteoarthritis: comparison of generic and disease-specific instruments. Clin Rheumatol. 2005;24:29–37.

    Article  PubMed  Google Scholar 

  37. Saltzman CL, Zimmerman MB, O’Rourke M, Brown TD, Buckwalter JA, Johnston R. Impact of comorbidities on the measurement of health in patients with ankle osteoarthritis. J Bone Joint Surg Am. 2006;88:2366–2372.

    Article  PubMed  Google Scholar 

  38. Soderman P, Malchau H. Validity and reliability of the Swedish WOMAC osteoarthritis index: a self-administered disease-specific questionnaire (WOMAC) versus generic instruments (SF-36 and NHP). Acta Orthop Scand. 2000;71:39–46.

    Article  PubMed  CAS  Google Scholar 

  39. SooHoo NF, Samimi D, Vyas R, Botzler T. Evaluation of the validity of the Foot Function Index in measuring outcomes in patients with foot and ankle disorders. Foot Ankle Int. 2006;27:38–42.

    PubMed  Google Scholar 

  40. SooHoo NF, Shuler MS, Fleming LL. Evaluation of the validity of the AOFAS Clinical Rating Systems by correlation to the SF-36. Foot Ankle Int. 2003;24:50–55.

    PubMed  Google Scholar 

  41. Suk M, Hanson B, Norvell D, Helfet DL. AO Handbook of Musculoskeletal Outcomes Measures and Instruments. New York, NY: Thieme; 2009.

    Google Scholar 

  42. Tuttleman M, Pillemer SR, Tilley BC, Fowler SE, Buckley LM, Alaracon GS, Trentham DE, Neuner R, Clegg DO, Leisen JC, Heyse SP. A cross sectional assessment of health status instruments in patients with rheumatoid arthritis participating in a clinical trial. J Rheumatol. 1997;24:1910–1915.

    PubMed  CAS  Google Scholar 

  43. Wright BD, Masters GN. Rating Scale Analysis: Rasch Measurement. Chicago, IL: MESA Press; 1982.

    Google Scholar 

Download references

Acknowledgments

The authors thank Ian Alexander MD (Cleveland Clinic–Medina Hospital, Medina, OH, USA) for his extensive contributions to the AOFAS OFAR Network and this project. We acknowledge the contributing investigators of the OFAR Network, including Timothy R. Daniels MD (University of Toronto, Toronto, ON, Canada), James Brodsky MD (Baylor University, Houston, TX, USA), Jon Deland MD (Hospital for Special Surgery, New York, NY, USA), Scott Ellis MD (Hospital for Special Surgery, New York, NY, USA), Phinit Phisitkul MD (University of Iowa, Iowa City, IA, USA), W. Hodges Davis MD (OrthoCarolina Research Institute, Charlotte, NC, USA), Susan Ishikawa MD (The University of Tennessee–Campbell Clinic, Memphis, TN, USA), Loretta B. Chou MD (Stanford University, Redwood City, CA, USA), John Ketz MD (University of Rochester, Rochester, NY, USA), Margaret Chilvers MD (University of Arizona, Tucson, AZ, USA), Sam Flemister MD (University of Rochester, Rochester, NY, USA), Florian Nickisch MD (University of Utah, Salt Lake City, UT, USA), and Timothy Beals MD (University of Utah, Salt Lake City, UT, USA). We are thankful for the contributions of each of the 10 clinical testing sites in the OFAR Network. We also thank Lousanne Lofgren BS (AOFAS, Rosemont, IL, USA) and Rose Olea BS (AOFAS, Rosemont, IL, USA) for their tireless work on this project. Special thanks go to Christine Cheng BS (University of Utah, Salt Lake City, UT, USA) and Shirley D. Hon BS (University of Utah, Salt Lake City, UT, USA) for their tremendous support in data collection, analysis, manuscript preparation, and review. Finally, the study design, statistical and psychometric support, and computing resources provided by the Center for Outcomes Research and Assessment at the University of Utah Department of Orthopaedics are gratefully acknowledged.

Author information

Authors and Affiliations

  1. Department of Orthopaedic Surgery Operations, University of Utah School of Medicine, 590 Wakara Way, Salt Lake City, UT, 84108, USA

    Man Hung PhD, MSTAT, MED & Charles L. Saltzman MD

  2. Department of Orthopaedic Surgery, University of Rochester School of Medicine and Dentistry, Rochester, NY, USA

    Judith F. Baumhauer MD, MPH

  3. Department of Orthopaedic Surgery, University of Arizona, Tucson, AZ, USA

    L. Daniel Latt MD, PhD

  4. Department of Orthopaedic Surgery, University of California Los Angeles School of Medicine, Los Angeles, CA, USA

    Nelson F. SooHoo MD

  5. Department of Orthopaedic Surgery, Stanford University, Redwood City, CA, USA

    Kenneth J. Hunt MD

Authors
  1. Man Hung PhD, MSTAT, MED
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Judith F. Baumhauer MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. L. Daniel Latt MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Charles L. Saltzman MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nelson F. SooHoo MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kenneth J. Hunt MD
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

National Orthopaedic Foot & Ankle Outcomes Research Network

Corresponding author

Correspondence to Man Hung PhD, MSTAT, MED.

Additional information

Each author certifies that he or she has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.

All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research editors and board members are on file with the publication and can be viewed on request.

Each author certifies that his or her institution approved the human protocol for this investigation, that all investigations were conducted in conformity with ethical principles of research, and that informed consent for participation in the study was obtained.

Subjects were enrolled through a distributed consortium of foot and ankle clinics participating in the National Orthopaedic Foot & Ankle Outcomes Research Network, sponsored by the American Orthopaedic Foot & Ankle Society®. Institutions participating in this network include Baylor University (Houston, TX, USA), The University of Tennessee–Campbell Clinic (Memphis, TN, USA), Hospital for Special Surgery (New York, NY, USA), OrthoCarolina Research Institute (Charlotte, NC, USA), Stanford University (Redwood City, CA, USA), University of Arizona (Tucson, AZ, USA), University of California Los Angeles (Los Angeles, CA, USA), University of Iowa (Iowa City, IA, USA), University of Rochester (Rochester, NY, USA), and University of Utah (Salt Lake City, UT, USA).

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 49 kb)

Supplementary material 2 (DOCX 35 kb)

Appendix 1

Appendix 1

National Orthopaedic Foot & Ankle Outcomes Research Network Sites and Principal Investigators

Site

Principal investigator

The University of Tennessee–Campbell Clinic (Memphis, TN, USA)

Sue Ishikawa MD, G. Andrew Murphy MD, David Richardson MD

Stanford University (Redwood City, CA, USA)

Kenneth J. Hunt MD

University of Arizona (Tucson, AZ, USA)

L. Daniel Latt MD

University of Utah (Salt Lake City, UT, USA)

Charles L. Saltzman MD, Man Hung PhD

University of Iowa (Iowa City, IA, USA)

Phinit Phisitkul MD

University of Rochester (Rochester, NY, USA)

Judith F. Baumhauer MD

Hospital for Special Surgery (New York, NY, USA)

Jonathan T. Deland MD, Scott Ellis MD

University of California Los Angeles (Los Angeles, CA, USA)

Nelson Soohoo MD

OrthoCarolina Research Institute (Charlotte, NC, USA)

W. Hodges Davis MD

Baylor University (Houston, TX, USA)

James W. Brodsky MD

About this article

Cite this article

Hung, M., Baumhauer, J.F., Latt, L.D. et al. Validation of PROMIS® Physical Function Computerized Adaptive Tests for Orthopaedic Foot and Ankle Outcome Research. Clin Orthop Relat Res 471, 3466–3474 (2013). https://doi.org/10.1007/s11999-013-3097-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11999-013-3097-1

Keywords

Search

Navigation