Clinical Orthopaedics and Related Research®

, Volume 469, Issue 6, pp 1750–1756 | Cite as

Can the Toronto Extremity Salvage Score Produce Reliable Results When Used Online?

Clinical Research



Web-based questionnaires have become popular, however, access to the Internet can be biased regarding age, gender, and education, among other factors. Therefore, it is unknown whether this is a reasonable avenue to administer a questionnaire to patients or whether Web-based can be a reliable alternative to paper-based.


We determined whether the Internet version of the Toronto Extremity Salvage Score is reproducible compared with the paper-based version and the compliance and completion rates.

Patients and Methods

The study population consisted of 81 adults who had had surgery for a musculoskeletal tumor of the lower extremity more than 12 months earlier. The Toronto Extremity Salvage Score was administered by paper at a baseline interview and then readministered via Internet 7 to 14 days later to those with access.


Sixty of the 81 patients (74%) were able to use the Internet. Increasing age and lower education levels were correlated with a lower likelihood of using the Internet. Questionnaires were done online and on paper by 56 patients but 10 were excluded because of self-reported change in circumstances. The mean TESS was 85.7 (range, 41.1–100; SD, 17.26) for the paper-based questionnaire and 85.2 (range, 42.5–100; SD, 17.47) for the Internet-based questionnaire. The intraclass correlation coefficient was 0.97.


The questionnaire can be transferred successfully to the Internet and can be used reliably instead of a paper-based instrument. Recruitment to use an Internet-based questionnaire is limited only by the percentage of patients able to access and use the Internet.


  1. 1.
    Athale N, Sturley A, Skoczen S, Kavanaugh A, Lenert L. A web-compatible instrument for measuring self-reported disease activity in arthritis. J Rheumatol. 2004;31:223–228.PubMedGoogle Scholar
  2. 2.
    Bälter KA, Bälter O, Fondell E, Lagerros YT. Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology. 2005;16:577–579.PubMedCrossRefGoogle Scholar
  3. 3.
    Bliven BD, Kaufman SE, Spertus JA. Electronic collection of health-related quality of life data: validity, time benefits, and patient preference. Qual Life Res. 2001;10:15–22.PubMedCrossRefGoogle Scholar
  4. 4.
    Buchanan T, Smith JL. Using the Internet for psychological research: personality testing on the World Wide Web. Br J Psychol. 1999;90:125–144.PubMedCrossRefGoogle Scholar
  5. 5.
    Bush N, Donaldson G, Moinpour C, Haberman M, Milliken D, Markle V, Lauson J. Development, feasibility and compliance of a web-based system for very frequent QOL and symptom home self-assessment after hematopoietic stem cell transplantation. Qual Life Res. 2005;14:77–93.PubMedCrossRefGoogle Scholar
  6. 6.
    Bushnell DM, Martin ML, Parasuraman B. Electronic versus paper questionnaires: a further comparison in persons with asthma. J Asthma. 2003;40:751–762.PubMedCrossRefGoogle Scholar
  7. 7.
    Cals JW, Boumans D, Lardinois RJ, Gonzales R, Hopstaken RM, Butler CC, Dinant GJ. Public beliefs on antibiotics and respiratory tract infections: an internet-based questionnaire study. Br J Gen Pract. 2007;57:942–947.PubMedCrossRefGoogle Scholar
  8. 8.
    Cantrell MA, Lupinacci P. Methodological issues in online data collection. J Adv Nurs. 2007;60:544–549.PubMedCrossRefGoogle Scholar
  9. 9.
    Davis AM, Bell RS, Badley EM, Yoshida K, Williams JI. Evaluating functional outcome in patients with lower extremity sarcoma. Clin Orthop Relat Res. 1999;358:90–100.PubMedCrossRefGoogle Scholar
  10. 10.
    Davis AM, Sennik S, Griffin AM, Wunder JS, O’Sullivan B, Catton CN, Bell RS. Predictors of functional outcomes following limb salvage surgery for lower-extremity soft tissue sarcoma. J Surg Oncol. 2000;73:206–211.PubMedCrossRefGoogle Scholar
  11. 11.
    Davis AM, Wright JG, Williams JI, Bombardier C, Griffin A, Bell RS. Development of a measure of physical function for patients with bone and soft tissue sarcoma. Qual Life Res. 1996;5:508–516.PubMedCrossRefGoogle Scholar
  12. 12.
    Davis RN. Web-based administration of a personality questionnaire: comparison with traditional methods. Behav Res Methods Instrum Comput. 1999;31:572–577.PubMedCrossRefGoogle Scholar
  13. 13.
    Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I. Increasing response rates to postal questionnaires: systematic review. BMJ. 2002;324:1183–1191.PubMedCrossRefGoogle Scholar
  14. 14.
    Ekman A, Dickman PW, Klint A, Weiderpass E, Litton JE. Feasibility of using Web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol. 2006;21:103–111.PubMedCrossRefGoogle Scholar
  15. 15.
    Ekman A, Klint A, Dickman PW, Adami HO, Litton JE. Optimizing the design of Web-based questionnaires: experience from a population-based study among 50,000 women. Eur J Epidemiol. 2007;22:293–300.PubMedCrossRefGoogle Scholar
  16. 16.
    Enneking W. Modification of the system for functional evaluation in the surgical management of musculoskeletal tumours. In: Enneking W, ed. Limb Salvage in Musculoskeletal Oncology. New York, NY: Churchill Livingstone; 1987;626–639.Google Scholar
  17. 17.
    Enneking WF, Dunham W, Gebhardt MC, Malawar M, Pritchard DJ. A system for the functional evaluation of reconstructive procedures after surgical treatment of tumors of the musculoskeletal system. Clin Orthop Relat Res. 1993;286:241-246.PubMedGoogle Scholar
  18. 18.
    Gerrand CH, Wunder JS, Kandel RA, O’Sullivan B, Catton CN, Bell RS, Griffin AM, Davis AM. The influence of anatomic location on functional outcome in lower-extremity soft-tissue sarcoma. Ann Surg Oncol. 2004;11:476–482.PubMedCrossRefGoogle Scholar
  19. 19.
    Gilmour JA, Scott SD, Huntington N. Nurses and Internet health information: a questionnaire survey. J Adv Nurs. 2008;61:19–28.PubMedGoogle Scholar
  20. 20.
    Hanscom B, Lurie JD, Homa K, Weinstein JN. Computerized questionnaires and the quality of survey data. Spine (Phila Pa 1976). 2002;27:1797–1801.Google Scholar
  21. 21.
    Harewood GC, Yacavone RF, Locke GR 3rd, Wiersema MJ. Prospective comparison of endoscopy patient satisfaction surveys: e-mail versus standard mail versus telephone. Am J Gastroenterol. 2001;96:3312–3317.PubMedCrossRefGoogle Scholar
  22. 22.
    Joinson A. Social desirability, anonymity, and Internet-based questionnaires. Behav Res Methods Instrum Comput. 1999;31:433–438.PubMedCrossRefGoogle Scholar
  23. 23.
    Jones JB, Snyder CF, Wu AW. Issues in the design of Internet-based systems for collecting patient-reported outcomes. Qual Life Res. 2007;16:1407–1417.PubMedCrossRefGoogle Scholar
  24. 24.
    Jones R, Pitt N. Health surveys in the workplace: comparison of postal, email and World Wide Web methods. Occup Med (Lond). 1999;49:556–558.CrossRefGoogle Scholar
  25. 25.
    Khalily C, Behnke S, Seligson D. Treatment of closed tibia shaft fractures: a survey from the 1997 Orthopaedic Trauma Association and Osteosynthesis International-Gerhard Kuntscher Kreis meeting. J Orthop Trauma. 2000;14:577–581.PubMedCrossRefGoogle Scholar
  26. 26.
    Kim HL, Gerber GS, Patel RV, Hollowell CM, Bales GT. Practice patterns in the treatment of female urinary incontinence: a postal and internet survey. Urology. 2001;57:45–48.PubMedCrossRefGoogle Scholar
  27. 27.
    Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: a randomized study of internet versus paper-and-pencil versions. J Med Internet Res. 2007;9:e25.PubMedCrossRefGoogle Scholar
  28. 28.
    Kumle M, Weiderpass E, Braaten T, Persson I, Adami H, Lund E. Use of oral contraceptives and breast cancer risk: The Norwegian-Swedish Women’s Lifestyle and Health Cohort Study. Cancer Epidemiol Biomarkers Prev. 2002;11:1375–1381.PubMedGoogle Scholar
  29. 29.
    Lampert MH, Gerber LH, Glatstein E, Rosenberg SA, Danoff JV. Soft tissue sarcoma: functional outcome after wide local excision and radiation therapy. Arch Phys Med Rehabil. 1984;65:477–480.PubMedGoogle Scholar
  30. 30.
    Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, Devereaux PJ, Guyatt GH. Internet versus mailed questionnaires: a controlled comparison (2). J Med Internet Res. 2004;6:e39.PubMedCrossRefGoogle Scholar
  31. 31.
    Matarasso A, Elkwood A, Rankin M, Elkowitz M. National plastic surgery survey: face lift techniques and complications. Plastic Reconstr Surg. 2000;106:1185–1195; discussion 1196.Google Scholar
  32. 32.
    Meropol NJ, Egleston BL, Buzaglo JS, Benson AB 3rd, Cegala DJ, Diefenbach MA, Fleisher L, Miller SM, Sulmasy DP, Weinfurt KP; CONNECT Study Research Group. Cancer patients’ preferences for quality and length of life. Cancer. 2008;113:3459–3466.PubMedCrossRefGoogle Scholar
  33. 33.
    O’Leary TE, Diller L, Recklitis CJ. The effects of response bias on self-reported quality of life among childhood cancer survivors. Qual Life Res. 2007;16:1211–1220.PubMedCrossRefGoogle Scholar
  34. 34.
    Ritter P, Lorig K, Laurent D, Matthews K. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. 2004;6:e29.PubMedCrossRefGoogle Scholar
  35. 35.
    Robinson MH, Spruce L, Eeles R, Fryatt I, Harmer CL, Thomas JM, Westbury G. Limb function following conservation treatment of adult soft tissue sarcoma. Eur J Cancer. 1991;27:1567–1574.PubMedCrossRefGoogle Scholar
  36. 36.
    Saleh KJ, Radosevich DM, Kassim RA, Moussa M, Dykes D, Bottolfson H, Gioe TJ, Robinson H. Comparison of commonly used orthopaedic outcome measures using palm-top computers and paper surveys. J Orthop Res. 2002;20:1146–1151.PubMedCrossRefGoogle Scholar
  37. 37.
    Schreiber D, Bell RS, Wunder JS, O’Sullivan B, Turcotte R, Masri BA, Davis AM. Evaluating function and health related quality of life in patients treated for extremity soft tissue sarcoma. Qual Life Res. 2006;15:1439–1446.PubMedCrossRefGoogle Scholar
  38. 38.
    Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psych Bull. 1979;86:420–428.CrossRefGoogle Scholar
  39. 39.
    Smith B, Smith TC, Gray GC, Ryan MA; Millennium Cohort Study Team. When epidemiology meets the Internet: Web-based surveys in the Millennium Cohort Study. Am J Epidemiol. 2007;166:1345–1354.PubMedCrossRefGoogle Scholar
  40. 40.
    Sugarbaker PH, Barofsky I, Rosenberg SA, Gianola FJ. Quality of life assessment of patients in extremity sarcoma clinical trials. Surgery. 1982;91:17–23.PubMedGoogle Scholar
  41. 41. Available at: Accessed July 29, 2010.
  42. 42.
    World Health Organization. International Classification of Impairments, Disabilities and Handicaps. Geneva, Switzerland: World Health Organization; 1990.Google Scholar
  43. 43.
    Wilson AS, Kitas GD, Carruthers DM, Reay C, Skan J, Harris S, Treharne GJ, Young SP, Bacon PA. Computerized information-gathering in specialist rheumatology clinics: an initial evaluation of an electronic version of the Short Form 36. Rheumatology (Oxford). 2002;41:268–273.CrossRefGoogle Scholar

Copyright information

© The Association of Bone and Joint Surgeons® 2010

Authors and Affiliations

  1. 1.Department of Orthopaedics and TraumaRoyal Adelaide HospitalAdelaideAustralia
  2. 2.Division of Health Care and Outcomes Research and Arthritis and Community Research and Evaluation Unit, Toronto Western Research InstituteUniversity of TorontoTorontoCanada

Personalised recommendations