Osteoporosis International

, Volume 23, Issue 2, pp 423–431

Pitfalls in the external validation of FRAX

Opinion Paper



Recent studies have evaluated the performance of FRAX® in independent cohorts. The interpretation of most is problematic for reasons summarised in this perspective.


FRAX is an extensively validated assessment tool for the prediction of fracture in men and women. The aim of this study was to review the methods used since the launch of FRAX to further evaluate this instrument.


This covers a critical review of studies investigating the calibration of FRAX or assessing its performance characteristics in external cohorts.


Most studies used inappropriate methodologies to compare the performance characteristics of FRAX with other models. These included discordant parameters of risk (comparing incidence with probabilities), comparison with internally derived predictors and inappropriate use and interpretation of receiver operating characteristic curves. These deficits markedly impair interpretation of these studies.


Cohort studies that have evaluated the performance of FRAX need to be interpreted with caution and preferably re-evaluated.


Clinical risk factors Fracture probability FRAX Osteoporotic fracture Receiver operating characteristics 


  1. 1.
    Hippisley-Cox J, Coupland C (2009) Predicting risk of osteoporotic fracture in men and women in England and Wales: prospective derivation and validation of QFractures Scores. Br Med J 339:b4229CrossRefGoogle Scholar
  2. 2.
    Chen P, Krege JH, Adachi JD et al (2009) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk. J Bone Miner 24:495–502CrossRefGoogle Scholar
  3. 3.
    Sornay-Rendu E, Munoz F, Delmas PD, Chapurlat RD (2010) The FRAX® tool in French women: how well does it describe the real incidence of fracture in the OFELY cohort. J Bone Miner Res 25:2101–2107PubMedCrossRefGoogle Scholar
  4. 4.
    Bolland MJ, Siu AT, Mason BH et al (2010) (2011) Evaluation of the FRAX and Garvan fracture risk calculators in older women. J Bone Miner Res 26:420–427CrossRefGoogle Scholar
  5. 5.
    Ensrud KE, Lui LY, Taylor BC et al (2009) A comparison of prediction models for fractures in older women: is more better? Arch Intern Med 169:2087–2094PubMedCrossRefGoogle Scholar
  6. 6.
    Tremollieres FA, Pouilles JM, Drewniak N, Laparra J, Ribot CA, Dargent-Molina P (2010) Fracture risk prediction using BMD and clinical risk factors in early postmenopausal women: sensitivity of the WHO FRAX tool. J Bone Miner Res 25:1002–1009PubMedCrossRefGoogle Scholar
  7. 7.
    Donaldson MG, Palermo L, Schousboe JT, Ensrud KE, Hochberg MC, Cummings SR (2009) FRAX and risk of vertebral fractures: the fracture intervention trial. J Bone Miner Res 24:1793–1799PubMedCrossRefGoogle Scholar
  8. 8.
    Hillier TA, Cauley JA, Rizzo JH (2011) The WHO absolute fracture risk models (FRAX): do clinical risk factors improve fracture prediction in older women without osteoporosis? J Bone Miner Res 26:1774–1782PubMedCrossRefGoogle Scholar
  9. 9.
    Kayan K, Johansson H, Oden A et al (2009) Can fall risk be incorporated into fracture risk assessment algorithms: a pilot study of responsiveness to clodronate. Osteoporos Int 20:2055–2061PubMedCrossRefGoogle Scholar
  10. 10.
    Pluskiewicz W, Adamczyk P, Franek E et al (2010) Ten-year probability of osteoporotic fracture in 2012 Polish women assessed by FRAX and nomogram by Nguyen et al.—conformity between methods and their clinical utility. Bone 46:1661–1667PubMedCrossRefGoogle Scholar
  11. 11.
    Kanis JA, Oden A, Johansson H, McCloskey EV (2008) Expressing fracture risk (letter). Osteoporos Int 19:593–594CrossRefGoogle Scholar
  12. 12.
    Kanis JA, Oden A, Johnell O et al (2007) The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int 18:1033–1046PubMedCrossRefGoogle Scholar
  13. 13.
    Kanis JA on behalf of the World Health Organization Scientific Group (2008) Assessment of osteoporosis at the primary health-care level. Technical Report. WHO Collaborating Centre, University of Sheffield, UK. Available at http://www.shef.ac.uk/FRAX/Google Scholar
  14. 14.
    Papaioannou A, Morin S, Cheung AM, Scientific Advisory Council of Osteoporosis Canada (2010) 2010 clinical practice guidelines for the diagnosis and management of osteoporosis in Canada: summary. CMAJ 23(182):864–873Google Scholar
  15. 15.
    Dawson-Hughes B, National Osteoporosis Foundation Guide Committee (2008) A revised clinician’s guide to the prevention and treatment of osteoporosis. J Clin Endocrinol Metab 93:2463–2465PubMedCrossRefGoogle Scholar
  16. 16.
    Grossman JM, Gordon R, Ranganath VK et al (2010) American College of Rheumatology 2010 recommendations for the prevention and treatment of glucocorticoid-induced osteoporosis. Arthritis Care Res (Hoboken) 62:1515–1526CrossRefGoogle Scholar
  17. 17.
    National Osteoporosis Foundation (2008) Clinician’s guide to prevention and treatment of osteoporosis. National Osteoporosis Foundation. www.nof.org, Washington, DCGoogle Scholar
  18. 18.
    Pencina MJ, D’Agostino RB, Sr RB, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statist Med 27:157–172CrossRefGoogle Scholar
  19. 19.
    Kanis JA, Johnell O, Oden A et al (2006) The use of multiple sites for the diagnosis of osteoporosis. Osteoporos Int 17:527–534PubMedCrossRefGoogle Scholar
  20. 20.
    Blake GM, Patel R, Knapp KM, Fogelman I (2003) Does the combination of two BMD measurements improve fracture discrimination? J Bone Miner Res 18:1955–1963PubMedCrossRefGoogle Scholar
  21. 21.
    Leslie WD, Lix LM, Tsang JF, Caetano PA, Program Manitoba Bone Density (2007) Single-site vs multisite bone density measurement for fracture prediction. Arch Intern Med 167:1641–1647PubMedCrossRefGoogle Scholar
  22. 22.
    Leslie WD, Lix LM, Johansson H, Oden A, McCloskey E, Kanis JA (2010) Spine-hip discordance and fracture risk assessment: a physician-friendly FRAX enhancement. Osteoporos Int. Oct 20. [Epub ahead of print] PubMed PMID: 20959961Google Scholar
  23. 23.
    Abrahamsen B, Vestergaard P, Rud B et al (2006) Ten-year absolute risk of osteoporotic fractures according to BMD T score at menopause: The Danish Osteoporosis Prevention Study. J Bone Miner Res 21:796–800PubMedCrossRefGoogle Scholar
  24. 24.
    Baron JA, Barrett J, Malenka D et al (1994) Racial differences in fracture risk. Epidemiol 5:42–47CrossRefGoogle Scholar
  25. 25.
    Jatrana S, Blakely T (2008) Ethnic inequalities in mortality among the elderly in New Zealand. Aust N Z J Public Health 32:437–443PubMedCrossRefGoogle Scholar
  26. 26.
    Levine S, Makin M, Menczel J, Robin G, Naor E, Steinberg R (1970) Incidence of fractures of the proximal end of the femur in Jerusalem: a study of ethnic factors. J Bone Joint Surg Am 52:1193–1202PubMedGoogle Scholar
  27. 27.
    Wittich A, Bagur A, Mautalen C et al (2010) Epidemiology of hip fracture in Tucuman, Argentina. Osteoporos Int 21:1803–1807PubMedCrossRefGoogle Scholar
  28. 28.
    Elffors I, Allander E, Kanis JA et al (1994) The variable incidence of hip fracture in southern Europe: the MEDOS Study. Osteoporos Int 4:253–263PubMedCrossRefGoogle Scholar
  29. 29.
    Jonsson B, Gardsell P, Johnell O, Redlund-Johnell I, Sernbo I (1992) Differences in fracture pattern between an urban and a rural population: a comparative population-based study in southern Sweden. Osteoporos Int 2:269–273PubMedCrossRefGoogle Scholar
  30. 30.
    Finsen V, Benum P (1987) Changing incidence of hip fractures in rural and urban areas of central Norway. Clin Orthop Relat Res 218:104–110PubMedGoogle Scholar
  31. 31.
    Bulajic-Kopjar M, Wiik J, Nordhagen R (1998) Regional differences in the incidence of femoral neck fractures in Norway. Tidsskr Nor Laegeforen 118:30–33PubMedGoogle Scholar
  32. 32.
    Chevalley T, Herrmann FR, Delmi et al (2002) Evaluation of the age-adjusted incidence of hip fractures between urban and rural areas: the difference is not related to the prevalence of institutions for the elderly. Osteoporos Int 13:113–118PubMedCrossRefGoogle Scholar
  33. 33.
    Matković V, Kostial K, Simonović I, Buzina R, Brodarec A, Nordin BE (1979) Bone status and fracture rates in two regions of Yugoslavia. Am J Clin Nutr 32:540–549PubMedGoogle Scholar
  34. 34.
    Madhok R, Melton LJ 3rd, Atkinson EJ, O’Fallon WM, Lewallen DG (1993) Urban vs rural increase in hip fracture incidence. Age and sex of 901 cases 1980–89 in Olmsted County, U.S.A. Acta Orthop Scand 64:543–548PubMedCrossRefGoogle Scholar
  35. 35.
    Kaastad TS, Meyer HE, Falch JA (2008) Incidence of hip fracture in Oslo, Norway: differences within the city. Bone 22:175–178CrossRefGoogle Scholar
  36. 36.
    Kanis JA, Oden A, Johnell O, Jonsson B, de Laet C, Dawson A (2001) The burden of osteoporotic fractures: a method for setting intervention thresholds. Osteoporos Int 12:417–427PubMedCrossRefGoogle Scholar
  37. 37.
    Melton LJ, Crowson CS, O’Fallon WM (1999) Fracture incidence in Olmsted County, Minnesota: comparison of urban and with rural rates and changes in urban rates over time. Osteoporos Int 9:29–37PubMedCrossRefGoogle Scholar
  38. 38.
    Singer BR, McLauchlan CJ, Robinson CM, Christie J (1998) Epidemiology of fracture in 15,000 adults. The influence of age and gender. J Bone Joint Surg 80B:234–238Google Scholar
  39. 39.
    Lippuner K, Johansson H, Kanis JA, Rizzoli R (2010) FRAX® assessment of osteoporotic fracture probability in Switzerland. Osteoporos Int 21:381–390PubMedCrossRefGoogle Scholar
  40. 40.
    Sanders KM, Pasco JA, Ugoni AM et al (1998) The exclusion of high trauma fractures may underestimate the prevalence of bone fragility fractures in the community: the Geelong Osteoporosis Study. J Bone Miner Res 13:1337–1342PubMedCrossRefGoogle Scholar
  41. 41.
    Melton LJ (1995) Epidemiology of fractures. In: Riggs BL, Melton LJ (eds) Osteoporosis: etiology, diagnosis and management, 2nd edn. Lippincott-Raven, Philadelphia, pp 225–227Google Scholar
  42. 42.
    Johnell O, Gullberg B, Kanis JA (1997) The hospital burden of vertebral fracture. A study of national register sources. Osteoporos Int 7:138–144PubMedCrossRefGoogle Scholar
  43. 43.
    Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2008) Development of prognostic nomograms for individualizing 5-year and 10-year fracture risks. Osteoporos Int 19:1431–1444PubMedCrossRefGoogle Scholar
  44. 44.
    Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2007) Development of a nomogram for individualizing hip fracture risk in men and women. Osteoporos Int 18:1109–1117PubMedCrossRefGoogle Scholar
  45. 45.
    Gillespie LD, Robertson MC, Gillespie WJ et al. (2009) Interventions for preventing falls in older people living in the community. Cochrane Database of Systematic Reviews 2009, Issue 2. Art.No.: D007146.doi:10.1002/14651858.CD007146.pub2
  46. 46.
    McClung MR, Geusens P, Miller PD et al (2001) Effect of risedronate on the risk of hip fracture in elderly women. Hip Intervention Program Study Group. N Engl J Med 344:333–340PubMedCrossRefGoogle Scholar
  47. 47.
    Langsetmo L, Nguyen TV, Nguyen ND et al (2010) Independent external validation of nomograms for predicting risk of low-trauma fracture and hip fracture. CMAJ. doi:10.1503/cmaj.100458
  48. 48.
    Sk S, Nguyen ND, Center JR et al (2010) Prognosis of fracture: evaluation of predictive accuracy of the FRAX algorithm and Garvan nomogram. Osteoporos Int 21:863–871CrossRefGoogle Scholar
  49. 49.
    Van Geel ACM, van den Bergh JPW, Dinant GJ, Geusens PP (2010) Individualizing fracture risk prediction. Maturitas 65:143–148PubMedCrossRefGoogle Scholar
  50. 50.
    Geusens P, Van Geel T, Van Den Berg J (2010) Can hip fracture prediction in women be estimated beyond bone mineral density measurement alone. Ther Adv Musculoskelet Dis 2:63–67CrossRefGoogle Scholar
  51. 51.
    Van Geel T, Huntjens K, Bours S et al (2010) Comparison of FRAX and Garvan case finding algorithms in patients presenting with a fracture. Bone 47:S188CrossRefGoogle Scholar
  52. 52.
    Collins GS, Mallett S, Altman DG (2011) Predicting risk of osteoporotic and hip fracture in the United Kingdom: prospective independent and external validation of QFractureScores. BMJ 342:d3651. doi:10.1136/bmj.d3651
  53. 53.
    Cummins NM, Poku EK, Towler MR, O'Driscoll OM, Ralston SH (2011) Clinical risk factors for osteoporosis in Ireland and the UK: a comparison of FRAX and QFractureScores. Calcif Tissue Int 89:172–177Google Scholar
  54. 54.
    Fujiwara S, Hamaya E, Goto W, Masunari N, Furukawa K, Fukunaga M, Nakamura T, Miyauchi A, Chen P (2011) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk in Japan. Bone 49(3):520–525Google Scholar

Copyright information

© International Osteoporosis Foundation and National Osteoporosis Foundation 2011

Authors and Affiliations

  • J. A. Kanis
    • 1
  • A. Oden
    • 1
  • H. Johansson
    • 1
  • E. McCloskey
    • 1
  1. 1.WHO Collaborating Centre for Metabolic Bone DiseasesUniversity of Sheffield Medical SchoolSheffieldUK

Personalised recommendations