International Journal of Legal Medicine

, Volume 133, Issue 3, pp 949–962 | Cite as

Observer error and its impact on ancestry estimation using dental morphology

  • Marin A. PilloudEmail author
  • Donovan M. Adams
  • Joseph T. Hefner
Original Article


Dental morphology is becoming increasingly visible in forensic anthropology as part of the estimation of ancestry. As methods are developed based on these data, it is important to understand the role of observer error in data collection and method application. In this study, 10 observers collected dental morphological data on 19 traits on the same set of nine plaques. Various measures of interrater reliability were calculated to assess observer error. Data were then input into one of three ancestry estimation methods based on dental morphology to understand the role of observer error in these methods. Results show low rater reliability for all dental morphological traits when all 10 observers are compared. Rater reliability increases when only experienced observers are compared and traits are dichotomized. Further, differences in trait scores by observers resulted in disparate estimations of ancestry in each of the methods. While observer error appears to be an issue in dental morphological methods of ancestry estimation, these problems can be addressed. An argument is made for advanced training in dental anthropology in laboratories and in graduate programs. Further, methods need to test for and employ traits with high rater agreement.


Forensic anthropology Interobserver error Rater reliability Dental non-metrics Breakpoints ASUDAS 



We thank the Defense POW/MIA Accounting Agency Laboratory for assistance and Heather J.H. Edgar for providing use of the casts from the Economides collection. Also, G. Richard Scott provided insight on the analyses of this manuscript.


  1. 1.
    Rhine S (1990) Non-metric skull racing. In: Gill GW, Rhine S (eds) Skeletal attribution of race: methods for forensic anthropology. Maxwell Museum Anthropological Papers No. 4, Albuquerque, N.M., pp 9–20Google Scholar
  2. 2.
    Hinkes MJ (1990) Shovel-shaped incisors in human identification. In: Gill G, Rhine S (eds) Skeletal attribution of race. vol 4. Maxwell Museum Anthropological Papers No. 4, Albuquerque, NM, pp 21–26Google Scholar
  3. 3.
    Birkby WH, Fenton TW, Anderson BE (2008) Identifying Southwest Hispanics using nonmetric traits and the cultural profile. J Forensic Sci 53(1):29–33. Google Scholar
  4. 4.
    Edgar HJH (2005) Prediction of race using characteristics of dental morphology. J Forensic Sci 50(2):269–273Google Scholar
  5. 5.
    Edgar HJH (2013) Estimation of ancestry using dental morphological characteristics. J Forensic Sci 58(s1):S3–S8Google Scholar
  6. 6.
    Scott GR, Pilloud MA, Navega D, d’Oliveira J, Cunha E, Irish DJ (2018) rASUDAS: a new web-based application for estimating ancestry from tooth morphology. Forensic Anthropol 1:18–31Google Scholar
  7. 7.
    Turner CG, Nichol CR, Scott GR (1991) Scoring procedures for key morphological traits of the permanent dentition: the Arizona State University Dental Anthropology System. In: Kelley MA, Larsen CS (eds) Advances in dental anthropology. Wiley-Liss, New York, pp 13–31Google Scholar
  8. 8.
    Pilloud MA, Maier C, Scott GR, Edgar HJH (2018) Molar crenulation trait definition and variation in modern human populations. Homo 69(3):77–85. Google Scholar
  9. 9.
    Scott GR, Irish JD (2017) Human crown and root morphology: the Arizona State University Dental Anthropology System. Cambridge University Press, CambridgeGoogle Scholar
  10. 10.
    Pilloud MA (2018) Technical note: the definition of new dental morphological variants related to malocclusion. Dent Anthropol 31(1):10–18Google Scholar
  11. 11.
    Burnett SE, Hawkey DE, Turner CG (2010) Brief communication: population variation in human maxillary premolar accessory ridges (MxPAR). Am J Phys Anthr 141(2):319–324. Google Scholar
  12. 12.
    Edgar HJH (2017) Dental morphology: an illustrated manual. Routledge, New YorkGoogle Scholar
  13. 13.
    Edgar HJH (2002) Biological distance and the African American dentition. PhD, The Ohio State University, ColumbusGoogle Scholar
  14. 14.
    Pilloud MA (2009) Community structure at Neolithic Çatalhöyük: biological distance analysis of household, neighborhood, and settlement. Unpublished PhD dissertation, The Ohio State University, Columbus, OhioGoogle Scholar
  15. 15.
    Scott GR (1973) Dental morphology: a genetic study of American white families and variation in living Southwest Indians. Unpublished PhD dissertation, Arizona State University, Tempe, AZGoogle Scholar
  16. 16.
    Sofaer JA, Niswander J, Maclean C, Workman P (1972) Population studies on southwestern Indian tribes. V. Tooth morphology as an indicator of biological distance. Am J Phys Anthr 37:357–366Google Scholar
  17. 17.
    Hsu JW, Tsai PL, Hsiao TH, Chang HP, Liu LM, Liu KM, Yu HS, Ferguson D (1999) Ethnic dental analysis of shovel and Carabelli’s traits in a Chinese population. Aust Dent J 44(1):40–45Google Scholar
  18. 18.
    Hasegawa Y, Rogers J, Scriven G, Townsend G (2010) Carabelli trait in Australian twins: reliability and validity of different scoring systems. Dent Anthropol 23:7–14Google Scholar
  19. 19.
    Willermet C, Edgar HJH, Ragsdale C, Aubry BS (2013) Biodistances among Mexica, Maya, Toltec, and Totonac groups of central and coastal Mexico. Chungará 45(3):447–459Google Scholar
  20. 20.
    Edgar HJH, Lease LR (2007) Correlations between deciduous and permanent tooth morphology in a European American sample. Am J Phys Anthr 133(1):726–734Google Scholar
  21. 21.
    Nichol CR, Turner CG (1986) Intra- and interobserver concordance in classifying dental morphology. Am J Phys Anthr 69(3):299–315Google Scholar
  22. 22.
    Edgar HJH, Daneshvari S, Harris EF, Kroth PJ (2011) Inter-observer agreement on subjects’ race and race-informative characteristics. PLoS One 6(8):e23986. Google Scholar
  23. 23.
    Scott GR, Turner CG (1997) The anthropology of modern human teeth. Cambridge University Press, CambridgeGoogle Scholar
  24. 24.
    Adams DM, George RL Fuzzy inference system (FIS) as a novel statistical method for forensic ancestry estimation. In: American Academy of Forensic Sciences, Seattle, WA, 2018. Proceedings: American Acadmy of Forensic Sciences, p 154Google Scholar
  25. 25.
    Maier C (2017) The combination of cranial morphoscopic and dental morphological methods to improve the forensic estimation of ancestry [PhD dissertation]. University of Nevada, RenoGoogle Scholar
  26. 26.
    Willermet CM, Daniels J, Edgar HJH Seeing RED: a new statistical solution to an old categorical data problem. In: American Association of Physical Anthropologists, Atlanta, Georgia, 2016. American Association of Physical AnthropologistsGoogle Scholar
  27. 27.
    R Core Team (2013) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna.
  28. 28.
    Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Am Psychol Assoc 70(4):213–220. Google Scholar
  29. 29.
    Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86(2):420–428Google Scholar
  30. 30.
    Koo TK, Li MY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15(2):155–163. Google Scholar
  31. 31.
    Klales AR, Burns TL (2017) Adapting and applying the Phenice (1969) adult morphological sex estimation technique to subadults. J Forensic Sci 62(3):747–752Google Scholar
  32. 32.
    Garvin HM, Klales AR (2018) A validation study of the Langley et al. (2017) decision tree model for sex estimation. J Forensic Sci 63(4):1243–1251Google Scholar
  33. 33.
    Shirley NR, Ramirez Montes PA (2015) Age estimation in forensic anthropology: quantification of observer error in phase versus component-based methods. J Forensic Sci 60(1):107–111Google Scholar
  34. 34.
    Byrnes JF, Kenyhercz MW, Berg GE (2017) Examining interobserver reliability of metric and morphoscopic characteristics of the mandible. J Forensic Sci 62(4):981–985. Google Scholar
  35. 35.
    Klales AR, Ousley SD, Vollner JM (2012) A revised method of sexing the human innominate using Phenice’s nonmetric traits and statistical methods. Am J Phys Anthr 149(1):104–114Google Scholar
  36. 36.
    Lesciotto KM, Doershuk LJ (2018) Accuracy and reliability of the Klales et al. (2012) morphoscopic pelvic sexing method. J Forensic Sci 63(1):214–220. Google Scholar
  37. 37.
    Fleiss JL, Cohen J (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 33:613–619Google Scholar
  38. 38.
    Feng GC (2015) Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodology 11(1):13–22. Google Scholar
  39. 39.
    Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382Google Scholar
  40. 40.
    Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88(2):322–328Google Scholar
  41. 41.
    Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 1(1):77–89. Google Scholar
  42. 42.
    McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med 22(3):276–282Google Scholar
  43. 43.
    Sim J, Wright CC (2005) The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 85(3):257–268. Google Scholar
  44. 44.
    Landis RJ, Koch GG (1977) The measure of observer agreement for categorical data. Biometrics 33:159–174Google Scholar
  45. 45.
    Taylor R (1990) Interpretation of the correlation coefficient: a basic review. J Diagn Med Sonograph 6(1):35–39. Google Scholar
  46. 46.
    Krippendorff K (2004) Reliability in content analysis. Hum Commun Res 30(3):411–433. Google Scholar
  47. 47.
    Krippendorff K (2004) Content analysis: an introduction to its methodology, 2nd edn. Sage, Thousand OaksGoogle Scholar
  48. 48.
    Scott GR (1977) Classification, sex dimorphism, association, and population variation of the canine distal accessory ridge. Hum Biol 49(3):453–469Google Scholar
  49. 49.
    Irish JD (1993) Biological affinities of late Pleistocene through modern Africal aboriginal populations: the dental evidence. Ph.D., Arizona State University, Tempe, AZGoogle Scholar
  50. 50.
    Scott GR, Turner CG, Townsend G, Martinon-Torres M (2018) Anthropology of modern human teeth: dental morphology and its variation in fossil and recent Homo sapiens. Cambridge University Press, CambridgeGoogle Scholar
  51. 51.
    Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61(1):29–48Google Scholar
  52. 52.
    Burnett SE, Irish JD, Fong MR (2013) Wear’s the problem? Examining the effect of dental wear on studies of crown morphology. In: Scott GR, Irish JD (eds) Anthropological perspectives on tooth morphology: genetics, evolution, variation. Cambridge University Press, Cambridge, pp 535–554Google Scholar
  53. 53.
    Stojanowski CM, Johnson KM (2015) Observer error, dental wear, and the inference of new world sundadonty. Am J Phys Anthr 156(3):349–362Google Scholar
  54. 54.
    Willermet CM, Edgar HJH (2009) Dental morphology and ancestry in Albuquerque, New Mexico Hispanics. HOMO - J Comp Hum Biol 60(3):207–224. Google Scholar
  55. 55.
    Scott GR, Navega D, Coelho J, Cunha E, Irish JD (2016) rASUDAS: a new method for estimating ancestry from tooth crown and root morphology. Am J Phys Anthr 159(S62):285–286Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of AnthropologyUniversity of NevadaRenoUSA
  2. 2.Department of AnthropologyMichigan State UniversityEast LansingUSA

Personalised recommendations