Evaluating the agreement and reliability of a web-based facial analysis tool for rhinoplasty



Rhinoplasty is one of the most common and challenging plastic surgery procedures. Facial analysis is a crucial step in planning. Utilizing three-dimensional (3D) model of a patient’s face is an emerging way of performing facial analysis. This paper evaluates the agreement and reliability of facial measurements taken using a web app, located at digitized-rhinoplasty.com, that utilizes 3D models of the patient’s face.


Eleven measurements were calculated on 16 human subjects. Three methods of measurements were performed: direct measurements on human subjects’ faces, measurements on 2D photographs, and measurements on 3D models of face scans. The Bland–Altman plot is used for testing the agreement between the web app and the well-known Blender 3D modeling software. Intra-rater and inter-rater reliability was calculated and compared for 2D and 3D methods using the intraclass correlation coefficient (ICC) method. The statistical analysis methods were checked for the normality and homoscedasticity assumptions.


The results indicate that the web app and Blender software show agreement within 95% confidence limits. The web app performs well in intra-rater and inter-rater reliability statistical analysis. The web app’s reliability scores are consistently better than facial analysis software which was found highly reliable in a previous study. We also compare the methods of measurements in terms of time, ease of use, and cost.


The utilization of 3D computer modeling for facial analysis has its advantages and started to become more common due to recent advances in technology. The web app utilizes 3D face scans for pre-operative planning and post-operative evaluation of facial surgeries. The web app performs well in agreement and inter-/intra-reliability analysis and performs consistently better than software that works utilizing 2D photographs. The web app provides accurate, repeatable, affordable, and fast facial measurements for facial analysis when compared to direct and 2D methods.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    3dMDFace: https://3dmd.com/3dmdface/.

  2. 2.

    Canfield Vectra:https://www.canfieldsci.com/imaging-systems/.

  3. 3.

    10. Crisalix:https://crisalix.com.

  4. 4.

    Di3D: https://www.di4d.com.

  5. 5.

    Lifeviz Mini: https://www.quantificare.com/3d-photography-systems/lifeviz-mini/.

  6. 6.

    Face Analyzer: https://www.digitized-rhinoplasty.com/app/analyzer.html.

  7. 7.

    Intel Realsense: https://www.intelrealsense.com/depth-camera-d435i/.

  8. 8.

    Structure Sensor: https://structure.io/structure-sensor.

  9. 9.

    Using an iPhone 11 and Bellus3D FaceApp version


  1. 1.

    Amornvit P, Sanohkan S (2019) The accuracy of digital face scans obtained from 3d scanners: an in vitro study. Int J Environ Res Public Health 16:5061. https://doi.org/10.3390/ijerph16245061

    Article  PubMed Central  Google Scholar 

  2. 2.

    Apaydin F, Akyildiz S, Hecht DA, Toriumi DM (2009) Rhinobase: a comprehensive database, facial analysis, and picture-archiving software for rhinoplasty. Arch Facial Plast Surg 11(3):203–211. https://doi.org/10.1001/archfacial.2009.35

    Article  Google Scholar 

  3. 3.

    Baysal A, Sahan AO, Ozturk MA, Uysal T (2016) Reproducibility and reliability of three-dimensional soft tissue landmark identification using three-dimensional stereophotogrammetry. Angle Orthod 86(6):1004–1009. https://doi.org/10.2319/120715-833.1

    Article  PubMed  Google Scholar 

  4. 4.

    Bobak C, Barr P, O’Malley A (2018) Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales. BMC Med Res Methodol. https://doi.org/10.1186/s12874-018-0550-6

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Ceinos R, Tardivo D, Bertrand MF, Lupi-Pegurier L (2016) Inter- and intra-operator reliability of facial and dental measurements using 3d-stereophotogrammetry. J Esthet Restor Dent 28(3):178–189. https://doi.org/10.1111/jerd.12194

    Article  PubMed  Google Scholar 

  6. 6.

    Celikoyar MM, Perez MF, Akbas MI, Topsakal O (2021) Facial surface anthropometric features and measurements with an emphasis on rhinoplasty. Aesthetic Surg J. https://doi.org/10.1093/asj/sjab190

    Article  Google Scholar 

  7. 7.

    Community BO (2021) Blender—a 3d modelling and rendering package. http://www.blender.org. Accessed 21 Mar 2021

  8. 8.

    Dindaroglu F, Kutlu P, Duran G, Gorgulu S, Aslan E (2015) Accuracy and reliability of 3d stereophotogrammetry: a comparison to direct anthropometry and 2d photogrammetry. Angle Orthod. https://doi.org/10.2319/041415-244.1

    Article  PubMed  Google Scholar 

  9. 9.

    Dobratz E, Tran V, Hilger P (2010) Comparison of techniques used to support the nasal tip and their long-term effects on tip position. Arch Facial Plast Surg 12(3):172–179

    Article  Google Scholar 

  10. 10.

    Dogan N (2018) Bland-Altman analysis: a paradigm to understand correlation and agreement. Turk J Emerg Med. https://doi.org/10.1016/j.tjem.2018.09.001

    Article  PubMed  PubMed Central  Google Scholar 

  11. 11.

    Farkas L (1994) Examination. In: Anthropometry of the head and face, 2 edn. Raven Press, New York, pp 3–56 (1994)

  12. 12.

    George D, Mallery P (2009) SPSS for windows step by step: a simple study guide and reference, 17.0 Update, 10th edn. Allyn amp; Bacon, Inc., USA

  13. 13.

    Giavarina D (2015) Understanding bland altman analysis. Biochemia Medica. https://doi.org/10.11613/BM.2015.015

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Goffart Y (2010) Morphing in rhinoplasty: predictive accuracy and reasons for use. B-ENT 6(Suppl 15):13–19

    PubMed  Google Scholar 

  15. 15.

    Heike C, Cunningham M, Hing A, Stuhaug E, Starr J (2009) Picture perfect? Reliability of craniofacial anthropometry using three-dimensional digital stereophotogrammetry. Plast Reconstr Surg 124:1261–1272. https://doi.org/10.1097/PRS.0b013e3181b454bd

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Hong C, Choi K, Kachroo Y, Kwon T, Nguyen A, McComb R, Moon W (2017) Evaluation of the 3dmdface system as a tool for soft tissue analysis. Orthod Craniofac Res 20(S1):119–124

    Article  Google Scholar 

  17. 17.

    Kau C, Richmond S, Zhurov A, Knox J, Chestnutt I, Hartles F, Playle R (2005) Reliability of measuring facial morphology with a 3-dimensional laser scanning system. Am J Orthod Dentofac Orthop 128(4):424–430

    Article  Google Scholar 

  18. 18.

    Koo T, Li M (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15(2):155–163

    Article  Google Scholar 

  19. 19.

    Landers R (2015) Computing intraclass correlations (ICC) as estimates of interrater reliability in SPSS. Winnower. https://doi.org/10.15200/winn.143518

    Article  Google Scholar 

  20. 20.

    Lekakis G, Claes P, Hamilton GS, Hellings PW (2016) Evolution of preoperative rhinoplasty consult by computer imaging. Facial Plast Surg 32(1):80–87

    CAS  Article  Google Scholar 

  21. 21.

    Lekakis G, Hens G, Claes P, Hellings PW (2019) Three-dimensional morphing and its added value in the rhinoplasty consult. Plast Reconstruct Surg Glob Open 7:1

    Google Scholar 

  22. 22.

    Martin Bland J, Altman D (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 327(8476):307–310 (Originally published as Volume 1, Issue 8476)

  23. 23.

    Meruane M, Ayala M, Garcia-Huidobro M, Andrades P (2016) Reliability of nasofacial analysis using rhinobase software. Aesthetic Plast Surg 40:149–156. https://doi.org/10.1007/s00266-015-0569-6

    Article  PubMed  Google Scholar 

  24. 24.

    Mokkink L, Prinsen C, Bouter L, De Vet H, Terwee C (2016) The consensus-based standards for the selection of health measurement instruments (COSMIN) and how to select an outcome measurement instrument. Braz J Phys Ther 20:105–113

    Article  Google Scholar 

  25. 25.

    Othman SA, Ahmad R, Merican A, Jamaludin M (2013) Reproducibility of facial soft tissue landmarks on facial images captured on a 3d camera. Aust Orthod J 29:58–65

    PubMed  Google Scholar 

  26. 26.

    Persing S, Timberlake A, Madari S, Steinbacher D (2018) Three-dimensional imaging in rhinoplasty: a comparison of the simulated versus actual result. Aesthetic Plast Surg 42(5):1331–1335

    Article  Google Scholar 

  27. 27.

    Plooij J, Swennen G, Rangel F, Maal T, Schutyser F, Bronkhorst E, Kuijpers-Jagtman A, Berge S (2009) Evaluation of reproducibility and reliability of 3d soft tissue analysis using 3d stereophotogrammetry. Int J Oral Maxillofac Surg 38:267–273

    CAS  Article  Google Scholar 

  28. 28.

    Stralen K, Dekker F, Zoccali C, Jager K (2012) Measuring agreement, more complicated than it seems. Nephron Clin Pract 120:c162–c167. https://doi.org/10.1159/000337798

    Article  PubMed  Google Scholar 

  29. 29.

    Toma A, Zhurov A, Playle R, Ong E, Richmond S (2009) Reproducibility of facial soft tissue landmarks on 3d laser-scanned facial images. Orthod Craniofac Res 12(1):33–42

    CAS  Article  Google Scholar 

  30. 30.

    Topsakal O, Akbas IM, Demirel D, Nunez R, Simith B, Perez M, Celikoyar MM (2020) Digitizing rhinoplasty: a web application with three-dimensional preoperative evaluation to assist rhinoplasty surgeons with surgical planning. Int J Comput Assist Radiol Surg 15(11):1941–1950. https://doi.org/10.1007/s11548-020-02251-7

    Article  PubMed  Google Scholar 

  31. 31.

    Toriumi DM, Dixon TK (2011) Assessment of rhinoplasty techniques by overlay of before-and-after 3D images. Facial Plast Surg Clin North Am 19(4):711–723

    Article  Google Scholar 

  32. 32.

    Tzou CHJ, Artner NM, Pona I, Hold A, Placheta E, Kropatsch WG, Frey M (2014) Comparison of three-dimensional surface-imaging systems. J Plast Reconstr Aesthet Surg 67(4):489–497. https://doi.org/10.1016/j.bjps.2014.01.003

    Article  PubMed  Google Scholar 

  33. 33.

    Willaert RV, Opdenakker Y, Sun Y, Politis C, Vermeersch H (2019) New technologies in rhinoplasty: a comprehensive workflow for computer-assisted planning and execution. Plast Reconstruct Surg Glob Open 7:3

    Google Scholar 

  34. 34.

    Wong DJY, Oh DAK, Ohta DE, Hunt DAT, Rogers DGF, Mulliken DJB, Deutsch DCK (2008) Validity and reliability of craniofacial anthropometric measurement of 3d digital photogrammetric images. Cleft Palate Craniofac J 45(3):232–239. https://doi.org/10.1597/06-175 (PMID: 18452351)

    Article  PubMed  Google Scholar 

  35. 35.

    Zaki R, Bulgiba A, Nordin N, Ismail N (2012) Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: A systematic review. PLoS ONE 7(5):37908

    Article  Google Scholar 

  36. 36.

    Zaki R, Bulgiba A, Nordin N, Ismail N (2013) A systematic review of statistical methods used to test for reliability of medical instruments measuring continuous variables. Iran J Basic Med Sci 16:803–807

    PubMed  PubMed Central  Google Scholar 

Download references


We would like to thank Elif Topsakal for her help in statistical analysis. We would like to thank Julian Maniquis for helping with the agreement analysis using the Blender. We thank all the volunteers who took part in this study. We also would like to thank the reviewers of IJCARS for their valuable and constructive feedback on the manuscript.


This study is not funded by any grant.

Author information



Corresponding author

Correspondence to Oguzhan Topsakal.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

The study was approved by the Clinical Research Ethics Committee of Demiroglu Bilim University, Istanbul, Turkey (Decision number 03.03.2020/2020.05.03). Written informed consent for performing the direct measurements, 2D photography, 3D scans, data analysis, and publication of associated results was obtained beforehand from all volunteers.

Informed consent

In accordance with the provisions of the General Data Protection Regulation (EU) 2016/679, all subjects showed in the images and the proprietaries of all the personal data showed in this article gave their written consent to conduct its publication. All authors have participated in (a) conception and design, or analysis and interpretation of the data; (b) drafting the article or revising it critically for important intellectual content; and (c) approval of the final version. This manuscript has not been submitted to, nor is under review at, another journal or other publishing venue. The authors have no affiliation with any organization with a direct or indirect financial interest in the subject matter discussed in the manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Topsakal, O., Akbaş, M.İ., Smith, B.S. et al. Evaluating the agreement and reliability of a web-based facial analysis tool for rhinoplasty. Int J CARS 16, 1381–1391 (2021). https://doi.org/10.1007/s11548-021-02423-z

Download citation


  • Surgery planning
  • Facial analysis
  • Rhinoplasty
  • Agreement
  • Reliability
  • Facial measurements
  • 3D model
  • Intraclass correlation coefficient
  • Bland–Altman