3D Research

, 4:3 | Cite as

On 3D object retrieval benchmarking

3DR Review
  • 284 Downloads

Abstract

The continuous evolution of 3D computer graphics and the progress of 3D digitization systems resulted in a continuous increase in the available 3D content. The widespread use of 3D objects in diverse domains contributed on forming 3D object retrieval as an active research field. In order to objectively evaluate the performance of retrieval methodologies there is a need for objective benchmarking schemes. In this work, we provide a comprehensive overview of the state-of-the-art evaluation methodologies including not only the performance measures but also the corresponding benchmark datasets. Meaningful benchmark datasets are discussed while a detailed list of publicly available 3D model repositories is given organized in terms of application domains, content magnitude and data types.

Keywords

3D object retrieval benchmark datasets performance evaluation measures 

References

  1. 1.
    Voorhees EM (2001) The philosophy of information retrieval evaluation, Lecture notes in Computer Science, 2406: 355–370. doi: 10.1007/3-540-45691-0_34.CrossRefGoogle Scholar
  2. 2.
    Buckland and Gey (1994) The relationship between recall and precision, JASIS, 45(1):12–19. doi:10.1002/(SICI)1097-4571(199401)45:1〈12::AID-ASI2〉3.0.CO;2-L.CrossRefGoogle Scholar
  3. 3.
    Zhou B, Yao Y (2010) Evaluating information retrieval system performance based on user preference, J Intelligent Information Systems, 34(3): 227–248. doi: 10.1007/s10844-009-0096-5.CrossRefGoogle Scholar
  4. 4.
    Shape Retrieval Contest (SHREC) (2006). http://www.aimatshape.net/event/SHREC. Accessed 11 July 2013.
  5. 5.
    Davis J, Goadrich M (2006) The relationship between precision-recall and ROC Curves, 23rd International Conference on Machine Learning, 25–29 June, Pittsburgh, Pennsylvania, 233–240.Google Scholar
  6. 6.
    Amato G, Savino P (2008) Approximate similarity in metric spaces using inverted files, 3rd Int. Conf. on scalable information systems, 4–6 July, Vico Equense, Italy, 1–10Google Scholar
  7. 7.
    Novotni M, Klein R (2004) Shape retrieval using 3D Zernike descriptors, Comp.-Aided Design, 36(11):1047–1062.doi:10.1016/j.cad.2004.01.005CrossRefGoogle Scholar
  8. 8.
    SHREC — Evaluation Methods (2009), http://www.itl.nist.gov/iad/vug/sharp/benchmark/shrecPartial/Evaluation.html Accessed 11 July 2013
  9. 9.
    Ishioka T (2004) Evaluation of criteria for information retrieval, System and Computers in Japan, 35(6):42–49.CrossRefGoogle Scholar
  10. 10.
    Babu K V S N J, Harshavardhan V, Kumar J S A (2012) The role of information retrieval in knowledge management, Int J Social Science and Interdisciplinary Research, 1(10):212–226.Google Scholar
  11. 11.
    Moreno AB, Sánchez A (2004) GavabDB: a 3D face database, Workshop on biometrics on the Internet COST, 25–26 March, Vigo, Spain, 275: 77–85.Google Scholar
  12. 12.
    Giorgi D, Biasotti S, Paraboschi L (2007) Shape retrieval contest 2007: Watertight models track, http://watertight.ge.imati.cnr.it/watertight-global.pdf. Accessed 11 July 2013Google Scholar
  13. 13.
    Jarvelin K, Kakelainen J (2002) Cumulated gain-based evaluation of IR techniques, ACM Trans. on Inf. Systems, 20(4):422–446. doi: 10.1145/582415.582418CrossRefGoogle Scholar
  14. 14.
    Zaharia T, Preteux F (2001) 3D shape-based retrieval within the MPEG-7 framework, SPIE Conference on Nonlinear Image Processing and Pattern Analysis XII, 2–6 February, San Francisco, California, 133–145Google Scholar
  15. 15.
    Shilane PN (2008) Shape Distinction for 3D Object Retrieval, University of Princeton.Google Scholar
  16. 16.
    Chen DY, Tian XP, Shen YT, Ouhyoung M (2003) On visual similarity based 3D model retrieval, Computer Graphics Forum, 22:223–232.doi:10.1111/1467-8659.00669CrossRefGoogle Scholar
  17. 17.
    NTU 3D Database: http://3d.csie.ntu.edu.tw/~dynamic/database (Accessed 10 July 2013).
  18. 18.
    Shilane P, Min P, Kazhdan M, Funkhouser T (2004) The Princeton shape benchmark, Shape Modeling International, 7–9 June, Genova, Italy, 167–178Google Scholar
  19. 19.
    Fang R, Godil A, Li X, Wagan A (2008) A new shape benchmark for 3D object retrieval, 4th International Symposium on Advances in Visual Computing, 1–3 December, Las Vegas, NV, 381–392Google Scholar
  20. 20.
    Porethi VT, Godil A, Dutagaci H, Furuya T, Lian Z, Ohbuchi R (2010) SHREC’10 Track: Generic 3D Warehouse, Eurographics Workshop on 3D Object Retrieva l, 2 May, Noprrkoping, Sweden, 93–100Google Scholar
  21. 21.
    Veltkamp RC, Giezeman GJ, Bast H et al (2010) SHREC’10 Track: Large scale retrieval, Eurographics Workshop on 3D Object Retrieval, 2 May, Noprrkoping, Sweden, 63–69Google Scholar
  22. 22.
    Tatsuma A, Koyanagi H, Aono M (2012) A large-scale shape benchmark for 3D object retrieval: Toyohashi Shape Benchmark, Signal and Information Processing Association, 3–6 December, Hollywood, CA 1–10Google Scholar
  23. 23.
    Axenopoulos et al (2009) SHREC 2009 — Shape retrieval contest of partial 3d models, Eurographics workshop on 3D object retrieval, 30 March, Munich, GermanyGoogle Scholar
  24. 24.
    Dutagaci H, Godil A, Cheung CP et al (2010) SHREC 2010 — Shape retrieval contest of range scans, Eurographics Workshop on 3D Object Retrieval, 2 May, Noprrkoping, Sweden, 109–115Google Scholar
  25. 25.
    SHREC — Shape retrieval contest of range scans (2011). http://www.itl.nist.gov/iad/vug/sharp/contest/2011/RangeScans. Accessed 13 July 2013
  26. 26.
    Lian Z, Godil A, Bustos B et al (2011) SHREC’11 Track: Shape retrieval on non-rigid 3D watertight meshes, 4th Eurographics conference on 3D Object Retrieval, 10 April, Llandudno, UK, 79–88Google Scholar
  27. 27.
    Zhang J, Siddiqi K, et al (2005) Retrieving articulated 3-d models using medial surfaces and their graph spectra, Int. Workshop On Energy Minimization Methods in Computer Vision and Pattern Recognition, 285–300CrossRefGoogle Scholar
  28. 28.
    Jayanti S, Kalyanaraman Y, Iyer N, Ramani K (2006) Developing an engineering shape benchmark for CAD models, Computer-Aided Design, 38(9): 939–953. doi:016/j.cad.2006.06.007CrossRefGoogle Scholar
  29. 29.
    National Design Repository (2013). http://edge.cs.drexel.edu/repository. Accessed 10 July 2013
  30. 30.
    Shape Retrieval Contest for CAD models (2008). https://engineering.purdue.edu/PRECISE/shrec08. Accessed 12 July 2013
  31. 31.
    Wessel R., Blumel I, Klein R (2009) A 3D shape benchmark for retrieval and automatic classification of architectural data, Eurographics Workshop on 3D Object Retrieval, 30 March–3 April, Munich, Germany, 53–56Google Scholar
  32. 32.
    Architectural Data Benchmark (2013). ftp://ftp.cg.cs.unibonn.de/pub/outgoing/ArchitectureBenchmark. Accessed 10 July 2013
  33. 33.
    SHREC — Architecture Benchmark (2010). http://givelab.cs.uu.nl/SHREC/shrec2010/arch.html. Accessed 10 July 2013
  34. 34.
    Zhang M, Zhang L, Mathiopoulos PT et al (2013) Perception-based shape retrieval for 3D building models, ISPRS J Photogrammetry Remote Sensing, 75:76–91. doi: 10.1016/j.isprsjprs.2012.10.001CrossRefGoogle Scholar
  35. 35.
    Goodall S, Lewis P H, Martinez K et al (2004) SCULPTEUR: Multimedia retrieval for museums, image and video retrieval, 21–23 July, Dublin, Ireland, 638–646Google Scholar
  36. 36.
    Gorisse D, Cord M, Jordan M et al (2007) 3D Content-Based Retrieval in Artwork Databases, IEEE 3DTV-Conference, 7–9 May, Kos Island, GreeceGoogle Scholar
  37. 37.
    Horr C, Brunnett G (2008) Similarity estimation on ancient vessels, GraphiCon, 23–27 June, Moscow State University, 94–100Google Scholar
  38. 38.
    Archaeological Heritage Service of Saxony (2013). http://sachsen.de/en/1826.htm. Accessed 10 July 2013
  39. 39.
    Koutsoudis A, Pavlidis G, Liami V et al (2010) 3D pottery content based retrieval based on pose normalisation and segmentation, J Cultural Herit, 11: 329–338. doi: 10.1016/j.culher.2010.02.002CrossRefGoogle Scholar
  40. 40.
    3D pottery content based retrieval benchmark dataset (2013). http://www.ipet.gr/~akoutsou/benchmark. Accessed 10 July 2013
  41. 41.
    Koutsoudis A, Pavlidis G, Arnaoutoglou F et al (2009) qp: A tool for generating 3D models of ancient Greek pottery, J Cultural Herit, 10: 281–295. doi: 10.1016/j.culher.2008.07.012CrossRefGoogle Scholar
  42. 42.
    Sfikas K, Pratikakis I, Koutsoudis A, Savelonas M, Theoharis T (2013), 3D Object partial matching using panoramic view, accepted in 2nd International Workshop on Multimedia for Cultural Heritage 2013, Naples, September 9, 2013Google Scholar
  43. 43.
    Temerinac M, Reisert M, Burkhardt H (2007) SHREC’07 Protein Retrieval Challenge, shape modelling international, 13–15 July, Lyon, FranceGoogle Scholar
  44. 44.
    Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces, SIGGRAPH’ 99, 8–13 August, Los Angeles, Califorinia, 187–194CrossRefGoogle Scholar
  45. 45.
    Blanz V, Vetter T (2003) Face recognition based on fitting a 3D morphable model, IEEE Trans on Pattern Analysis and Machine Intelligence, 25(9):1063–1074. doi: 10.1109/TPAMI.2003.1227983CrossRefGoogle Scholar
  46. 46.
    Veltkamp RC, van Jole S, Drira H et al (2011) SHREC’11 Track: 3D Face Models Retrieval, Eurographics Workshop on 3D Object Retrieval, 10 April, Llandudno, UK, 89–95Google Scholar
  47. 47.
    Ekman P, Friensen W (1978) Facial action coding system: A technique for the measurement of facial movement, Palo Alto: Consulting Psychologist Press.Google Scholar
  48. 48.
    Lijun Y, Xiaozhou W, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behaviour research, IEEE 7th Automatic Face and Gesture Recognition, 2–6 April, Southampton, UK, 211–216Google Scholar
  49. 49.
    3dMD (2013). http://www.3dmd.com. Accessed 10 July 2013
  50. 50.
    Lijun Y, Chen X, Sun Y, Worm T, Reale M (2008) A highresolution 3D dynamic facial expression database, IEEE Automatic Face & Gesture Recognition, 17–19 September, Binghamton, NY, 1–6.Google Scholar
  51. 51.
    Dimensional Imaging (2013). http://www.di3d.com. Accessed 10 July 2013.
  52. 52.
    Danelakis A, Theoharis T, Pratikakis I (2013) Facial Expression Recognition in 3D Video Sequences: A Survey, Preprint submitted to Computers & Graphics, January 31, 2012Google Scholar
  53. 53.
    Gkalelis N, Kim H, Hilton A et al (2009) The i3DPost multiview and 3D human action/interaction database, Visual Media Production, 12–13 November, 159–168Google Scholar
  54. 54.
    Cosker D, Krumhuber E, Hilton A (2011) A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modelling, 13th International Conference on Computer Vision, 6–13 November, Barcelona, SpainGoogle Scholar
  55. 55.
    Matuszewski B, Quan W, Shark L et al (2012) Hi4D-ADSIP 3D dynamic facial articulation database, Image and Vision Computing, 1–15. doi:10.1016/j.imavis.2012.02.002Google Scholar
  56. 56.
    SHREC — Watertight Models Track (2007). http://watertight.ge.imati.cnr.it. Accessed 11 July 2013
  57. 57.
    Sumner RW, Popovic J (2004) Deformation transfer for triangle meshes, ACM Trans on Graphics, 23:399–405. doi:10.1145/1015706.1015736CrossRefGoogle Scholar
  58. 58.
    Hartveldt J et al (2009) SHREC’09 Track: Structural shape retrieval on watertight models, Eurographics Workshop on 3D Object Retrieval, 30 March, Munich, GermanyGoogle Scholar
  59. 59.
    SHREC — Robustness benchmark (2010). http://tosca.cs.technion.ac.il/book/shrec_robustness2010.html. Accessed 11 July 2013
  60. 60.
    TOSCA (2013) Non-rigid shape comparison and analysis. http://tosca.cs.technion.ac.il. Accessed 11 July 2013Google Scholar
  61. 61.
    Sumner RW (2013) Mesh Data from deformation transfer for triangle meshes. http://people.csail.mit.edu/sumner/research/deftransfer/data.html. Accessed 11 July 2013Google Scholar
  62. 62.
    Papadakis P, Pratikakis I, Theoharis T, Perantonis S (2010) PANORAMA: A 3D shape descriptor based on panoramic views for unsupervised 3D object retrieval, Int J Computer Vision, Springer, 89(2): 177–192.CrossRefGoogle Scholar
  63. 63.
    SHREC — Correspondence benchmark (2010). http://tosca.cs.technion.ac.il/book/shrec_correspondence2010.html. Accessed 11 July 2013
  64. 64.
    SHREC — Correspondence benchmark (2011). http://tosca.cs.technion.ac.il/book/shrec_correspondence.html. Accessed 11 July 2013
  65. 65.
    SHREC — 3D-mesh segmentation track (2012). http://wwwrech.telecom-lille1.eu/shrec2012-segmentation. Accessed 11 July 2013
  66. 66.
    Benhabiles H, Vandeborre JP, Lavoué G, Daoudi A (2010) A comparative study of existing metrics for 3D-mesh segmentation evaluation, The Vis. Computer, 26(12): 1451–1466. doi:10.1007/s00371-010-0494-2CrossRefGoogle Scholar
  67. 67.
    Biasotti S, Bai X, Bustos B, Cerri A et al (2012) SHREC’12 track: Stability on abstract shapes, Eurographics Workshop on 3D Object Retrieval, 13 May, Cagliari, Italy, 101–107.Google Scholar
  68. 68.
    Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketchbased 3D model retrieval using diffusion tensor fields of suggestive contours, Int. conf. on Multimedia, October, Florence, Italy, 193–200.Google Scholar
  69. 69.
    Snodgrass J G, Vanderwart M (1980) A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity, J Experimental Psychology: Human Learning and Memory, 6(2):174–215. doi: 10.1037/0278-7393.6.2.174.CrossRefGoogle Scholar
  70. 70.
    SHREC — Shape Retrieval Contest based on Generic 3D Warehouse (2010). http://www.itl.nist.gov/iad/vug/sharp/contest/2010/Generic3DWarehouse. Accessed 10 July 2013.
  71. 71.
    Godil A, Lian Z, Dutagaci H, Fang R, Vanalami TP, Cheung CP (2010) Benchmarks, performance evaluation and contests for 3D shape retrieval, Performance metrics for intelligent systems workshop, 28–30 September, Baltimore, MD, 42–47CrossRefGoogle Scholar
  72. 72.
    SHREC — Shape Retrieval Contest of 3D Face Models (2007). http://give-lab.cs.uu.nl/SHREC/shrec2007. Accessed 10 July 2013.
  73. 73.
    SHREC — Robustness benchmark (2011). http://tosca.cs.technion.ac.il/book/shrec_robustness.html. Accessed 11 July 2013.
  74. 74.
    SHREC — Utrecht Large Scale 3D Shape Retrieval Benchmark (2010). http://givelab.cs.uu.nl/SHREC/ULS3SRB/2010. Accessed 11 July 2013.
  75. 75.
    Bustos B, Keim D, Saupe D, Schreck T (2007) Contentbased 3D object retrieval, IEEE Computer Graphics and Applications, 27(4):22–27. doi:10.1109/MCG.2007.80.CrossRefGoogle Scholar
  76. 76.
    Siddiqi K, Zhang J, Macrini D et al. (2008) Retrieving articulated 3D models using medial surfaces, Machine vision and applications, 19(4):261–274. doi:10.1007/s00138-007-0097-8.CrossRefGoogle Scholar
  77. 77.
    SHREC — Structural Shape Retrieval (2009). http://shrec.ge.imati.cnr.it/SHREC_2009_-_Structural_Shape_Retrieval/Home.html. Accessed 11 July 2013.
  78. 78.
    Toyohashi Shape Benchmark (2013). http://www.kde.cs.tut.ac.jp/benchmark/tsb. Accessed 11 July 2013.

Copyright information

© 3D Research Center, Kwangwoon University and Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Institute for Language and Speech Processing, Multimedia DepartmentAthena Research and Innovation CentreXanthiGreece
  2. 2.Department of Electrical and Computer EngineeringDemocritus University of ThraceXanthiGreece

Personalised recommendations