Overview of the S3 Contest: Performance Evaluation of Semantic Service Matchmakers



This chapter provides an overview of the organization and latest results of the international contest series on semantic service selection (S3). In particular, we introduce its publicly available S3 evaluation framework including the standard OWL-S and SAWSDL service retrieval test collections OWLS-TC and SAWSDL-TC as well as its retrieval performance evaluation tool SME2. Further, we classify and present representative examples of Semantic Web service matchmakers which participated in the S3 contest from 2007 to 2010. Eventually, we present and discuss selected results of the comparative experimental performance evaluation of all matchmakers that have been contested in the past editions of the S3 series.


Average Precision Service Request Service Selection Average Response Time Semantic Annotation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    S. Dietze, N. Benn, J. Domingue, A. Conconi, F. Cattaneo, Two-fold semantic web service matchmaking – applying ontology mapping for service discovery, in Proceedings of 4th Asian Semantic Web Conference, Shanghai, 2009Google Scholar
  2. 2.
    M. Dudev, P. Kapahnke, J. Misutka, M. Vasileski, M. Klusch, SME2: Semantic Service Matchmaker Evaluation Environment. Latest version 2.2 was released on Dec 2, 2010, at semwebcentral: http://projects.semwebcentral.org/projects/sme2/. First version of SME2 was released at semwebcentral on 17 Apr 2008. Online support for the SME2 tool is provided by Patrick Kapahnke DFKI (2007)
  3. 3.
    C. Kiefer, A. Bernstein, The creation and evaluation of iSPARQL strategies for matchmaking, in Proceedings of the 5th European Semantic Web Conference (ESWC), Tenerife, 2008Google Scholar
  4. 4.
    M. Klusch, Semantic web service description, in CASCOM – Intelligent Service Coordination in the Semantic Web, ed. by M. Schumacher, H. Helin, Chapter 3 (Birkhäuser Verlag, Springer, Basel, 2008)Google Scholar
  5. 5.
    M. Klusch, Semantic web service coordination, in CASCOM – Intelligent Service Coordination in the Semantic Web, ed. by M. Schumacher, H. Helin, Chapter 4 (Birkhäuser Verlag, Springer, Basel, 2008)Google Scholar
  6. 6.
    M. Klusch, K. Sycara, Brokering and matchmaking for coordination of agent societies: a survey, in Coordination of Internet Agents, ed. by A. Omicini et al., Chapter 8 (Springer, London, 2001)Google Scholar
  7. 7.
    M. Klusch, P. Kapahnke, OWLS-MX3: an adaptive hybrid semantic service matchmaker for OWL-S, in CEUR Proceedings of 3rd International Workshop on Semantic Matchmaking and Resource Retrieval (SMR2), Washington, 2009Google Scholar
  8. 8.
    M. Klusch, B. Fries, K. Sycara, OWLS-MX: a hybrid semantic web service matchmaker for OWL-S services. Web Semant. 7(2), 121–133 (2009), ElsevierGoogle Scholar
  9. 9.
    M. Klusch, P. Kapahnke, I. Zinnikus, Adaptive hybrid semantic selection of SAWSDL services with SAWSDL-MX2. Semant. Web Inf. Syst. 6(4), 1–26 (2011), IGI GlobalGoogle Scholar
  10. 10.
    OWLS-TC: OWL-S service retrieval test collection. Latest version OWLS-TC 4.0 (OWLS-TC4) published on 21 Sept 2010, at semwebcentral: http://projects.semwebcentral.org/projects/owls-tc/. First version of OWLS-TC was created by Benedikt Fries, Mahboob Khalid, Matthias Klusch (DFKI) and published at semwebcentral on 11 Apr 2005
  11. 11.
    P. Plebani, B. Pernici, URBE: web service retrieval based on similarity evaluation. IEEE Trans. Knowl. Data Eng. 21, 1629–1642 (2010)Google Scholar
  12. 12.
    SAWSDL-TC: SAWSDL service retrieval test collection. Latest version SAWSDL-TC 3.0 (SAWSDL-TC3) published on 22 Sept 2009, at semwebcentral: http://projects.semwebcentral.org/projects/sawsdl-tc/. First version of SAWSDL-TC was created by Patrick Kapahnke, Martin Vasileski, Matthias Klusch (DFKI) and published at semwebcentral on 28 July 2008
  13. 13.
    M.L. Sbodio, D. Martin, C. Moulin, Discovering semantic web services using SPARQL and intelligent agents. Web Semant. 8(4), 310–328 (2010), ElsevierGoogle Scholar
  14. 14.
    E. Toch, A. Gal, I. Reinhartz-Berger, D. Dori, A semantic approach to approximate service retrieval. ACM Trans. Internet Technol. (TOIT) 8(1), 2-es (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.German Research Center for Artificial Intelligence (DFKI)SaarbrueckenGermany

Personalised recommendations