• M. Brian Blake
  • Liliana Cabral
  • Birgitta König-Ries
  • Ulrich Küster
  • David Martin


This introduction will provide the necessary background on Semantic Web Services and their evaluation. It will then introduce SWS evaluation goals, dimensions and criteria and compare the existing community efforts with respect to these. This allows comprehending the similarities and differences of these complementary efforts and the motivation of their design.


Service Composition Test Collection Missing Element Code Inspection Evaluation Initiative 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    T. Berners-Lee, J. Hendler, O. Lassila, The semantic web. Sci. Am. 284(5), 34–43 (2001)Google Scholar
  2. 2.
    W. Beywl, Selected comments to the standards for evaluation of the german evaluation society – English edition. Technical report, German Evaluation Society (DeGEval), (2003)Google Scholar
  3. 3.
    M.B. Blake, H. Gomaa, Agent-oriented compositional approaches to services-based cross-organizational workflow. Decis. Support Syst. 40(1), 31–50 (2005)Google Scholar
  4. 4.
    L. Cabral, I. Toma, Evaluating semantic web services tools using the SEALS platform, in Proceedings of IWEST Workshop at ISWC 2010, Shanghai, 2010Google Scholar
  5. 5.
    J. Domingue, L. Cabral, S. Galizia, V. Tanasescu, A. Gugliotta, B. Norton, C. Pedrinaci, IRS-III: a broker-based approach to semantic web services. J. Web Semant. 6(2), 109–132 (2008)Google Scholar
  6. 6.
    N. Fenton, S.L. Pfleeger, R.L. Glass, Science and substance: a challenge to software engineers. IEEE Softw. 11(4), 86–95 (1994)Google Scholar
  7. 7.
    G. Gediga, K.-C. Hamborg, I. Düntsch, Evaluation of software systems, in Encyclopedia of Computer Science and Technology, vol. 45, ed. by A. Kent, J.G. Williams (Marcel Dekker, Inc., CRC, New York, 2002), pp. 127–153Google Scholar
  8. 8.
    D. Harman, Overview of the first Text REtrieval Conference (TREC-1), in Proceedings of the First Text REtrieval Conference (TREC-1), Gaithersbury, 1992Google Scholar
  9. 9.
    J. Kopecky, T. Vitvar, WSMO-Lite: lowering the semantic web services barrier with modular and light-weight annotations, in Proceedings of 2nd IEEE International Conference on Semantic Computing (ICSC), Santa Clara, CA, USA, 2008Google Scholar
  10. 10.
    U. Küster, An evaluation methodology and framework for semantic web services technology, University of Jena, Berlin Logos, 2010Google Scholar
  11. 11.
    U. Küster, B. König-Ries, Relevance judgments for web services retrieval-a methodology and test collection for sws discovery evaluation, in 2009 Seventh IEEE European Conference on Web Services (IEEE, Los Alamitos, 2009), pp. 17–26Google Scholar
  12. 12.
    U. Küster, B. König-Ries, C. Petrie, M. Klusch, On the evaluation of semantic web service frameworks. Int. J. Semant. Web Inf. Syst. 4(4), 31–55, (2008)Google Scholar
  13. 13.
    H. Lausen, C. Petrie, M. Zaremba, W3C SWS testbed incubator group charter (2007), Available online at
  14. 14.
    D. Martin, M. Burstein, D. McDermott, S. McIlraith, M. Paolucci, K. Sycara, D.L. McGuinness, E. Sirin, N. Srinivasan, Bringing semantics to web services with OWL-S. World Wide Web 10(3), 243–277 (2007)Google Scholar
  15. 15.
    C. Petrie, U. Küster, T. Margaria-Steffen, W3C SWS challenge testbed incubator methodology report. W3c incubator report, W3C, (2008), Available online at
  16. 16.
    W.F. Tichy, Should computer scientists experiment more? IEEE Comput. 31(5), 32–40 (1998)Google Scholar
  17. 17.
    M.V. Zelkowitz, D.R. Wallace, Experimental models for validating technology. Computer 31(5), 23–31 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • M. Brian Blake
    • 1
  • Liliana Cabral
    • 2
  • Birgitta König-Ries
    • 3
  • Ulrich Küster
    • 3
  • David Martin
    • 4
  1. 1.University of MiamiCoral GablesUSA
  2. 2.KMi, The Open UniversityMilton KeynesUK
  3. 3.Department of Computer ScienceUniversity JenaJenaGermany
  4. 4.Apple, Inc.CupertinoUSA

Personalised recommendations