VCWC: A Versioning Competition Workflow Compiler

  • Günther Charwat
  • Giovambattista Ianni
  • Thomas Krennwallner
  • Martin Kronegger
  • Andreas Pfandler
  • Christoph Redl
  • Martin Schwengerer
  • Lara Katharina Spendier
  • Johannes Peter Wallner
  • Guohui Xiao
Conference paper

DOI: 10.1007/978-3-642-40564-8_23

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8148)
Cite this paper as:
Charwat G. et al. (2013) VCWC: A Versioning Competition Workflow Compiler. In: Cabalar P., Son T.C. (eds) Logic Programming and Nonmonotonic Reasoning. LPNMR 2013. Lecture Notes in Computer Science, vol 8148. Springer, Berlin, Heidelberg

Abstract

System competitions evaluate solvers and compare state-of-the-art implementations on benchmark sets in a dedicated and controlled computing environment, usually comprising of multiple machines. Recent initiatives such as [6] aim at establishing best practices in computer science evaluations, especially identifying measures to be taken for ensuring repeatability, excluding common pitfalls, and introducing appropriate tools. For instance, Asparagus [1] focusses on maintaining benchmarks and instances thereof. Other known tools such as Runlim (http://fmv.jku.at/runlim/) and Runsolver [12] help to limit resources and measure CPU time and memory usage of solver runs. Other systems are tailored at specific needs of specific communities: the not publicly accessible ASP Competition evaluation platform for the 3rd ASP Competition 2011 [4] implements a framework for running a ASP competition. Another more general platform is StarExec [13], which aims at providing a generic framework for competition maintainers. The last two systems are similar in spirit, but each have restrictions that reduce the possibility of general usage: the StarExec platform does not provide support for generic solver input and has no scripting support, while the ASP Competition evaluation platform has no support for fault-tolerant execution of instance runs.Moreover, benchmark statistics and ranking can only be computed after all solver runs for all benchmark instances have been completed.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Günther Charwat
    • 1
  • Giovambattista Ianni
    • 2
  • Thomas Krennwallner
    • 1
  • Martin Kronegger
    • 1
  • Andreas Pfandler
    • 1
  • Christoph Redl
    • 1
  • Martin Schwengerer
    • 1
  • Lara Katharina Spendier
    • 1
  • Johannes Peter Wallner
    • 1
  • Guohui Xiao
    • 1
  1. 1.Institute of Information SystemsVienna University of TechnologyViennaAustria
  2. 2.Dipartimento di Matematica e InformaticaUniversità della CalabriaRendeItaly

Personalised recommendations