Ranking and Reputation Systems in the QBF Competition

  • Massimo Narizzano
  • Luca Pulina
  • Armando Tacchella
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4733)


Systems competitions play a fundamental role in the advancement of the state of the art in several automated reasoning fields. The goal of such events is to answer the question: “Which system should I buy?”. In this paper, we consider voting systems as an alternative to other procedures which are well established in automated reasoning contests. Our research is aimed to compare methods that are customary in the context of social choice, with methods that are targeted to artificial settings, including a new hybrid method that we introduce.


Social Choice Vote System Automate Reasoning Reputation System Aggregation Procedure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sutcliffe, G., Suttner, C.: The CADE ATP System Competition (April 22, 2007),
  2. 2.
    Le Berre, D., Simon, L.: The SAT Competition (April 22, 2007),
  3. 3.
    Long, D., Fox, M.: The 3rd International Planning Competition: Results and Analysis. Artificial Intelligence Research 20, 1–59 (2003)zbMATHCrossRefGoogle Scholar
  4. 4.
    van Dongen, M.R.C.: Introduction to the Solver Competition. In: CPAI 2005 proceedings (2005)Google Scholar
  5. 5.
    Barrett, C.W., de Moura, L., Stump, A.: SMT-COMP: Satisfiability Modulo Theories Competition. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 20–23. Springer, Heidelberg (2005)Google Scholar
  6. 6.
    Le Berre, D., Simon, L., Tacchella, A.: Challenges in the QBF arena: the SAT’03 evaluation of QBF solvers. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, Springer, Heidelberg (2004)Google Scholar
  7. 7.
    Le Berre, D., Narizzano, M., Simon, L., Tacchella, A.: The second QBF solvers evaluation. In: Hoos, H.H., Mitchell, D.G. (eds.) SAT 2004. LNCS, vol. 3542, Springer, Heidelberg (2005)Google Scholar
  8. 8.
    Narizzano, M., Pulina, L., Tacchella, A.: The third QBF solvers comparative evaluation. Journal on Satisfiability, Boolean Modeling and Computation 2, 145–164 (2006), available on-line at zbMATHGoogle Scholar
  9. 9.
    Hooker, J.N.: Testing Heuristics: We Have It All Wrong. Journal of Heuristics 1, 33–42 (1996)CrossRefGoogle Scholar
  10. 10.
    Saari, D.G.: Chaotic Elections! A Mathematician Looks at Voting. American Mathematical Society (2001)Google Scholar
  11. 11.
    Smith, W.D.: Range voting (April 22, 2007), available on-line at
  12. 12.
    Schulze, M.: A New Monotonic and Clone-Independent Single-Winner Election Method. Voting Matters, 9–19 (2003)Google Scholar
  13. 13.
    Narizzano, M., Pulina, L., Taccchella, A.: QBF solvers competitive evaluation (QBFEVAL),
  14. 14.
  15. 15.
    Arrow, K.J., Sen, A.K., Suzumura, K. (eds.): Handbook of Social Choice and Welfare, vol. 1. Elsevier, Amsterdam (2002)Google Scholar
  16. 16.
    Conitzer, V., Sandholm, T.: Common Voting Rules as Maximum Likelihood Estimators. In: EC-05. 6th ACM Conference on Electronic Commerce. LNCS, pp. 78–87. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  17. 17.
    Pulina, L.: Empirical Evaluation of Scoring Methods. In: Proc. STAIRS 2006. Frontiers in Artificial Intelligence and Applications, vol. 142, pp. 108–119 (2006)Google Scholar
  18. 18.
    Narizzano, M., Pulina, L., Tacchella, A.: Competitive Evaluation of QBF Solvers: noisy data and scoring methods. Technical report, STAR-Lab - University of Genoa (May 2006)Google Scholar
  19. 19.
    Giunchiglia, E., Narizzano, M., Pulina, L., Tacchella, A.: Quantified Boolean Formulas satisfiability library (QBFLIB),
  20. 20.
    Van Gelder, A., Le Berre, D., Biere, A., Kullmann, O., Simon, L.: Purse-Based Scoring for Comparison of Exponential-Time Programs (2006) (unpublished draft)Google Scholar
  21. 21.
    Narizzano, M., Pulina, L., Tacchella, A.: Competitive Evaluation of Automated Reasoning Tools: Statistical Testing and Empirical Scoring. In: EMAA 2006. First Workshop on Empirical Methods for the Analysis of Algorithms (2006)Google Scholar
  22. 22.
    Schulze, M.: Extending schulze’s method to obtain an overall ranking. Personal communicationsGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Massimo Narizzano
    • 1
  • Luca Pulina
    • 1
  • Armando Tacchella
    • 1
  1. 1.DIST, Università di Genova, Viale Causa, 13 – 16145 GenovaItaly

Personalised recommendations