Advertisement

Selection and Configuration of Parallel Portfolios

  • Marius Lindauer
  • Holger Hoos
  • Frank Hutter
  • Kevin Leyton-Brown
Chapter

Abstract

In recent years the availability of parallel computation resources has grown rapidly. Nevertheless, even for the most widely studied constraint programming problems such as SAT, solver development and applications remain largely focussed on sequential rather than parallel approaches. To ease the burden usually associated with designing, implementing and testing parallel solvers, in this chapter, we demonstrate how methods from automatic algorithm design can be used to construct effective parallel portfolio solvers from sequential components. Specifically, we discuss two prominent approaches for this problem. (I) Parallel portfolio selection involves selecting a parallel portfolio consisting of complementary sequential solvers for a specific instance to be solved (as characterised by cheaply computable instance features). Applied to a broad set of sequential SAT solvers from SAT competitions, we show that our generic approach achieves nearly linear speedup on application instances, and super-linear speedups on combinatorial and random instances. (II) Automatic construction of parallel portfolios via algorithm configuration involves a parallel portfolio of algorithm parameter configurations that is optimized for a given set of instances. Applied to gold-medal-winning parameterized SAT solvers, we show that our approach can produce significantly better-performing SAT solvers than state-of-the-art parallel solvers constructed by human experts, reducing time-outs by 17% and running time (PAR10 score) by 13% under competition conditions.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgement

M. Lindauer was supported by the DFG (German Research Foundation) under Emmy Noether grant HU 1900/2-1 and project SCHA 550/8-3, H. Hoos and K. Leyton-Brown by NSERC Discovery Grants, K. Leyton-Brown also by an NSERC E.W.R. Steacie Fellowship, and F. Hutter also by the DFG under Emmy Noether grant HU 1900/2-1.

References

  1. [1]
    Aigner, M., Biere, A., Kirsch, C., Niemetz, A., Preiner, M.: Analysis of portfolio-style parallel SAT solving on current multi-core architectures. In: Berre, D.L. (ed.) Proceedings of the Fourth Pragmatics of SAT workshop. EPiC Series in Computing, vol. 29, pp. 28–40. EasyChair (2014)Google Scholar
  2. [2]
    Alfonso, E., Manthey, N.: New CNF features and formula classification. In: Berre, D.L. (ed.) Proceedings of the Fifth Pragmatics of SAT workshop. EPiC Series in Computing, vol. 27, pp. 57–71. EasyChair (2014)Google Scholar
  3. [3]
    Alfonso, E., Manthey, N.: Riss 4.27 BlackBox. In: Belov, A., Diepold, D., Heule, M., Järvisalo, M. (eds.) Proceedings of SAT Competition 2014. Department of Computer Science Series of Publications B, vol. B-2014-2, pp. 68–69. University of Helsinki, Helsinki, Finland (2014)Google Scholar
  4. [4]
    Ansótegui, C., Malitsky, Y., Sellmann, M.: MaxSAT by improved instancespecific algorithm configuration. In: Brodley, C., Stone, P. (eds.) Proceedings of the Twenty-eighth National Conference on Artificial Intelligence (AAAI’14). pp. 2594–2600. AAAI Press (2014)Google Scholar
  5. [5]
    Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Gent, I. (ed.) Proceedings of the Fifteenth International Conference on Principles and Practice of Constraint Programming (CP’09). Lecture Notes in Computer Science, vol. 5732, pp. 142–157. Springer-Verlag (2009)Google Scholar
  6. [6]
    Audemard, G., Hoessen, B., Jabbour, S., Lagniez, J.M., Piette, C.: Penelope, a parallel clause-freezer solver. In: Balint et al. [10], pp. 43–44Google Scholar
  7. [7]
    Audemard, G., Simon, L.: Glucose 2.1. in the SAT challenge 2012. In: Balint et al. [10], pp. 23–23Google Scholar
  8. [8]
    Audemard, G., Simon, L.: Lazy clause exchange policy for parallel SAT solvers. In: Sinz, C., Egly, U. (eds.) Proceedings of the Seventeenth International Conference on Theory and Applications of Satisfiability Testing (SAT’14). Lecture Notes in Computer Science, vol. 8561, pp. 197–205. Springer (2014)Google Scholar
  9. [9]
    Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-Race algorithm: Sampling design and iterative refinement. In: Bartz-Beielstein, T., Aguilera, M., Blum, C., Naujoks, B., Roli, A., Rudolph, G., Sampels, M. (eds.) International Workshop on Hybrid Metaheuristics. Lecture Notes in Computer Science, vol. 4771, pp. 108–122. Springer (2007)Google Scholar
  10. [10]
    Balint, A., Belov, A., Diepold, D., Gerber, S., Järvisalo, M., Sinz, C. (eds.): Proceedings of SAT Challenge 2012: Solver and Benchmark Descriptions. University of Helsinki (2012)Google Scholar
  11. [11]
    Balint, A., Belov, A., Heule, M., Järvisalo, M. (eds.): Proceedings of SAT Competition 2013: Solver and Benchmark Descriptions, Department of Computer Science Series of Publications B, vol. B-2013-1. University of Helsinki (2013)Google Scholar
  12. [12]
    Belov, A., Diepold, D., Heule, M., Järvisalo, M.: The application and the hard combinatorial benchmarks in SAT competition 2014. In: Belov, A., Diepold, D., Heule, M., Järvisalo, M. (eds.) Proceedings of SAT Competition 2014. Department of Computer Science Series of Publications B, vol. B-2014-2, pp. 80–83. University of Helsinki, Helsinki, Finland (2014)Google Scholar
  13. [13]
    Biere, A.: Lingeling, Plingeling, PicoSAT and PrecoSAT at SAT race 2010. Tech. Rep. 10/1, Institute for Formal Models and Verification. Johannes Kepler University (2010)Google Scholar
  14. [14]
    Biere, A.: Lingeling and friends at the SAT competition 2011. Technical Report FMV 11/1, Institute for Formal Models and Verification, Johannes Kepler University (2011)Google Scholar
  15. [15]
    Biere, A.: Lingeling, Plingeling and Treengeling entering the SAT competition 2013. In: Balint et al. [11], pp. 51–52Google Scholar
  16. [16]
    Biere, A.: Lingeling and friends entering the SAT race 2015. Tech. rep., Institute for Formal Models and Verification, Johannes Kepler University (2015)Google Scholar
  17. [17]
    Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M., Malitsky, Y., Frechétte, A., Hoos, H., Hutter, F., Leyton-Brown, K., Tierney, K., Vanschoren, J.: ASlib: A benchmark library for algorithm selection. Artificial Intelligence 237, 41–58 (2016)Google Scholar
  18. [18]
    Brochu, E., Cora, V., de Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Computing Research Repository (CoRR) http://arXiv.org/abs/1012.2599 (2010)
  19. [19]
    Cameron, C., Hoos, H., Leyton-Brown, K.: Bias in algorithm portfolio performance evaluation. In: Kambhampati, S. (ed.) Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI’16). pp. 712–719. IJCAI/AAAI Press (2016)Google Scholar
  20. [20]
    Cheeseman, P., Kanefsky, B., Taylor, W.: Where the really hard problems are. In: Mylopoulos, J., Reiter, R. (eds.) Proceedings of the 12th International Joint Conference on Artificial Intelligence. pp. 331–340. Morgan Kaufmann (1991)Google Scholar
  21. [21]
    Chen, J.: Phase selection heuristics for satisfiability solvers. CoRR http://arXiv.org/abs/1106.1372(v1) (2011)
  22. [22]
    Cimatti, A., Sebastiani, R. (eds.): Proceedings of the Fifteenth International Conference on Theory and Applications of Satisfiability Testing (SAT’12), Lecture Notes in Computer Science, vol. 7317. Springer-Verlag (2012)Google Scholar
  23. [23]
    Falkner, S., Lindauer, M., Hutter, F.: SpySMAC: Automated configuration and performance analysis of SAT solvers. In: Heule, M., Weaver, S. (eds.) Proceedings of the Eighteenth International Conference on Theory and Applications of Satisfiability Testing (SAT’15). pp. 1–8. Lecture Notes in Computer Science, Springer-Verlag (2015)Google Scholar
  24. [24]
    Gebser, M., Kaufmann, B., Schaub, T.: Conflict-driven answer set solving: From theory to practice. Artificial Intelligence 187-188, 52–89 (2012)Google Scholar
  25. [25]
    Gebser, M., Kaufmann, B., Schaub, T.: Multi-threaded ASP solving with clasp. TPLP 12(4-5), 525–545 (2012)Google Scholar
  26. [26]
    van Gelder, A.: Contrasat - a contrarian SAT solver. Journal on Satisfiability, Boolean Modeling and Computation 8(1/2), 117–122 (2012)Google Scholar
  27. [27]
    Grinten, A., Wotzlaw, A., Speckenmeyer, E., Porschen, S.: satUZK: Solver description. In: Balint et al. [10], pp. 54–55Google Scholar
  28. [28]
    Hamadi, Y., Jabbour, S., Sais, L.: ManySAT: a parallel SAT solver. Journal on Satisfiability, Boolean Modeling and Computation 6, 245–262 (2009)Google Scholar
  29. [29]
    Hamadi, Y., Schoenauer, M. (eds.): Proceedings of the Sixth International Conference on Learning and Intelligent Optimization (LION’12), Lecture Notes in Computer Science, vol. 7219. Springer-Verlag (2012)Google Scholar
  30. [30]
    Harman, M., Jones, B.: Search-based software engineering. Information and Software Technology 43(14), 833–839 (2001)Google Scholar
  31. [31]
    Helmert, M., Röger, G., Karpas, E.: Fast Downward Stone Soup: A baseline for building planner portfolios. In: ICAPS-2011 Workshop on Planning and Learning (PAL). pp. 28–35 (2011)Google Scholar
  32. [32]
    Herwig, P.: Using graphs to get a better insight into satisfiability problems. Master’s thesis, Delft University of Technology, Department of Electrical Engineering, Mathematics and Computer Science (2006)Google Scholar
  33. [33]
    Heule, M., Dufour, M., van Zwieten, J., van Maaren, H.: March_eq: Implementing additional reasoning into an efficient look-ahead SAT solver. In: Hoos, H., Mitchell, D. (eds.) Proceedings of the Seventh International Conference on Theory and Applications of Satisfiability Testing (SAT’04). Lecture Notes in Computer Science, vol. 3542, pp. 345–359. Springer-Verlag (2004)Google Scholar
  34. [34]
    Holte, R., Howe, A. (eds.): Proceedings of the Twenty-second National Conference on Artificial Intelligence (AAAI’07). AAAI Press (2007)Google Scholar
  35. [35]
    Hoos, H.: Programming by optimization. Communications of the ACM 55(2), 70–80 (2012)Google Scholar
  36. [36]
    Hoos, H., Kaminski, R., Lindauer, M., Schaub, T.: aspeed: Solver scheduling via answer set programming. Theory and Practice of Logic Programming 15, 117–142 (2015)Google Scholar
  37. [37]
    Hoos, H., Lindauer, M., Schaub, T.: claspfolio 2: Advances in algorithm selectionfor answer set programming. Theory and Practice of Logic Programming 14, 569–585 (2014)Google Scholar
  38. [38]
    Hsu, E., Muise, C., Beck, C., McIlraith, S.: Probabilistically estimating backbones and variable bias: Experimental overview. In: Stuckey, P. (ed.) Proceedings of the Fourteenth International Conference on Principles and Practice of Constraint Programming (CP’08). Lecture Notes in Computer Science, vol. 5202, pp. 613–617. Springer (2008)Google Scholar
  39. [39]
    Huberman, B., Lukose, R., Hogg, T.: An economic approach to hard computational problems. Science 275, 51–54 (1997)Google Scholar
  40. [40]
    Hutter, F., Babi´c, D., Hoos, H., Hu, A.: Boosting verification by automatic tuning of decision procedures. In: O’Conner, L. (ed.) Formal Methods in Computer Aided Design (FMCAD’07). pp. 27–34. IEEE Computer Society Press (2007)Google Scholar
  41. [41]
    Hutter, F., Hoos, H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. In: Lodi, A., Milano, M., Toth, P. (eds.) Proceedings of the Seventh International Conference on Integration of AI and OR Techniques in Constraint Programming (CPAIOR’10). Lecture Notes in Computer Science, vol. 6140, pp. 186–202. Springer-Verlag (2010)Google Scholar
  42. [42]
    Hutter, F., Hoos, H., Leyton-Brown, K.: Bayesian optimization with censored response data. In: NIPS workshop on Bayesian Optimization, Sequential Experimental Design, and Bandits (BayesOpt’11) (2011)Google Scholar
  43. [43]
    Hutter, F., Hoos, H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C. (ed.) Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION’11). Lecture Notes in Computer Science, vol. 6683, pp. 507–523. Springer-Verlag (2011)Google Scholar
  44. [44]
    Hutter, F., Hoos, H., Leyton-Brown, K.: Parallel algorithm configuration. In: Hamadi and Schoenauer [29], pp. 55–70Google Scholar
  45. [45]
    Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research 36, 267–306 (2009)Google Scholar
  46. [46]
    Hutter, F., Hoos, H., Stützle, T.: Automatic algorithm configuration based on local search. In: Holte and Howe [34], pp. 1152–1157Google Scholar
  47. [47]
    Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H., Leyton-Brown, K.: The configurable SAT solver challenge (CSSC). Artificial Intelligence Journal (AIJ) 243, 1–25 (2017)Google Scholar
  48. [48]
    Hutter, F., Xu, L., Hoos, H., Leyton-Brown, K.: Algorithm runtime prediction: Methods and evaluation. Artificial Intelligence 206, 79–111 (2014)Google Scholar
  49. [49]
    Kadioglu, S., Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm selection and scheduling. In: Lee, J. (ed.) Proceedings of the Seventeenth International Conference on Principles and Practice of Constraint Programming (CP’11). Lecture Notes in Computer Science, vol. 6876, pp. 454–469. Springer-Verlag (2011)Google Scholar
  50. [50]
    Kadioglu, S., Malitsky, Y., Sellmann, M., Tierney, K.: ISAC - instance-specific algorithm configuration. In: Coelho, H., Studer, R., Wooldridge, M. (eds.) Proceedings of the Nineteenth European Conference on Artificial Intelligence (ECAI’10). pp. 751–756. IOS Press (2010)Google Scholar
  51. [51]
    Kotthoff, L.: Algorithm selection for combinatorial search problems: A survey. AI Magazine pp. 48–60 (2014)Google Scholar
  52. [52]
    Kotthoff, L.: Ranking algorithms by performance. In: Pardalos, P., Resende, M. (eds.) Proceedings of the Eighth International Conference on Learning and Intelligent Optimization (LION’14). Lecture Notes in Computer Science, Springer-Verlag (2014)Google Scholar
  53. [53]
    Kotthoff, L., Gent, I., Miguel, I.: An evaluation of machine learning in algorithm selection for search problems. AI Communications 25(3), 257–270 (2012)Google Scholar
  54. [54]
    Li, C.M., Wei, W., Li, Y.: Exploiting historical relationships of clauses and variables in local search for satisfiability. In: Cimatti and Sebastiani [22], pp. 479–480Google Scholar
  55. [55]
    Lindauer, M., Bergdoll, D., Hutter, F.: An empirical study of per-instance algorithm scheduling. In: Festa, P., Sellmann, M., Vanschoren, J. (eds.) Proceedings of the Tenth International Conference on Learning and Intelligent Optimization (LION’16). pp. 253–259. Lecture Notes in Computer Science, Springer-Verlag (2016)Google Scholar
  56. [56]
    Lindauer, M., Hoos, H., Hutter, F.: From sequential algorithm selection to parallel portfolio selection. In: Dhaenens, C., Jourdan, L., Marmion, M. (eds.) Proceedings of the Ninth International Conference on Learning and Intelligent Optimization (LION’15). pp. 1–16. Lecture Notes in Computer Science, Springer-Verlag (2015)Google Scholar
  57. [57]
    Lindauer, M., Hoos, H., Hutter, F., Schaub, T.: Autofolio: An automatically configured algorithm selector. Journal of Artificial Intelligence Research 53, 745–778 (Aug 2015)Google Scholar
  58. [58]
    Lindauer, M., Hoos, H., Leyton-Brown, K., Schaub, T.: Automatic construction of parallel portfolios via algorithm configuration. Artificial Intelligence 244, 272–290 (2017)Google Scholar
  59. [59]
    López-Ibáñez, M., Dubois-Lacoste, J., Caceres, L.P., Birattari, M., Stützle, T.: The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives 3, 43–58 (2016)Google Scholar
  60. [60]
    Malitsky, Y., Mehta, D., O’Sullivan, B.: Evolving instance specific algorithm configuration. In: Helmert, M., Röger, G. (eds.) Proceedings of the Sixth Annual Symposium on Combinatorial Search (SOCS). AAAI Press (2013)Google Scholar
  61. [61]
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Parallel SAT solver selection and scheduling. In: Milano, M. (ed.) Proceedings of the Eighteenth International Conference on Principles and Practice of Constraint Programming (CP’12). Lecture Notes in Computer Science, vol. 7514, pp. 512–526. Springer-Verlag (2012)Google Scholar
  62. [62]
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm portfolios based on cost-sensitive hierarchical clustering. In: Rossi, F. (ed.) Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI’13). pp. 608–614 (2013)Google Scholar
  63. [63]
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Parallel lingeling, CCASat, and CSCH-based portfolio. In: Balint et al. [11], pp. 26–27Google Scholar
  64. [64]
    Maratea, M., Pulina, L., Ricca, F.: A multi-engine approach to answer-set programming. Theory and Practice of Logic Programming 14, 841–868 (2014)Google Scholar
  65. [65]
    Nudelman, E., Leyton-Brown, K., Andrew, G., Gomes, C., McFadden, J., Selman, B., Shoham, Y.: Satzilla 0.9 (2003), solver description, International SAT CompetitionGoogle Scholar
  66. [66]
    Nudelman, E., Leyton-Brown, K., Hoos, H., Devkar, A., Shoham, Y.: Understanding random SAT: beyond the clauses-to-variables ratio. In: Wallace, M. (ed.) Proceedings of the international conference on Principles and Practice of Constraint Programming. Lecture Notes in Computer Science, vol. 3258, pp. 438–452. Springer (2004)Google Scholar
  67. [67]
    O’Mahony, E., Hebrard, E., Holland, A., Nugent, C., O’Sullivan, B.: Using case based reasoning in an algorithm portfolio for constraint solving. In: Bridge, D., Brown, K., O’Sullivan, B., Sorensen, H. (eds.) Proceedings of the Nineteenth Irish Conference on Artificial Intelligence and Cognitive Science (AICS’08) (2008)Google Scholar
  68. [68]
    Pulina, L., Tacchella, A.: A self-adaptive multi-engine solver for quantified boolean formulas. Constraints 14(1), 80–116 (2009)Google Scholar
  69. [69]
    Rice, J.: The algorithm selection problem. Advances in Computers 15, 65–118 (1976)Google Scholar
  70. [70]
    Roussel, O.: Description of ppfolio (2011), available at http://www.cril.univ-artois.fr/~roussel/ppfolio/solver1.pdf
  71. [71]
    Schneider, M., Hoos, H.: Quantifying homogeneity of instance sets for algorithm configuration. In: Hamadi and Schoenauer [29], pp. 190–204Google Scholar
  72. [72]
    Schubert, T., Lewis, M., Becker, B.: Pamiraxt: Parallel SAT solving with threads and message passing. JSAT 6(4), 203–222 (2009)Google Scholar
  73. [73]
    Seipp, J., Sievers, S., Helmert, M., Hutter, F.: Automatic configuration of sequential planning portfolios. In: Bonet, B., Koenig, S. (eds.) Proceedings of the Twenty-ninth National Conference on Artificial Intelligence (AAAI’15). AAAI Press (2015)Google Scholar
  74. [74]
    Soos, M., Nohl, K., Castelluccia, C.: Extending SAT solvers to cryptographic problems. In: Kullmann, O. (ed.) Proceedings of the Twelfth International Conference on Theory and Applications of Satisfiability Testing (SAT’09). Lecture Notes in Computer Science, vol. 5584, pp. 244–257. Springer (2009)Google Scholar
  75. [75]
    Streeter, M., Golovin, D., Smith, S.: Combining multiple heuristics online. In: Holte and Howe [34], pp. 1197–1203Google Scholar
  76. [76]
    Thornton, C., Hutter, F., Hoos, H., Leyton-Brown, K.: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: I. Dhillon, Koren, Y., Ghani, R., Senator, T., Bradley, P., Parekh, R., He, J., Grossman, R., Uthurusamy, R. (eds.) The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13). pp. 847–855. ACM Press (2013)Google Scholar
  77. [77]
    Tompkins, D., Balint, A., Hoos, H.: Captain Jack – new variable selection heuristics in local search for SAT. In: Sakallah, K., Simon, L. (eds.) Proceedings of the Fourteenth International Conference on Theory and Applications of Satisfiability Testing (SAT’11). Lecture Notes in Computer Science, vol. 6695, pp. 302–316. Springer (2011)Google Scholar
  78. [78]
    Wotzlaw, A., van der Grinten, A., Speckenmeyer, E., Porschen, S.: pfolioUZK: Solver description. In: Balint et al. [10], p. 45Google Scholar
  79. [79]
    Xu, L., Hoos, H., Leyton-Brown, K.: Hydra: Automatically configuring algorithms for portfolio-based selection. In: Fox, M., Poole, D. (eds.) Proceedings of the Twenty-fourth National Conference on Artificial Intelligence (AAAI’10). pp. 210–216. AAAI Press (2010)Google Scholar
  80. [80]
    Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research 32, 565–606 (2008)Google Scholar
  81. [81]
    Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In: RCRA workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI) (2011)Google Scholar
  82. [82]
    Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: Evaluating component solver contributions to portfolio-based algorithm selectors. In: Cimatti and Sebastiani [22], pp. 228–241Google Scholar
  83. [83]
    Xu, L., Hutter, F., Shen, J., Hoos, H., Leyton-Brown, K.: SATzilla2012: improved algorithm selection based on cost-sensitive classification models. In: Balint et al. [10], pp. 57–58Google Scholar
  84. [84]
    Yun, X., Epstein, S.: Learning algorithm portfolios for parallel execution. In: Hamadi and Schoenauer [29], pp. 323–338Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Marius Lindauer
    • 1
  • Holger Hoos
    • 2
    • 3
  • Frank Hutter
    • 1
  • Kevin Leyton-Brown
    • 2
  1. 1.University of FreiburgFreiburgGermany
  2. 2.University of British ColumbiaColumbiaCanada
  3. 3.Leiden UniversityLeidenNetherlands

Personalised recommendations