Towards Automated Benchmarking and Evaluation of Heterogeneous Systems in Finance

  • Christian De SchryverEmail author
  • Carolina Pereira Nogueira


Benchmarking and fair evaluation of computing systems is a challenge for High Performance Computing (HPC) in general, and for financial systems in particular. The reason is that there is no optimal solution for a specific problem in most cases, but the most appropriate models, algorithms, and their implementations depend on the desired accuracy of the result or the input parameters, for instance. In addition, flexibility and development effort of those systems are important metrics for purchasers from the finance domain and thus need to be well-quantified. In this section we introduce a precise terminology for separating the problem, the employed model, and a solution that consists of a selected algorithm and its implementation. We show how the design space (the space of all possible solutions to a problem) can be systematically structured and explored. In order to evaluate and characterize systems independent of their underlying execution platforms, we illustrate the concept of application-level benchmarks and summarize the state-of-the-art for financial applications. In particular for heterogeneous and Field Programmable Gate Array (FPGA)-accelerated systems, we present a framework structure for automatically executing and evaluating such benchmarks. We describe the framework structure in detail and show how this generic concept can be integrated with existing computing systems. A generic implementation of this framework is freely available for download.


Monte Carlo Design Space Option Price Field Programmable Gate Array Software Architecture 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We gratefully acknowledge the partial financial support from the Center of Mathematical and Computational Modelling (CM)2 of the University of Kaiserslautern, from the German Federal Ministry of Education and Research under grant number 01LY1202D, and from the Deutsche Forschungsgemeinschaft (DFG) within the RTG GRK 1932 “Stochastic Models for Innovations in the Engineering Sciences”, project area P2. The authors alone are responsible for the content of this work. Furthermore, we thank Gordon Inggs from Imperial College London for his helpful inputs related to the current benchmark state and scheduler expertise.


  1. 1.
    Armstrong, B., Bae, H., Eigenmann, R., Saied, F., Sayeed, M., Zheng, Y.: HPC benchmarking and performance evaluation with realistic applications. In: SPEC Benchmarking Workshop, Austin (2006)Google Scholar
  2. 2.
    Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax (1998)Google Scholar
  3. 3.
    Berry, M., Chen, D., Koss, P., Kuck, D., Lo, S., Pang, Y., Pointer, L., Roloff, R., Sameh, A., Clementi, E., Chin, S., Schneider, D., Fox, G., Messina, P., Walker, D., Hsiung, C., Schwarzmeier, J., Lue, K., Orszag, S., Seidl, F., Johnson, O., Goodrum, R., Martin, J.: The perfect club benchmarks: effective performance evaluation of supercomputers. Int. J. High Perform. Comput. Appl. 3(3), 5–40 (1989)Google Scholar
  4. 4.
    Bondi, A.B.: Characteristics of scalability and their impact on performance. In: Proceedings of the 2nd International Workshop on Software and Performance (WOSP ’00), Ottawa, pp. 195–203. ACM, New York (2000)Google Scholar
  5. 5.
    Clark, D.D., Tennenhouse, D.L.: Architectural considerations for a new generation of protocols. In: Proceedings of the ACM Symposium on Communications Architectures & Protocols (SIGCOMM ’90), Philadelphia, pp. 200–208. ACM, New York (1990)Google Scholar
  6. 6.
    de Schryver, C.: Design methodologies for hardware accelerated heterogeneous computing systems. PhD thesis, University of Kaiserslautern (2014)Google Scholar
  7. 7.
    de Schryver, C., Jung, M., Wehn, N., Marxen, H., Kostiuk, A., Korn, R.: Energy efficient acceleration and evaluation of financial computations towards real-time pricing. In: König, A., Dengel, A., Hinkelmann, K., Kise, K., Howlett, R.J., Jain, L.C. (eds.) Knowledge-Based and Intelligent Information and Engineering Systems. Volume 6884 of Lecture Notes in Computer Science, pp. 177–186. Springer, Berlin/Heidelberg (2011). Proceedings of 15th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES)Google Scholar
  8. 8.
    de Schryver, C., Marxen, H., Schmidt, D.: Hardware accelerators for financial mathematics – methodology, results and benchmarking. In: Proceedings of the 1st Young Researcher Symposium (YRS) 2011, pp. 55–60. Center for Mathematical and Computational Modelling (CM)2, (CM)2 Nachwuchsring (2011). ISSN 1613-0073, urn:nbn:de:0074-750-0Google Scholar
  9. 9.
    de Schryver, C., Shcherbakov, I., Kienle, F., Wehn, N., Marxen, H., Kostiuk, A., Korn, R.: An energy efficient FPGA accelerator for Monte Carlo option pricing with the Heston model. In: Proceedings of the 2011 International Conference on Reconfigurable Computing and FPGAs (ReConFig), Cancun, Dec 2011, pp. 468–474 (2011)Google Scholar
  10. 10.
    Di Pierro, M.: web2py, 5th edn. Experts4Solutions (2013)Google Scholar
  11. 11.
    Fielding, R.T.: Architectural styles and the design of network-based software architectures. PhD thesis, University of California, Irvine (2000)Google Scholar
  12. 12.
    Fiksman, E., Salahuddin, S.: STAC-A2 on Intel architecture: from scalar code to heterogeneous application. In: Proceedings of the 7th Workshop on High Performance Computational Finance (WHPCF ’14), New Orleans, pp. 53–60 (2014). IEEE, PiscatawayGoogle Scholar
  13. 13.
    Haas, H., Brown, A.: Web services glossary. W3C note, W3C, February (2004)
  14. 14.
    Hull, J.C.: Options, Futures, And Other Derivatives, 8th edn. Pearson, Harlow (2012)Google Scholar
  15. 15.
    Inggs, G., Thomas, D., Luk, W.: A heterogeneous computing framework for computational finance. In: 2013 42nd International Conference on Parallel Processing (ICPP), Lyon, Oct 2013, pp. 688–697 (2013)Google Scholar
  16. 16.
    Jin, Q., Luk, W., Thomas, D.B.: On comparing financial option price solvers on FPGA. In: 2011 IEEE 19th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), Salt Lake City, May 2011, pp. 89–92 (2011)Google Scholar
  17. 17.
    Kienle, F., Wehn, N., Meyr, H.: On complexity, energy- and implementation-efficiency of channel decoders. IEEE Trans. Commun. 59(12), 3301–3310 (2011)Google Scholar
  18. 18.
    Lankford, P., Ericson, L., Nikolaev, A.: End-user driven technology benchmarks based on market-risk workloads. In: Proceedings of the 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), Salt Lake City, Nov 2012, pp. 1171–1175 (2012)Google Scholar
  19. 19.
    Marxen, H.: Aspects of the application of multilevel Monte Carlo methods in the Heston model and in a Lévy process framework. PhD thesis, University of Kaiserslautern (2012)Google Scholar
  20. 20.
    Marxen, H., Kostiuk, A., Korn, R., de Schryver, C., Wurm, S., Shcherbakov, I., Wehn, N.: Algorithmic complexity in the Heston model: an implementation view. In: Proceedings of the Fourth Workshop on High Performance Computational Finance (WHPCF ’11), Seattle, Nov 2011, pp. 5–12. ACM, New York (2011). ISBN 978-1-4503-1108-3Google Scholar
  21. 21.
    Morris, G.W., Aubury, M.: Design space exploration of the European option benchmark using hyperstreams. In: International Conference on Field Programmable Logic and Applications (FPL 2007), Aug 2007, Amsterdam, pp. 5–10 (2007)Google Scholar
  22. 22.
    Nikolaev, A., Burylov, I., Salahuddin, S.: Intel®; version of STAC-A2 benchmark: toward better performance with less effort. In: Proceedings of the 6th Workshop on High Performance Computational Finance (WHPCF ’13), Portland, pp. 7:1–7:7. ACM, New York (2013)Google Scholar
  23. 23.
    Nogueira, C.P.: Benchmark management of option pricer implementations with hardware accelaration based on the heston model. Bachelor’s thesis, Universidade Federal do Rio Grande do Sul, (2014)Google Scholar
  24. 24.
    Securities Technology Analysis Center LLC: STAC-A2 Central. (2012). Last access 14 Feb 2015
  25. 25.
    The University of Tennessee Knoxville: HPC challenge benchmark. Last access 14 Feb 2015
  26. 26.
    van der Steen, A.: The art of benchmarking HPC systems. (2009). Last access 14 Feb 2015

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Christian De Schryver
    • 1
    Email author
  • Carolina Pereira Nogueira
    • 1
  1. 1.Microelectronic Systems Design Research GroupUniversity of KaiserslauternKaiserslauternGermany

Personalised recommendations