A Framework for Benchmarking BPMN 2.0 Workflow Management Systems
The diverse landscape of Workflow Management Systems (WfMSs) makes it challenging for users to compare different solutions to identify the ones most suitable to their requirements. Thus a comparison framework that would define common grounds in many different aspects, such as price, reliability, security, robustness and performance is necessary. In this paper we focus on the performance aspect, and we present a framework for automatic and reliable calculation of performance metrics for BPMN 2.0 WfMSs. We validate the framework by applying it on two open-source WfMSs. The goal is to contribute to the improvement of existing WfMSs by pinpointing performance bottlenecks, and to empower end users to make informed decisions when selecting a WfMS.
KeywordsBPMN 2.0 Workflow management systems Benchmarking
Unable to display preview. Download preview PDF.
- 1.Bianculli, D., Binder, W., et al.: Automated performance assessment for service-oriented middleware: a case study on BPEL engines. In: 19th WWW, pp. 141–150. ACM, Raleigh (2010)Google Scholar
- 2.Felter, W., Ferreira, A., Rajamony, R., Rubio, J.: An updated performance comparison of virtual machines and linux containers. Tech. rep, IBM, July 2014Google Scholar
- 3.Garro, J.M., Bazán, P.: Constructing and monitoring processes in BPM using hybrid architectures. IJACSA 3(4), 78–85 (2014)Google Scholar
- 5.Gray, J.: The Benchmark Handbook for Database and Transaction Systems, 2nd edn. Morgan Kaufmann, San Francisco (1992) Google Scholar
- 6.Harrer, S., Wirtz, G., Nizamic, F., Lazovik, A.: Towards a robustness evaluation framework for BPEL engines. In: 7th SOCA, pp. 199–206. IEEE, Matsue (2014)Google Scholar
- 7.Harrer, S., et al.: Automated and isolated tests for complex middleware products: the case of BPEL engines. In: 7th ICSTW, pp. 390–398. IEEE, Cleveland (2014)Google Scholar
- 8.IBM: IBM Rational Performance Tester. http://www.ibm.com/software/products/en/performance
- 9.Lazowska, E.D., et al.: Quantitative System Performance: Computer System Analysis Using Queueing Network Models. Prentice-Hall, Upper Saddle River (1984) Google Scholar
- 10.McGill, R., Tukey, J.W., Larsen, W.A.: Variations of box plots. The American Statistician 32(1), 12–16 (1978)Google Scholar
- 11.Molyneaux, I.: The Art of Application Performance Testing: From Strategy to Tools. O’Reilly Media Inc., Sebastopol (2014) Google Scholar
- 12.Röck, C., et al.: Performance benchmarking of BPEL engines: a comparisonframework, status quo evaluation and challenges. In: 26th SEKE, Pittsburgh(2014)Google Scholar
- 13.Skouradaki, M., Roller, D.H., et al.: On the road to benchmarking BPMN 2.0 workflow engines. In: 6th ICPE, pp. 301–304. ACM, Austin (2015)Google Scholar