ICIEIS 2011: Informatics Engineering and Information Science pp 189-203 | Cite as
Workflow Engine Performance Evaluation by a Black-Box Approach
Abstract
Workflow Management Systems (WfMSs) are complex software systems that require proper support in terms of WfMS performance. We propose here an approach to obtain some performance measurements for WfMSs (in order to compare them) by adopting a black box approach – an aspect that is not yet adequately studied in literature – and report some preliminary results: this allows us to evaluate at run-time the overall performance of a WfMS, comprising all of its constituent elements.
We set up two reference processes and four different experiments, to simulate real circumstances of load, ranging from one process instance to several process instances, entering the system either gradually or simultaneously. We identify some key performance indicators (CPU, main memory and disk workloads, and completion time) for the tests. We choose five WfMSs (some publicly available, some commercially available), and install them in their respective default configuration on five different and separate virtual machines (VMware). For every WfMS and for every experiment, we perform measurements and specifically focus on the completion time. Results enable us to measure how efficient the WfMSs are in general and how well they react to an increase of workload.
Keywords
Performance evaluation Workflow management system black-box approach virtual machinePreview
Unable to display preview. Download preview PDF.
References
- 1.Reijers, H.A., van der Aalst, W.M.P.: The effectiveness of workflow management systems - predictions and lessons learned. Journal of Information Management, 457–471 (2005)Google Scholar
- 2.van der Aalst, W.M.P., ter Hofstede, A.H.M., Kiepuszewski, B., Barros, A.P.: Workflow patterns. Distributed and Parallel Databases 14, 5–51 (2003)CrossRefGoogle Scholar
- 3.Li, J., Fan, Y., Zhou, M.: Performance modeling and analysis of workflow. IEEE Transactions on Systems, Man, and Cybernetics, Part A 34, 229–242 (2004)CrossRefGoogle Scholar
- 4.Reijers, H.A., Song, M.S., Jeong, B.: On the Performance of Workflow Processes with Distributed Actors: Does Place Matter? In: Alonso, G., Dadam, P., Rosemann, M. (eds.) BPM 2007. LNCS, vol. 4714, pp. 32–47. Springer, Heidelberg (2007)CrossRefGoogle Scholar
- 5.Kim, K.-H., Ellis, C.A.: Performance analytic models and analyses for workflow architectures. Information Systems Frontiers 3, 339–355 (2001)CrossRefGoogle Scholar
- 6.Casati, F., Ceri, S., Pernici, B., Pozzi, G.: Workflow Evolution. Data Knowl. Eng. 24, 211–238 (1998)CrossRefMATHGoogle Scholar
- 7.Kim, K.-H., Ahn, H.-J.: An EJB-Based Very Large Scale Workflow System and its Performance Measurement. In: Fan, W., Wu, Z., Yang, J. (eds.) WAIM 2005. LNCS, vol. 3739, pp. 526–537. Springer, Heidelberg (2005)CrossRefGoogle Scholar
- 8.Browne, J.C.: A critical overview of computer performance evaluation. In: ICSE, pp. 138–145 (1976)Google Scholar
- 9.Casati, F., Ceri, S., Paraboschi, S., Pozzi, G.: Specification and Implementation of Exceptions in Workflow Management Systems. ACM Trans. Database Syst. 24, 405–451 (1999)CrossRefGoogle Scholar
- 10.Casati, F., Pozzi, G.: Modeling Exceptional Behaviors in Commercial Workflow Management Systems. In: CoopIS, pp. 127–138. IEEE Computer Society (1999)Google Scholar
- 11.Gillmann, M., Weißenfels, J., Weikum, G., Kraiss, A.: Performance assessment and configuration of enterprise-wide workflow management systems. In: Dadam, P., Reichert, M. (eds.) Enterprise-wide and Cross-enterprise Workflow Management. CEUR Workshop Proceedings, vol. 24, pp. 18–24. CEUR-WS.org (1999)Google Scholar