Comparing Performance Heatmaps
The performance of parallel job schedulers is often expressed as an average metric value (e.g. response time) for a given average load. An alternative is to acknowledge the wide variability that exists in real systems, and use a heatmap that portrays the distribution of jobs across the performance \(\times \) load space. Such heatmaps expose a wealth of details regarding the conditions that occurred in production use or during a simulation. However, heatmaps are a visual tool, lending itself to high-resolution analysis of a single system but not conducive for a direct comparison between different schedulers or environments. We propose a number of techniques that allow to compare heatmaps. The first two treat the heatmaps as images, and focus on the differences between them. Two other techniques are based on tracking how specific jobs fare under the compared scenarios, and drawing underlying trends. This enables a detailed analysis of how different schedulers affect the workload, and what leads to the observed average results.
Many thanks to all those who have made their workload data available through the Parallel Workloads Archive.
- 1.Anscombe, F.J.: Graphs in Statistical Analysis. Am. Stat. 27(1), 17–21 (1973)Google Scholar
- 2.Feitelson, D.G.: Looking at data. In: 22nd International Parallel & Distributed Processing Symposium (IPDPS), April 2008Google Scholar
- 3.Chapin, S.J., Cirne, W., Feitelson, D.G., Jones, J.P., Leutenegger, S.T., Schwiegelshohn, U., Smith, W., Talby, D.: Benchmarks and standards for the evaluation of parallel job schedulers. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1999, IPPS-WS 1999, and SPDP-WS 1999. LNCS, vol. 1659, pp. 67–90. Springer, Heidelberg (1999) CrossRefGoogle Scholar
- 11.Parallel Workloads Archive. http://www.cs.huji.ac.il/labs/parallel/workload/
- 13.Performance Heatmap Utilities. https://bitbucket.org/krakov/heatmaps