Resilient Optimistic Termination Detection for the Async-Finish Model
- 573 Downloads
Driven by increasing core count and decreasing mean-time-to-failure in supercomputers, HPC runtime systems must improve support for dynamic task-parallel execution and resilience to failures. The async-finish task model, adapted for distributed systems as the asynchronous partitioned global address space programming model, provides a simple way to decompose a computation into nested task groups, each managed by a ‘finish’ that signals the termination of all tasks within the group.
For distributed termination detection, maintaining a consistent view of task state across multiple unreliable processes requires additional book-keeping when creating or completing tasks and finish-scopes. Runtime systems which perform this book-keeping pessimistically, i.e. synchronously with task state changes, add a high communication overhead compared to non-resilient protocols. In this paper, we propose optimistic finish, the first message-optimal resilient termination detection protocol for the async-finish model. By avoiding the communication of certain task and finish events, this protocol allows uncertainty about the global structure of the computation which can be resolved correctly at failure time, thereby reducing the overhead for failure-free execution.
Performance results using micro-benchmarks and the LULESH hydrodynamics proxy application show significant reductions in resilience overhead with optimistic finish compared to pessimistic finish. Our optimistic finish protocol is applicable to any task-based runtime system offering automatic termination detection for dynamic graphs of non-migratable tasks.
KeywordsAsync-finish Termination detection Resilience
We would like to thank David Grove and Olivier Tardieu of IBM T. J. Watson Research Center and Peter Strazdins of the Australian National University for their valuable comments on this work. The research used resources from the National Computational Infrastructure and NECTAR cloud, which are supported by the Australian Government.
- 1.Bland, W., Bouteiller, A., Herault, T., Hursey, J., Bosilca, G., Dongarra, J.J.: An evaluation of user-level failure mitigation support in MPI. In: Träff, J.L., Benkner, S., Dongarra, J.J. (eds.) EuroMPI 2012. LNCS, vol. 7490, pp. 193–203. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33518-1_24CrossRefGoogle Scholar
- 3.Cunningham, D., et al.: Resilient X10: efficient failure-aware programming. In: ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), pp. 67–80 (2014)Google Scholar
- 5.Kestor, G., Krishnamoorthy, S., Ma, W.: Localized fault recovery for nested fork-join programs. In: 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 397–408. IEEE (2017)Google Scholar
- 7.Lifflander, J., Miller, P., Kale, L.: Adoption protocols for fanout-optimal fault-tolerant termination detection. In: ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). ACM (2013)Google Scholar
- 8.Milthorpe, J., Grove, D., Herta, B., Tardieu, O.: Exploring the APGAS programming model using the LULESH proxy application. Technical report, RC25555, IBM Research (2015)Google Scholar
- 9.Stewart, R., Maier, P., Trinder, P.: Transparent fault tolerance for scalable functional computation. J. Funct. Program. 26 (2016)Google Scholar
- 10.The TLA Home Page. http://lamport.azurewebsites.net/tla/tla.html
- 11.TLA+ specification of the optimistic finish protocol and the replication protocol. https://github.com/shamouda/x10-formal-spec