Dynamic Load Balancing and Scheduling for Parallel Power System Dynamic Contingency Analysis

  • Siddhartha Kumar KhaitanEmail author
  • James D. McCalley
Part of the Power Systems book series (POWSYS)


Power system simulations involving solution of thousands of stiff differential and algebraic equations (DAE) are extremely computationally intensive and yet crucial for grid security and reliability. Online simulation of minutes to hours for a large number of contingencies requires computational efficiency several orders of magnitude greater than what is todays state-of-the-art. We have developed an optimized simulator for single contingency analysis using efficient numerical algorithms implementation for solving DAE, and scaled it up for large-scale contingency analysis using MPI. A prototype parallel high speed extended term simulator (HSET) on in-house high performance computing (HPC) resources at Iowa State University (ISU) (namely Cystorm Supercomputer) is being developed. Since the simulation times across contingencies vary considerably, we have focused our efforts towards development of efficient scheduling algorithms through work stealing for maximal resource utilization and minimum overhead to perform faster than real time analysis. This chapter introduces a novel implementation of dynamic load balancing algorithm for dynamic contingency analysis. Results indicate potential for significant improvements over the state-of-the-art methods especially master-slave based load balancing typically used in power system community. Simulations of thousands of contingencies on a large real system were conducted and computational savings and scalability results are reported.


Power System High Performance Computing Master Node Static Schedule Slave Node 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, A., Misra, S., Honbo, D., Choudhary, A.: Mpipairwisestatsig: Parallel pairwise statistical significance estimation of local sequence alignment. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 470–476. ACM (2010)Google Scholar
  2. 2.
    Agrawal, A., et al.: Parallel pairwise statistical significance estimation of local sequence alignment using message passing interface library. Concurrency and Computation: Practice and Experience (2011)Google Scholar
  3. 3.
    Arora, N., Blumofe, R., Plaxton, C.: Thread scheduling for multiprogrammed multiprocessors. In: Proceedings of the tenth Annual ACM Symposium on Parallel Algorithms and Architectures, pp. 119–129. ACM (1998)Google Scholar
  4. 4.
    Backschat, M., Pfaffinger, A., Zenger, C.: Economic-based dynamic load distribution in large workstation networks. In: Fraigniaud, P., Mignotte, A., Robert, Y., Bougé, L. (eds.) Euro-Par 1996. LNCS, vol. 1124, pp. 631–634. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  5. 5.
    Baldeschwieler, J., Blumofe, R., Brewer, E.: A tlas: an infrastructure for global computing. In: Proceedings of the 7th Workshop on ACM SIGOPS European Workshop: Systems Support for Worldwide Applications, pp. 165–172. ACM (1996)Google Scholar
  6. 6.
    Berger, N.: GPUs in experimental particle physics. Bulletin of the American Physical Society 57 (2012)Google Scholar
  7. 7.
    Blumofe, R., Joerg, C., Kuszmaul, B., Leiserson, C., Randall, K., Zhou, Y.: Cilk: An efficient multithreaded runtime system, vol. 30. ACM (1995)Google Scholar
  8. 8.
    Blumofe, R., Leiserson, C.: Scheduling multithreaded computations by work stealing. In: Proceedings. 35th Annual Symposium on Foundations of Computer Science, pp. 356–368. IEEE (1994)Google Scholar
  9. 9.
    Chen, Q., McCalley, J.: Operational defence of cascading sequences. In: 2011 IEEE Power and Energy Society General Meeting, pp. 1–8 (July 2011)Google Scholar
  10. 10.
    Chen, Y., Huang, Z., Chavarría-Miranda, D.: Performance evaluation of counter-based dynamic load balancing schemes for massive contingency analysis with different computing environments. In: 2010 IEEE Power and Energy Society General Meeting, pp. 1–6. IEEE (2010)Google Scholar
  11. 11.
    Collazuol, G., Lamanna, G., Pinzino, J., Sozzi, M.: Fast online triggering in high-energy physics experiments using GPUs. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment (2011)Google Scholar
  12. 12.
    Davis, T.: Algorithm 832: Umfpack v4. 3—an unsymmetric-pattern multifrontal method. ACM Transactions on Mathematical Software (TOMS) 30(2), 196–199 (2004)zbMATHCrossRefGoogle Scholar
  13. 13.
    Davis, T., Stanley, K.: Klu: a“ clark kent” sparse lu factorization algorithm for circuit matrices. In: SIAM Conference on Parallel Processing for Scientific Computing, PP 2004 (2004)Google Scholar
  14. 14.
    Dinan, J., Larkins, D., Sadayappan, P., Krishnamoorthy, S., Nieplocha, J.: Scalable work stealing. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, p. 53. ACM (2009)Google Scholar
  15. 15.
    Dinan, J., Olivier, S., Sabin, G., Prins, J., Sadayappan, P., Tseng, C.: Dynamic load balancing of unbalanced computations using message passing. In: IEEE International Parallel and Distributed Processing Symposium, IPDPS 2007, pp. 1–8. IEEE (2007)Google Scholar
  16. 16.
    Dobson, I., McCalley, J., Liu, C.: Fast simulation, monitoring and mitigation of cascading failure. Tech. rep., Power System Engineering Research Center (PSERC) (October 2010)Google Scholar
  17. 17.
    Gorton, I., Huang, Z., Chen, Y., Kalahar, B., Jin, S., Chavarria-Miranda, D., Baxter, D., Feo, J.: A high-performance hybrid computing approach to massive contingency analysis in the power grid. In: Fifth IEEE International Conference on e-Science 2009, pp. 277–283 (December 2009)Google Scholar
  18. 18.
    Green, R., Wang, L., Alam, M.: High performance computing for electric power systems: Applications and trends. In: 2011 IEEE Power and Energy Society General Meeting, pp. 1–8. IEEE (2011)Google Scholar
  19. 19.
    Green, R., Wang, L., Alam, M., Singh, C.: Intelligent and parallel state space pruning for power system reliability analysis using MPI on a multicore platform. In: 2011 IEEE PES Innovative Smart Grid Technologies (ISGT), pp. 1–8. IEEE (2011)Google Scholar
  20. 20.
    Guo, Y., Barik, R., Raman, R., Sarkar, V.: Work-first and help-first scheduling policies for async-finish task parallelism. In: IEEE International Symposium on Parallel & Distributed Processing, IPDPS 2009, pp. 1–12. IEEE (2009)Google Scholar
  21. 21.
    Hiraishi, T., Yasugi, M., Umatani, S., Yuasa, T.: Backtracking-based load balancing. In: ACM Sigplan Notices, vol. 44, pp. 55–64. ACM (2009)Google Scholar
  22. 22.
    Honbo, D., Agrawal, A., Choudhary, A.: Efficient pairwise statistical significance estimation using fpgas. In: Proceedings of BIOCOMP 2010, pp. 571–577 (2010)Google Scholar
  23. 23.
    Huang, Z., Chen, Y., Nieplocha, J.: Massive contingency analysis with high performance computing. In: IEEE Power and Energy Society General Meeting 2009. IEEE (July 2009)Google Scholar
  24. 24.
    Jacobsen, D., Thibault, J., Senocak, I.: An MPI-CUDA implementation for massively parallel incompressible flow computations on multi-GPU clusters. Mechanical and Biomedical Engineering Faculty Publications and Presentations, p. 5 (2010)Google Scholar
  25. 25.
    Jalili-Marandi, V., Zhou, Z., Dinavahi, V.: Large-scale transient stability simulation of electrical power systems on parallel GPUs. IEEE Transactions on Parallel and Distributed Systems 99, 1 (2011)Google Scholar
  26. 26.
    Khaitan, S., Fu, C., McCalley, J.: Fast parallelized algorithms for on-line extended-term dynamic cascading analysis. In: Power Systems Conference and Exposition, PSCE 2009. IEEE/PES, pp. 1–7. IEEE (2009)Google Scholar
  27. 27.
    Khaitan, S., McCalley, J.: A class of new preconditioners for linear solvers used in power system time-domain simulation. IEEE Transactions on Power Systems 25(4), 1835–1844 (2010)CrossRefGoogle Scholar
  28. 28.
    Khaitan, S., McCalley, J., Chen, Q.: Multifrontal solver for online power system time-domain simulation. IEEE Transactions on Power Systems 23(4), 1727–1737 (2008)CrossRefGoogle Scholar
  29. 29.
    Khaitan, S., McCalley, J., Raju, M.: Numerical methods for on-line power system load flow analysis. Energy Systems 1(3), 273–289 (2010)CrossRefGoogle Scholar
  30. 30.
    Kurowski, K., Kulczewski, M., Dobski, M.: Parallel and GPU based strategies for selected cfd and climate modeling models. Information Technologies in Environmental Engineering, 735–747 (2011)Google Scholar
  31. 31.
    Michael, M., Vechev, M., Saraswat, V.: Idempotent work stealing. In: ACM Sigplan Notices, vol. 44, pp. 45–54. ACM (2009)Google Scholar
  32. 32.
    Mielikainen, J., Huang, B., Huang, H., Goldberg, M.: GPU acceleration of the updated goddard shortwave radiation scheme in the weather research and forecasting (wrf) model. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 5(2), 555–562 (2012)CrossRefGoogle Scholar
  33. 33.
    Milshteyn, A., Alegre, A., Estrada, J., Lam, A., Beltran, S., Adigwu, J., Boussalis, H., Liu, C.: High-performance parallel processing aerospace information server. Journal of Next Generation Information Technology 1(3) (2010)Google Scholar
  34. 34.
    Pezzi, G., Cera, M., Mathias, E., Maillard, N.: On-line scheduling of MPI-2 programs with hierarchical work stealing. In: 19th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2007, pp. 247–254. IEEE (2007)Google Scholar
  35. 35.
    Schenk, O., Gärtner, K.: Solving unsymmetric sparse systems of linear equations with pardiso. Future Generation Computer Systems 20(3), 475–487 (2004)CrossRefGoogle Scholar
  36. 36.
    Serban, R., Petra, C., Hindmarsh, A.C.: User documentation for IDAS v1.0.0 (2009),
  37. 37.
    Srinivasa, A., Sosonkina, M., Maris, P., Vary, J.: Dynamic adaptations in ab-initio nuclear physics calculations on multicore computer architectures. In: 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW), pp. 1332–1339. IEEE (2011)Google Scholar
  38. 38.
    Tzannes, A., Caragea, G., Barua, R., Vishkin, U.: Lazy binary-splitting: a run-time adaptive work-stealing scheduler. In: ACM SIGPLAN Notices, vol. 45, pp. 179–190. ACM (2010)Google Scholar
  39. 39.
    Van Nieuwpoort, R., Kielmann, T., Bal, H.: Efficient load balancing for wide-area divide-and-conquer applications. In: ACM SIGPLAN Notices, vol. 36, pp. 34–43. ACM (2001)Google Scholar
  40. 40.
    Zhang, Y., Misra, S., Agrawal, A., Patwary, M., Liao, W., Qin, Z., Choudhary, A.: Accelerating pairwise statistical significance estimation for local alignment by harvesting GPU’s power. BMC Bioinformatics 13(suppl. 5), S3 (2012)Google Scholar
  41. 41.
    Zhang, Y., et al.: Efficient pairwise statistical significance estimation for local sequence alignment using GPU. In: 2011 IEEE 1st International Conference on Computational Advances in Bio and Medical Sciences (ICCABS), pp. 226–231. IEEE (2011)Google Scholar
  42. 42.
    Zhang, Y., et al.: Accelerating pairwise statistical significance estimation using NUMA machine. Journal of Computational Information Systems 8(9), 3887–3894 (2012)Google Scholar
  43. 43.
    Zhang, Y., Patwary, M.M.A., Misra, S., Agrawal, A., Liao, W.K., Choudhary, A.: Enhancing parallelism of pairwise statistical significance estimation for local sequence alignment. In: 2nd HiPC Workshop on Hybrid Multi-Core Computing, WHMC 2011, pp. 1–8 (2011)Google Scholar
  44. 44.
    Zhang, Y., et al.: Par-psse: Software for pairwise statistical significance estimation in parallel for local sequence alignment. International Journal of Digital Content Technology and its Applications (JDCTA) 6(5), 200–208 (2012)CrossRefGoogle Scholar
  45. 45.
    Zhou, K., Hou, Q., Ren, Z., Gong, M., Sun, X., Guo, B.: Renderants: interactive reyes rendering on GPUs. ACM Transactions on Graphics (TOG) 28, 155 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Electrical and Computer EngineeringIowa State UniversityIowaUSA

Personalised recommendations