Journal of Scientific Computing

, Volume 10, Issue 2, pp 233–269 | Cite as

Scalability of parallel spatial direct numerical simulations on intel hypercube and IBM SP1 and SP2

  • Ronald D. Joslin
  • Ulf R. Hanebutte
  • Mohammad Zubair


The implementation and performance of a parallel spatial direct numerical simulation (PSDNS) approach on the Intel iPSC/860 hypercube and IBM SP1 and SP2 parallel computers is documented. Spatially evolving disturbances associated with laminar-to-turbulent transition in boundary-layer flows are computed with the PSDNS code. The feasibility of using the PSDNS to perform transition studies on these computers is examined. The results indicate that PSDNS approach can effectively be parallelized on a distributed-memory parallel machine by remapping the distributed data structure during the course of the calculation. Scalability information is provided to estimate computational costs to match the actual costs relative to changes in the number of grid points. By increasing the number of processors, slower than linear speedups are achieved with optimized (machine-dependent library) routines. This slower than linear speedup results because the computational cost is dominated by FFT routine, which yields less than ideal speedups. By using appropriate compile options and optimized library routines on the SP1, the serial code achieves 52–56 Mflops on a single node of the SP1 (45 percent of theoretical peak performance). The actual performance of the PSDNS code on the SP1 is evaluated with a “real world” simulation that consists of 1.7 million grid points. One time step of this simulation is calculated on eight nodes of the SP1 in the same time as required by a Cray Y/MP supercomputer. For the same simulation, 32-nodes of the SP1 and SP2 are required to reach the performance of a Cray C-90. A 32 node SP1 (SP2) configuration is 2.9 (4.6) times faster than a Cray Y/MP for this simulation, while the hypercube is roughly 2 times slower than the Y/MP for this application.

Key words

Spatial direct numerical simulations incompressible viscous flows spectral methods finite differences parallel computing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Amdahl, G. (1967). Validity of the single-processor approach to achieving large scale computing capabilities.Proceedings of the AFIPS Conference, pp. 483–485.Google Scholar
  2. Bestek, H., Thumm, A., and Fasel, H. F. (1992). Numerical investigation of later stages of transition in transonic boundary layers. InFirst European Forum on Laminar Flow Technology, March 16–18, 1992. Hamburg, Germany.Google Scholar
  3. Bokhari, S. H. (1991). Complete Exchange on the iPSC/860. NASA CR 187498.Google Scholar
  4. Bushnell, D. M., Hefner, J. N., and Ash, R. L. (1977). Effect of compliant wall motion on turbulent boundary layers,Phys. Fluids 20(10), s31–48.Google Scholar
  5. Canuto, C., Hussaini, M. Y., Quarteroni, A., and Zang, T. A. (1988).Spectral Methods in Fluid Dynamics. Springer-Verlag, New York.Google Scholar
  6. Danabasoglu, G., Biringen, S., and Streett, C. L. (1990). Numerical simulation of spatially-evolving instability control in plane channel flow. AIAA Paper No. 90-1530.Google Scholar
  7. Danabasoglu, G., Biringen, S., and Streett, C. L. (1991). Spatial simulation of instability control by periodic suction blowing.Phys. Fluids A 3(9), 2138–2147.Google Scholar
  8. Dongarra, J. J., DuCroz, J., Hammarling, S., and Duff, I. (1990). A set of level 3 basic linear algebra subprograms.ACM Transactions on Mathematical Software 16(1), 18–28.Google Scholar
  9. Eidson, T. M., and Erlebacher, G. (1995). Implementation of a fully-balanced periodic tridiagonal solver on a parallel distributed memory architecture (to appear)Concurrency: Practice and Experience.Google Scholar
  10. Fasel, H. F. (1976). Investigation of the stability of boundary layers by a finite-difference model of the Navier-Stokes equations.J. Fluid Mech. 78, 355–383.Google Scholar
  11. Fasel, H. F., Rist, U., and Konzelmann, U. (1990). Numerical investigation of the three-dimensional development in boundary-layer transition.AIAA J. 28(1), 29–37.Google Scholar
  12. Fischer, P. F., Ho, L.-W., Karniadakis, G. E., Ronquist, E. M., and Patera, A. T. (1988). Recent advances in parallel spectral element simulation of unsteady incompressible flows.Computers and Structures 30(1–2), 217–231.Google Scholar
  13. Fischer, P. F., and Patera, A. T. (1994). Parallel simulation of viscous incompressible flows.Ann. Rev. Fluid. Mech. 26, 483–527.Google Scholar
  14. Gropp, W., Lusk, E., and Pieper, S. C. (1994). Users Guide for the Argonne National Laboratory IBM SP1. Argonne National Laboratory.Google Scholar
  15. Hanebutte, U. R., Joslin, R. D., and Zubair, M. (1994). Scalability of a parallel spatial direct numerical simulation Code on the IBM SP1 parallel supercomputer. NASA CR 194975.Google Scholar
  16. Henderson, R., and Karniadakis, G. E. (1991). Hybrid spectral-element-low-order methods for incompressible flows.J. Sci. Comp. 6(2), 79–99.Google Scholar
  17. Herbert, Th., and Bertolotti, F. P. (1987). Stability analysis of nonparallel boundary layers.Bull. Am. Phys. Soc. 32, 2079.Google Scholar
  18. IBM Parallel Programming Reference (1993). AIX Parallel Environment, Release 1.0. SH26-7228-00.Google Scholar
  19. IBM Guide and Reference (1992). Engineering and Scientific Subroutine Library, Version 2. SH23-0526-00.Google Scholar
  20. IBM Press Release (1994). Cornell Theory Center First To Receive IBMs Newest High Performance Power Parallel System. Obtained via World Wide Web, April 5, 1994.Google Scholar
  21. Jackson, E., She, Z.-S., and Orszag, S. A. (1991). A case study in parallel computing: I. Homogeneous turbulence on a hypercube,J. Sci. Comp. 6(1), 27–45.Google Scholar
  22. Joslin, R. D., Streett, C. L., and Chang, C.-L. (1992). Validation of three-dimensional incompressible spatial direct numerical simulation code—a comparison with linear stability and parabolic stability equations theories for boundary-layer transition on a flat plate. NASA TP 3205.Google Scholar
  23. Joslin, R. D., Streett, C. L., and Chang, C.-L. (1993). Spatial DNS of boundary-layer transition mechanisms: Validation of PSE theory.Theoret. Comput. Fluid Dynamics,4(6), 271–288.Google Scholar
  24. Joslin, R. D., and Streett, C. L. (1994). The role of stationary crossflow vortices in boundary-layer transition on swept wings.Phys. Fluids 6(10), 3442–3453.Google Scholar
  25. Joslin, R. D., and Zubair, M. (1993). Parallel spatial direct numerical simulations on the Intel iPSC/860 hypercube. NASA CR 191513.Google Scholar
  26. Kleiser, L., and Zang, T. A. (1991). Numerical simulation of transition in wall-bounded shear flows.Ann. Rev. Fluid Mech. 23, 495–537.Google Scholar
  27. Laurien, E., and Kleiser, L. (1989). Numerical simulation of boundary-layer transition and transition control.J. Fluid Mech. 199, 403–440.Google Scholar
  28. Lele, S. K. (1992). Compact finite difference schemes with spectral-like resolution.J. Comput. Phys. 103, 16–42.Google Scholar
  29. Lynch, R. E., Rice, J. R., and Thomas, D. H. (1964). Direct solution of partial difference equations by tensor product methods.Num. Math. 6, 185–199.Google Scholar
  30. Malik, M. R., and Li, F. (1992). Three-dimensional boundary layer stability and transition.Aerotech '92, Paper No. 91991.Google Scholar
  31. Moin, P., and Kim, J. (1982). Numerical investigation of turbulent channel flow.J. Fluid Mech. 118, 341–377.Google Scholar
  32. NAS: Numerical Aerodynamic Simulation (NAS) NASA Ames Research Center, Overview of the NAS SP2. Obtained via the World Wide Web, last updated Dec. 14, 1994.Google Scholar
  33. Ortega, J. M., and Voigt, R. G. (1988). A bibliography on parallel and vector numerical algorithms. ICASE Interim Report 6.Google Scholar
  34. Otto, J. C. (1993). Parallel execution of a three-dimensional, chemically reacting, Navier-Stokes code on distributed-memory machines. AIAA Paper 93-3307-CP.Google Scholar
  35. Piomelli, U., Zang, T. A., Speziale, C. G., and Hussaini, M. Y. (1990). On the large-eddy simulation of transitional wall-bounded flows.Phys. Fluids A 2(2), 257–265.Google Scholar
  36. Rai, M. M., and Moin, P. (1991a). Direct numerical simulation of transition and turbulence in a spatially-evolving boundary layer. AIAA Paper No. 91-1607.Google Scholar
  37. Rai, M. M., and Moin, P. (1991b). Direct numerical simulation of turbulent flow using finite-difference schemes.J. Comput. Phys. 96, 15–53.Google Scholar
  38. Reed, H. L. (1994). Direct numerical simulation of Transition: The spatial approach. InProgress is Transition Modelling, AGARD-R-793, 6.1-6.46.Google Scholar
  39. Saini, S. (1994). The IBM SP2: hardware, software, porting, and optimization overview. Numerical Aerodynamics Simulation Program, NASA Ames Research Center, NAS User Seminar, July 27, 1994.Google Scholar
  40. Smith, A. M. O., and Gamberoni, N. (1956). Transition, pressure gradients, and stability theory. Douglas Aircraft Company Report No. ES-26388.Google Scholar
  41. Spalart, P. R. (1989). Direct numerical study of leading-edge contamination. InFluid Dynamics of Three-Dimensional Turbulent Shear Flows and Transition, AGARD-CP-438, 5.1–5.13.Google Scholar
  42. Street, C. L., and Macaraeg, M. G. (1989). Spectral multi-domain for large-scale fluid dynamic simulations.Int. J. Appl. Numer. Math. 6, 123–140.Google Scholar
  43. Streett, C. L., and Hussaini, M. Y. (1991). A numerical simulation of the appearance of chaos in finite-length Taylor-Couette flow.Appl. Numer. Math. 7, 41–71.Google Scholar
  44. Sun, X.-H., and Zhu, J. (1994). Shared virtual memory and generalized speedup. NASA CR 191592.Google Scholar
  45. van Ingen, J. L. (1956). A suggested semi-empirical method for the calculation of the boundary-layer transition region. University of Delft Report VTH-74, Department of Aerospace Engineering, Delft, The Netherlands.Google Scholar
  46. Williamson, J. H. (1980). Low-storage Runge-Kutta schemes.J. Comput. Phys. 35(1), 48–56.Google Scholar
  47. Zang, T. A., and Hussaini, M. Y. (1987). Numerical simulation of nonlinear interactions in channel and boundary-layer transition.Nonlinear Wave Interactions in Fluids 87, 131–145.Google Scholar
  48. Zang, T. A., and Hussaini, M. Y. (1990). Multiple paths to subharmonic laminar breakdown in a boundary layer.Phys. Rev. Lett. 64, 641–644.Google Scholar

Copyright information

© Plenum Publishing Corporation 1995

Authors and Affiliations

  • Ronald D. Joslin
    • 1
  • Ulf R. Hanebutte
    • 2
  • Mohammad Zubair
    • 3
  1. 1.Flow Modeling and Control BranchNASA Langley Research CenterHampton
  2. 2.Reactor Analysis DivisionArgonne National LaboratoryArgonne
  3. 3.IBM Thomas J. Watson Research CenterYorktown Heights

Personalised recommendations