Skip to main content
SpringerLink
Log in

Parallel Computing Techniques

  • Chapter
  • First Online:
Handbook of Computational Statistics

Part of the book series: Springer Handbooks of Computational Statistics ((SHCS))

Abstract

Parallel computing means to divide a job into several tasks and use more than one processor simultaneously to perform these tasks. Assume you have developed a new estimation method for the parameters of a complicated statistical model. After you prove the asymptotic characteristics of the method (for instance, asymptotic distribution of the estimator), you wish to perform many simulations to assure the goodness of the method for reasonable numbers of data values and for different values of parameters. You must generate simulated data, for example, 100,000 times for each length and parameter value. The total simulation work requires a huge number of random number generations and takes a long time on your PC. If you use 100 PCs in your institute to run these simulations simultaneously, you may expect that the total execution time will be 1100. This is the simple idea of parallel computing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Amdahl, G.M.: Validity of the single-processor approach to achieving large scale computing capabilities. In AFIPS Conference Proceedings, vol. 30, pp. 483–485 (1967)

    Google Scholar 

  • Balay, S., Buschelman, K., Gropp, W.D., Kaushik, D., Knepley, M., McInnes, L.C., Smith, B.F., Zhang, H.: PETSc home page (2001); http://www.mcs.anl.gov/petsc

  • Beddo, V.: Applications of parallel programming in Statistics. Ph.D. dissertation, University of California, Los Angeles (2002); http://theses.stat.ucla.edu/19/parallel_programming_beddo.pdf

  • Blackford, L., Choi, J., Cleary, A., D’Azevedo, E., Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R.C.: ScaLAPACK Users’ Guide. SIAM Press (1997)

    Google Scholar 

  • Bull, J.M., Riley, G.D., Rasbash, J., Goldstein, H.: Parallel implementation of a multilevel modelling package. Comput. Stat. Data Anal. 31(4), 457–474 (1999)

    Article  MATH  Google Scholar 

  • Butenhof, D.R.: Programming with POSIX Threads. Addison Wesley, Reading, MA, USA (1997)

    Google Scholar 

  • Chambers, J.M.: Programming with Data: A Guide to the S Language. Springer, Berlin (1998)

    MATH  Google Scholar 

  • Chandra, R., Dagum, L., Kohr, D., Maydan, D., McDonald, J., Menon, R. Parallel Programming in OpenMP. Morgan Kaufman, Los Altos, CA (2001)

    Google Scholar 

  • Christofides, A., Tanyi, B., Christofides, D., Whobrey, D., Christofides, N.: The optimal discretization of probability density functions. Comput. Stat. Data Anal. 31(4), 475–486 (1999)

    Article  MATH  Google Scholar 

  • Flynn, M.: Very high-speed computing systems. Proc. IEEE 54(12), 1901–1909 (1966)

    Article  Google Scholar 

  • Geist, A., Beguelin, A., Dongarra, J., Jiang, W., Manchek, R., Sunderam, V.S.: PVM: Parallel Virtual Machine: A Users’ Guide and Tutorial for Networked Parallel Computing. MIT Press, Cambridge, MA (1994)

    MATH  Google Scholar 

  • Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message-Passing Interface, 2nd Edition. MIT Press, Cambridge, MA (1999a)

    Google Scholar 

  • Gropp, W., Lusk, E., Thakur, R.: Using MPI-2: Advanced Features of the Message-Passing Interface. MIT Press, Cambridge, MA (1999b)

    Google Scholar 

  • Gustafson, J.L.: Reevaluating amdahl’s law. Comm. ACM 31(5), 532–533 (1988)

    Article  Google Scholar 

  • Hardwick, J., Oehmke, R., Stout, Q.F.: A program for sequential allocation of three bermoulli populations. Comput. Stat. Data Anal. 31(4), 397–416 (1999)

    Article  MATH  Google Scholar 

  • Hegland, M., McIntosh, I., Turlach, B.A.: A parallel solver for generalized additive models. Comput. Stat. Data Anal. 31(4), 377–396 (1999)

    Article  MATH  Google Scholar 

  • High Performance Fortran Forum: HPF: The high performance fortran home page (2004); http://www.crpc.rice.edu/HPFF/.

  • Jones, H., Mitra, G., Parkinson, D., Spinks, T.: A parallel implementation of the maximum likelihood method in positron emission tomography image reconstruction. Comput. Stat. Data Anal. 31(4), 417–439 (1999)

    Article  MATH  Google Scholar 

  • Koelbel, C.H., Loveman, D.B., Schreiber, R.S., Steele, J. G.L., Zosel, M.E.: The High Performance Fortran Handbook. MIT Press, Cambridge, MA (1993)

    Google Scholar 

  • LAM Team: LAM/MPI parallel computing (2004); http://www.lam-mpi.org/.

  • Li, N., Rossini, A.: RPVM: Cluster statistical computing in R. R News 1(3), 4–7 (2001); http://CRAN.R-project.org/doc/Rnews/.

  • Malard, J.M.: Parallel restricted maximum likelihood estimation for linear models with a dense exogenous matrix. Parall. Comput. 28(2), 343–353 (2002)

    Article  MATH  Google Scholar 

  • Mascagni, M.: SPRNG: A scalable library for pseudorandom number generation. In: Spanier, J. et al. (eds.) Proceedings of the Third International Conference on Monte Carlo and Quasi Monte Carlo Methods in Scientific Computing. Springer, Berlin (1999)

    Google Scholar 

  • Message Passing Interface (MPI) Forum: Message passing interface (MPI) forum home page (2004); http://www.mpi-forum.org/.

  • MPICH Team: MPICH – A portable mpi implementation (2004); http://www-unix.mcs.anl.gov/mpi/mpich/.

  • Murphy, K., Clint, M., Perrott, R.H.: Re-engineering statistical software for efficient parallel execution. Comput. Stat. Data Anal. 31(4), 441–456 (1999)

    Article  MATH  Google Scholar 

  • Nakano, J., Fujiwara, T., Yamamoto, Y., Kobayashi, I.: A statistical package based on Pnuts. In: Bethlehem, J.G., van der Heijden, P.G.M. (eds.) COMPSTAT 2000 Proceedings in Computational Statistics, pp. 361–366. Physica, Wurzburg (Wien) (2000)

    Google Scholar 

  • Oaks, S., Wong, H.: Java Threads. (2nd edn.), O’Reilly (1999)

    Google Scholar 

  • OpenMP Architecture Review Board: OpenMP: Simple, portable, scalable SMP programming (2004); http://www.openmp.org/.

  • PVM Project Members: PVM: Parallel virtual machine (2004); http://www.csm.ornl.gov/pvm/pvm_home.html.

  • Racine, J.: Parallel distributed kernel estimation. Comput. Stat. Data Anal. 40(2), 293–302 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  • Rossini, A., Tierney, L., Li, N.: Simple parallel statistical computing in R. UW Biostatistics working paper series, Paper 193, University of Washington, USA (2003); http://www.bepress.com/uwbiostat/paper193.

  • Schervish, M.J.: Applications of parallel computation to statistical inference. J. Aner. Statist. Assoc. 83, 976–983 (1988)

    Article  MATH  Google Scholar 

  • Sterling, T., Salmon, J., Becker, D.J., Savarese, D.F.: How to Build a Beowulf : A Guide to the Implementation and Application of PC Clusters. MIT Press, Cambridge, MA (1999)

    Google Scholar 

  • Swann, C.A.: Maximum likelihood estimation using parallel computing: An introduction to MPI. Comput. Econ. 19, 145–178 (2002)

    Article  MATH  Google Scholar 

  • Tanenbaum, A.S.: Modern Operating Systems. (2nd edn.), Prentice Hall, Netherland (2001)

    Google Scholar 

  • Temple Lang, D.: A multi-threaded extension to a high level interactive statistical computing environment. Ph.D. dissertation, University of California, Berkeley (1997); http://cm.bell-labs.com/stat/doc/multi-threaded-S.ps.

  • The R Development Core Team: The R project for statistical computing (2004); http://www.r-project.org/.

  • Tuomi, I.: The lives and death of moore’s law. First Monday 7(11), (2002); http://firstmonday.org/issues/issue7_11/tuomi/index.html.

  • van de Geijin, R.A.: Using PLAPACK. MIT Press, Cambridge, MA (1997)

    Google Scholar 

  • Yamamoto, Y., Nakano, J.: Distributed processing functions of a time series analysis system. J. Jpn. Soc. Comput. Stat. 15(1), 65–77 (2002)

    MathSciNet  Google Scholar 

  • Yu, H.: Rmpi: Parallel statistical computing in R. R News 2(2), 10–14 (2002); http://CRAN.R-project.org/doc/Rnews/.

Download references

Author information

Authors and Affiliations

  1. Department of Data Science, The Institute of Statistical Mathematics, Tachikawa, Tokyo, Japan

    Junji Nakano

Authors
  1. Junji Nakano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junji Nakano .

Editor information

Editors and Affiliations

  1. Dept. Computational & Data, Sciences, George Mason University, University Drive 4400, Fairfax, 22030-4444, Virginia, USA

    James E. Gentle

  2. L.v.Bortkiewicz Chair of Statistics, C.A.S.E. Centre f. Appl. Stat. & Econ., Humboldt-Universität zu Berlin, Unter den Linden 6, Berlin, 10099, Germany

    Wolfgang Karl Härdle

  3. Dept. Socioinformation, Okayama University, Ridai-cho 1-1, Okayama, 700-0005, Japan

    Yuichi Mori

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Nakano, J. (2012). Parallel Computing Techniques. In: Gentle, J., Härdle, W., Mori, Y. (eds) Handbook of Computational Statistics. Springer Handbooks of Computational Statistics. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21551-3_9

Download citation

Publish with us

Policies and ethics

Search

Navigation