Abstract
The complexity of modern data analysis techniques and the increasing amounts of data gushing out from neuroscientific experiments place new demands on the computing infrastructure required for data processing. The needs exceed the speed and memory constraints of a classical serial program design and require scientists to parallelize their analysis processes on distributed computer systems. In this chapter we explore, step by step, how to transform a typical data analysis program into a parallelized application. On the conceptual level, we demonstrate how to identify those parts of a serial program best suited for parallel execution. On the level of the practical implementation, we introduce four methods that assist in managing and distributing the parallelized code. By combining high-level scientific programming languages with modern techniques for job control and metaprogramming, no knowledge of system-level parallelization and the hardware architecture is required. We describe the solutions in a general fashion to facilitate the transfer of insights to the specific software and computer system environment of a particular laboratory.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Backofen R, Borrmann H-G, Deck W, Dedner A, De Raedt L, Desch K, Diesmann M, Geier M, Greiner A, Hess W, Honerkamp J, Jankowski S, Krossing I, Liehr A, Karwath A, Kloefkorn R, Pesche R, Potjans T, Roettger M, Schmiedt-Thieme L, Schneider G, Voss B, Wiebelt B, Wienemann P, Winterer V-H (2006) A bottom–up approach to grid-computing at a university: the black-forest-grid initiative. PIK 29(2):81–87
Bader DA (2008) Petascale computing: algorithms and applications. Chapman & Hall/CRC, Boca Raton
Butenhof DR (1997) Programming with POSIX threads. Addison-Wesley, Boston
Czarnecki K, Eisenecker U (2000) Generative programming: methods, tools, and applications. Addison-Wesley, Reading (illustrated ed)
Fliegner D, Grün S, Messer P, Diesmann M, Geisel M (2003). Distributed computing for neuroscience-data analysis. In: Proc. 5th Meeting German Neuroscience Society, pp 659–660
Foster I (1995) Designing and building parallel programs. Addison-Wesley, Reading
Foster I, Kesselman C (2004). The Grid: Blueprint for a New Computing Infrastructure, 2nd edn. Morgan-Kaufman, San Francisco
Gerstein G, Bloom MJ, Espinosa IE, Evanczuk S (1983) Design of a laboratory for multiunit studies. IEEE Trans Sys M Cybern 5:668–676
Kötter R (ed) (2009) Frontiers in Neuroscience special issue: Python in Neuroscience
Langtangen HP (2006) Python scripting for computational neuroscience, 2nd edn. Springer, Berlin
Miller R, Boxer L (2005) Algorithms sequential and parallel: a unified approach, 2nd edn. Charles River Media, Hingham
Morrison A, Mehring C, Geisel T, Aertsen A, Diesmann M (2005) Advancing the boundaries of high connectivity network simulation with distributed computing. Neural Comput 17(8):1776–1801
Newham C, Rosenblatt B (1995) Learning the bash shell, 1st edn. O’Reilly & Associates, Inc., Sebastopol
Oram A, Talbott S (1991) Managing projects with make, 2nd edn. O’Reilly & Associates, Inc., Sebastopol
Wilkinson B, Allen M (2004) Parallel programming: techniques and applications using networked workstations and parallel computers, 2nd edn. Prentice Hall, New York
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Denker, M., Wiebelt, B., Fliegner, D., Diesmann, M., Morrison, A. (2010). Practically Trivial Parallel Data Processing in a Neuroscience Laboratory. In: GrĂĽn, S., Rotter, S. (eds) Analysis of Parallel Spike Trains. Springer Series in Computational Neuroscience, vol 7. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-5675-0_20
Download citation
DOI: https://doi.org/10.1007/978-1-4419-5675-0_20
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-5674-3
Online ISBN: 978-1-4419-5675-0
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)