Introduction to Parallel Computation

Abstract

This chapter is intended to provide an overview of the fundamental concepts and ideas shaping the field of parallel computation. If serial (or sequential) algorithms are designed for the generic uni-processor architecture of the Random Access Machine (RAM), in the case of parallel algorithms there are a variety of models and architectures supporting the parallel mode of operation: shared-memory models, interconnection networks, combinational circuits, clusters and grids.

Sometimes, the methods used in designing sequential algorithms can also lead to efficient parallel algorithms, as it is the case with divide and conquer techniques. In other cases, the particularities of a certain model or architecture impose specific tools and methods that need to be used in order to fully exploit the potential offered by that model. In all situations, however, we seek an improvement either in the running time of the parallel algorithm or in the quality of the solution produced by the parallel algorithm with respect to the best sequential algorithm dealing with the same problem.

The improvement in performance can even become superlinear with respect to the number of processors employed by the parallel model under consideration. This is the case, for example, of computations performed under real-time constraints, when the deadlines imposed on the availability of the input and/or output data leave little room for sequentially simulating the parallel approach. Furthermore, in the examples presented at the end of the chapter, the impossibility to simulate a parallel solution on a sequential machine is due to the intrinsically parallel nature of the computation, rather than being an artifact of externally imposed time constraints.

In this respect, parallelism proves to be the vehicle leading to a Non-Universality result in computing: there is no finite computational device, sequential or parallel, conventional or unconventional, that is able to simulate all others.