Parallel Adaptive Cross Approximation for the Multi-trace Formulation of Scattering Problems
- 180 Downloads
Abstract
We present a highly parallel version of the boundary element method accelerated by the adaptive cross approximation for the efficient solution of scattering problems with composite scatterers. Individual boundary integral operators are treated independently, i.e. the boundary of every homogeneous subdomain is decomposed into clusters of elements defining a block structure of the local matrix. The blocks are distributed across computational nodes by a graph algorithm providing a load balancing strategy. The intra-node implementation further utilizes threading in shared memory and in-core SIMD vectorization to make use of all features of modern processors. The suggested approach is validated on a series of numerical experiments presented in the paper.
Keywords
Boundary element method Adaptive cross approximation Multi-trace formulation Distributed parallelization1 Introduction
Boundary integral equations present a valuable tool for the description of natural phenomena including wave scattering problems. Their numerical counterpart, the boundary element method (BEM), has become an alternative to volume based approaches such as finite element or finite volume methods. Except for a natural transition of the problem to the skeleton of the computational domain, BEM has found its place in HPC implementations of PDE solvers. One of the reasons is high computational intensity of system matrix assemblers, which fits well with today’s design of HPC clusters, where memory accesses are much more costly than arithmetic operations. To overcome the quadratic complexity of BEM, several methods have been proposed including the fast multipole [4, 15], or adaptive cross approximation (ACA) [2, 16, 17] methods, with the latter one utilized in this paper.
To describe wave scattering in a composite scatterer and its complement we utilize the local version of the multi-trace formulation (MTF) as introduced in [5, 6, 7] and presented here briefly in Sect. 2. The formulation leads to a block matrix with individual boundary integral operators for every homogeneous subdomain and coupling matrices on the off-diagonals. Although not described in detail in this paper, MTF allows for a natural operator preconditioning. In Sect. 3 we propose a strategy to parallelize the assembly of the MTF matrix blocks and their application in an iterative solver based on the approach presented in [11, 12, 13] for single domain problems. Except for the distributed parallelism, the method takes full advantage of the BEM4I library [14, 20, 21] and its assemblers parallelized in shared memory and vectorized by OpenMP. We provide the results of numerical experiments in Sect. 4.
2 Multi-trace Formulation
3 Parallel ACA
The matrices produced by a classical BEM are usually dense. Although the number of degrees of freedom is smaller compared to the volume-based methods (e.g., finite element method), some of the so-called fast BEM have to be applied in order to solve large-scale problems. These methods are usually based on a hierarchical clustering of the underlying mesh and approximations of matrix blocks corresponding to pairs of sufficiently separated clusters. Here we use the adaptive cross approximation (ACA) method since it is a purely algebraic approach and once properly implemented, it can be used for various kind of problems. Other approaches include the fast multipole method [9, 15] or the wavelet compression [8].
After hierarchical clustering, ACA proceeds by approximating sub-matrices corresponding to pairs of well-separated (admissible) clusters by a product of two low-rank matrices. Non-admissible clusters are assembled as dense matrices. Due to a limited scope of this paper, we refer the reader to [3, 16] for more details. The parallelization of the method based on the cyclic graph decomposition was presented in [13] where only certain special numbers of processors were discussed. In [12] we further extended the approach to support general number of processors. In the following section we recollect its basic principle and extend it to support the solution of MTF systems.
The extension of the proposed parallelization scheme to local multi-trace operators can be implemented in a straight-forward manner. Let \(\varOmega _{0}, \varOmega _{1}, \cdots , \varOmega _{m}\) denote the subdomains with their respective boundaries \(\varGamma _{0}, \varGamma _{1}, \cdots , \varGamma _{m}\). We apply our parallel ACA scheme to each subdomain individually, i.e. each \(\varGamma _{j}\) is split into P submeshes. The BEM operators \({{\mathsf {K}}}_{\kappa _j,h}, {{\mathsf {V}}}_{\kappa _j,h}, {{\mathsf {D}}}_{\kappa _j,h}\) are treated as \(P \times P\) block operators, each assembled in parallel according to the corresponding cyclic graph decomposition. This approach works reasonably well, however, distributing each boundary among the same number of processes may lead to a poor parallel efficiency in the case of \(\varGamma _j\) with small number of elements. Thus, the goal of our future research is to design an advanced parallel scheme for the assembly of local operators such that \({{\mathsf {K}}}_{\kappa _j,h}, {{\mathsf {V}}}_{\kappa _j,h}, {{\mathsf {D}}}_{\kappa _j,h}\) are assembled by various numbers of MPI processes.
4 Numerical Experiments
Our main interest lies in the study of the parallel matrix assembly and efficiency of the matrix-vector multiplication. The results of strong and weak scalability experiments are presented in Tables 1 and 2, respectively.
All the tests have been performed on a cubical geometry split into three parts, see Fig. 3 for a central cut of the domain and also the depiction of the resulting total field. The wave numbers are \(\kappa _{0}=4\), \(\kappa _{1}=2\), \(\kappa _{2}=5\), \(\kappa _{3}=3\) and \(\mu _{0}=\mu _{1}=\mu _{2}=\mu _{3}=1\). We used globally continuous piecewise linear trial and test functions for all operators included in the formulation. The parameters controlling the complexity of ACA were set to \(\varepsilon _{\mathrm {ACA}}=10^{-6}\), \(\mu _{\mathrm {ACA}} = 1.2\). The relative precision for the GMRES solver was set to \(\varepsilon _{\mathrm {GMRES}}=10^{-6}\).
The measured quantities are the time in seconds to assemble the matrices, the time to perform one matrix-vector multiplication without MPI synchronization, and also the overhead required to synchronize the results via MPI. We also present the efficiency of the parallel solver. We performed a series of five experiments and the presented values are the averages of the results.
we split the outer vector into 4 disjoint intervals (one for each subdomain),
then we perform all necessary computations for each interval followed by a non-blocking allreduce across all processes,
to facilitate the progress of the non-blocking reductions, we perform periodic calls to MPI_Test on the master OMP thread.
Strong scalability of operator assembly, matrix-vector multiplication and total runtime of the solver on \(1, 2, \dots , 64\) compute nodes (MPI synchronization included).
Matrix assembly | Matrix-vector multiply | Total time | |||||
---|---|---|---|---|---|---|---|
P | [s] | eff. [%] | apply [s] | sync. [s] | eff. [%] | [s] | eff. [%] |
2 | 627.1 | 100.0 | 2.55 | 0.010 | 100.0 | 1614.6 | 100.0 |
4 | 316.2 | 99.2 | 1.27 | 0.028 | 98.8 | 884.1 | 91.3 |
8 | 177.2 | 88.5 | 0.66 | 0.028 | 92.8 | 542.4 | 74.4 |
16 | 112.5 | 69.7 | 0.34 | 0.039 | 83.5 | 377.6 | 53.4 |
32 | 54.7 | 71.7 | 0.19 | 0.052 | 66.7 | 273.7 | 36.9 |
64 | 26.8 | 73.0 | 0.10 | 0.076 | 45.1 | 226.9 | 22.2 |
128 | 14.3 | 68.5 | 0.06 | 0.091 | 26.0 | 214.1 | 11.8 |
Efficiency of the weak scaling of operator assembly and matrix-vector multiplication on 1, 4, 16 compute nodes (MPI synchronization included).
Matrix assembly | Matrix-vector multiply | ||||
---|---|---|---|---|---|
P | t [s] | eff. [%] | apply t [s] | sync. t [s] | eff. [%] |
2 | 206.7 | 100.0 | 0.79 | 0.0069 | 100.0 |
8 | 247.9 | 94.0 | 1.11 | 0.0476 | 80.2 |
32 | 322.2 | 80.6 | 1.99 | 0.2080 | 49.6 |
5 Conclusion
We briefly presented a local multi-trace formulation for scattering problems with heterogeneous scatterers and applied the so-called cyclic graph decomposition to define a block-based workload distribution for distributed parallel matrix assembly and matrix-vector multiplication. We performed experiments on up to 64 compute nodes and presented strong and weak scaling properties of our approach. In our following work, we plan to refine and generalize the proposed methods to directly decompose the skeleton of the scatterer instead of the individual subdomains. We believe this could lead to a better scalability and ability to deal with problems with differently sized subdomains.
Notes
Acknowledgement
The authors acknowledge the support provided by The Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPS II) project ‘IT4Innovations excellence in science – LQ1602’ and from the Large Infrastructures for Research, Experimental Development and Innovations project ‘IT4Innovations National Supercomputing Center – LM2015070’. MK was further supported by the ESF in ‘Science without borders’ project, reg. nr. CZ.02.2.69/0.0./0.0./16_027/0008463 within the Operational Programme Research, Development and Education.
References
- 1.Amann, D.: Boundary element methods for Helmholtz transmission problems. Master’s thesis, TU Graz (2014)Google Scholar
- 2.Bebendorf, M.: Hierarchical Matrices: A Means to Efficiently Solve Elliptic Boundary Value Problems. Lecture Notes in Computational Science and Engineering. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-77147-0CrossRefzbMATHGoogle Scholar
- 3.Bebendorf, M., Rjasanow, S.: Adaptive low-rank approximation of collocation matrices. Computing 70(1), 1–24 (2003). https://doi.org/10.1007/s00607-002-1469-6MathSciNetCrossRefzbMATHGoogle Scholar
- 4.Chaillat, S., Collino, F.: A wideband fast multipole method for the Helmholtz kernel: theoretical developments. Comput. Math. Appl. 70(4), 660–678 (2015)MathSciNetCrossRefGoogle Scholar
- 5.Claeys, X., Dolean, V., Gander, M.: An introduction to multi-trace formulations and associated domain decomposition solvers. Appl. Numer. Math. 135, 69–86 (2019)MathSciNetCrossRefGoogle Scholar
- 6.Claeys, X., Hiptmair, R.: Multi-trace boundary integral formulation for acoustic scattering by composite structures. Commun. Pure Appl. Math. 66(8), 1163–1201 (2013)MathSciNetCrossRefGoogle Scholar
- 7.Claeys, X., Hiptmair, R., Jerez-Hancknes, C., Pintarelli, S.: Novel multi-trace boundary integral equations for transmission boundary value problems. In: Unified Transform for Boundary Value Problems: Applications and Advances, pp. 227–258. SIAM (2015)Google Scholar
- 8.Dahlke, S., Harbrecht, H., Utzinger, M., Weimar, M.: Adaptive wavelet BEM for boundary integral equations: theory and numerical experiments. Numer. Functional Anal. Optim. 39(2), 208–232 (2018)MathSciNetCrossRefGoogle Scholar
- 9.Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. J. Comput. Phys. 135(2), 280–292 (1997)MathSciNetCrossRefGoogle Scholar
- 10.Karypis, G., Kumar, V.: A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. 20(1), 359–392 (1999)MathSciNetCrossRefGoogle Scholar
- 11.Kravcenko, M., Maly, L., Merta, M., Zapletal, J.: Parallel assembly of ACA BEM matrices on Xeon Phi clusters. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds.) PPAM 2017. LNCS, vol. 10777, pp. 101–110. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-78024-5_10CrossRefGoogle Scholar
- 12.Kravčenko, M., Merta, M., Zapletal, J.: Distributed fast boundary element methods for Helmholtz problems. Appl. Math. Comput. 362, 124503 (2019)MathSciNetzbMATHGoogle Scholar
- 13.Lukáš, D., Kovář, P., Kovářová, T., Merta, M.: A parallel fast boundary element method using cyclic graph decompositions. Numer. Algorithms 70(4), 807–824 (2015)MathSciNetCrossRefGoogle Scholar
- 14.Merta, M., Zapletal, J.: BEM4I. IT4Innovations, VŠB - Technical University of Ostrava, 17. listopadu 2172/15, 708 00 Ostrava-Poruba, Czech Republic (2013). http://bem4i.it4i.cz/
- 15.Of, G.: Fast multipole methods and applications. In: Schanz, M., Steinbach, O. (eds.) Boundary Element Analysis. Lecture Notes in Applied and Computational Mechanics, vol. 29, pp. 135–160. Springer, Berlin Heidelberg (2007). https://doi.org/10.1007/978-3-540-47533-0_6CrossRefGoogle Scholar
- 16.Rjasanow, S., Steinbach, O.: The Fast Solution of Boundary Integral Equations. Mathematical and Analytical Techniques with Applications to Engineering. Springer, Heidelberg (2007). https://doi.org/10.1007/0-387-34042-4
- 17.Rjasanow, S., Weggler, L.: Matrix valued adaptive cross approximation. Math. Methods Appl. Sci. 40(7), 2522–2531 (2017)MathSciNetCrossRefGoogle Scholar
- 18.Sauter, S.A., Schwab, C.: Boundary Element Methods. Springer Series in Computational Mathematics. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-540-68093-2CrossRefzbMATHGoogle Scholar
- 19.Steinbach, O.: Numerical Approximation Methods for Elliptic Boundary Value Problems: Finite and Boundary Elements. Texts in Applied Mathematics. Springer, New York (2008). https://doi.org/10.1007/978-0-387-68805-3CrossRefzbMATHGoogle Scholar
- 20.Zapletal, J., Merta, M., Malý, L.: Boundary element quadrature schemes for multi- and many-core architectures. Comput. Math. Appl. 74(1), 157–173 (2017). 5th European Seminar on Computing ESCO 2016Google Scholar
- 21.Zapletal, J., Of, G., Merta, M.: Parallel and vectorized implementation of analytic evaluation of boundary integral operators. Engineering Analysis with Boundary Elements 96, 194–208 (2018)MathSciNetCrossRefGoogle Scholar
Copyright information
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.