Construction of the energy matrix for complex atoms

A parametric analysis of the hyperfine structure (hfs) for the even parity configurations of atomic terbium (Tb I) is presented in this work. We introduce the complete set of 4fN-core states in our high-performance computing (HPC) calculations. For calculations of the huge hyperfine structure matrix, requiring approximately 5000 hours when run on a single CPU, we propose the methods utilizing a personal computer cluster or, alternatively a cluster of Microsoft Azure virtual machines (VM). These methods give a factor 12 performance boost, enabling the calculations to complete in an acceptable time.


Introduction
The present paper is the eighth one in the series of our methodological approach regarding the atomic structure calculations. Six previously published papers, under the common title Construction of the energy matrix for complex atoms, contain a description of our method for semi-empirical analysis of complex electronic systems in multiconfiguration approximation up to the second order of the perturbation theory [1][2][3][4][5][6]. The seventh paper [7] of the above-mentioned cycle was the application of our many-body parametrization method to analyze fine structure in 4f-and 5f-shell atoms.
The choice of the investigated element was caused by a new experimental data of the hyperfine structure splitting of the atomic terbium levels obtained by our experimental group [8][9][10][11]. Within the work [10] the fine-and hyperfine structure analysis of seven even-parity configurations of Tb I (4f 8 5d 3 , 4f 8 5d 2 6s, 4f 8 5d6s 2 , 4f 9 6s6p, 4f 9 6s7p, 4f 9 6s8p, 4f 9 5d6p) was performed. The calculations were carried out in limited basis states, it means for 4f 8 -core to 13 terms and for 4f 9 -core to 3 terms.
After applying of novelty type of optimizing calculation procedures we were able to repeat the fine structure (fs) analysis for terbium atom in a full number of 4f 8 and 4f 9 -core states and the results were described in the aforementioned paper [7]. The huge energy matrix of the configuration system under consideration contained 74418 possible energy levels. The size of the largest submatrix, for J = 9/2, was 9936. The results of the fine structure calculations obtained within paper [7] compared to those published earlier [10] conducted with the restricted 4f N core, were definitely better. In the fine structure least-squares fit we achieved mean error for energy levels values of σ(E) = 37 cm −1 (old value was σ(E) = 56 cm −1 ). We used 99 known experimental even-parity energy levels and 26 fitted parameters. The description of the levels above 17000 cm −1 based on comparison with experimental g j -Landé factors seemed to be correctly determined. However, the confirmation of the obtained levels designation could be possible after the performing of the hyperfine structure parameterization in the same, complete basis states.
Therefore, in the current article we are reporting the results of the hyperfine structure analysis in the full number of 4f 8 and 4f 9 -core states, abandoning the previous limitations. Such huge hyperfine structure calculations have not been presented in literature so far. Computation of each J is assigned to a DS11v2 -2 cores -Azure VM in a way to minimize the total execution time, now dominated by the J = 9/2 submatrix. Running the calculations on 6 VMs 2 cores each resulted in 12 times performance boost. In this version, the electronic configurations are statically assigned to two threads. One thread calculates 4f 8 5d 2 6s, 4f 9 5d6p, the other one 4f 8 5d 3 , 4f 8 5d6s 2 , 4f 9 6s6p, 4f 9 6s7p, 4f 9 6s8p.
The next section of this paper contains the details of computational procedures optimization. The results of fineand hyperfine structure many-body parametrization method for the even configurations system of terbium atom are presented in sect. 3.

Computing hfs angular coefficients with Azure HPC infrastructure
At least two levels of parallel computation are possible and desired as the calculating angular coefficient is a time consuming process. One level is straightforward as the underlying Hamiltonian is block diagonal with respect to J. In consequence, the individual blocks can be processed independently. This independence allows distribution of the computation among multiple nodes as there are no interactions, except the final barrier when the calculations for all J complete. At J block level of parallelism, this property makes our computation location transparent as each J block can be assigned to a designated thread or to a separate node. Note, that distributing the computation among multiple nodes results in better scalability and requires only allocating an appropriate number of nodes (in our case six Azure VMs - fig. 1).
The second level of parallelism is more subtle and requires some care. There are multiple (i.e. 8 in the case of odd configurations of atomic terbium) configurations within each J which can also be computed concurrently, but now the dependencies between partial results are present as the parameters from all concurrently processed configurations must be ordered to produce properly structured data which is subsequently used as an input for hfs fitting. There are two phases of the calculations at the electronic configuration level: the first, more time-consuming phase computes the coefficients resulting from internal interactions, the second (much shorter) computes the inter-configuration part, hence a barrier must be present to co-ordinate the execution. When a thread calculating individual configuration reaches the barrier, it blocks until all other configurations complete and then the angular coefficients are ordered and the final phase, which calculate inter-configuration interactions is executed. As this final phase is relatively fast, it is assigned to a single thread.
The main factor affecting the computation time for a single configuration can be estimated from the rules of total angular momentum coupling which determine the base size. The other factor is the number of parameters which, however, is known in advance (this is simply our input). With these two factors available one can easily forecast the execution time needed to process each configuration and allocate the resources, assigning individual configurations to computational nodes in a way that results in an approximately optimal execution. Observe, that on a single CPU the total execution time would equal the sum of bars lengths in fig. 1. Despite this significant performance boost, in our future work we are going to present a solution that allows full scalability, i.e. the optimal utilization of an arbitrary number of cores.

Results of the semi-empirical approach
In our earlier paper on Tb I [10] we investigated the atomic structure of 7 even-parity configurations system using a semi-empirical parametrization method, taking into account electromagnetic interactions up to the second-order perturbation theory. The fine structure angular coefficient matrices for the multiconfiguration system, listed in the introduction, necessary for the least-squares fitting program, were constructed with the use of our computer code. The calculations were restricted to the lowest lying states of 4f 8 -and 4f 9 -core. The contributions from the secondorder perturbation theory concerning the configuration interaction (CI) effects, electrostatically correlated spin-orbit interactions (CSO), as well as electrostatically correlated hyperfine interactions (CHFS), described in the series [1][2][3][4][5][6], were possible to a limited extent only. The CI effects of two-electron excitations were included in the consideration by adding the term αL(L + 1) + βS(S + 1), according to [12,13] and [14]. For the CSO interactions only the excitations of one electron from closed n 0 d 10 shells into an open 5d-shell were taken into consideration. For the CHFS interactions, the excitations of one electron from closed n 0 s shells to empty n's shells or to an open 6s-shell were taken into account. A detailed description of our approach of the fine-and hyperfine structure analysis of Tb I was presented by us in sect. 4 in paper [10] and the final results of the semi-empirical calculations were summarized in four tables. The values of the intra-configuration and inter-configuration fs radial parameters were contained in tables 3 and 4, respectively. A comparison of the experimental and calculated energy values and hfs A and B constants were given in table 5. The values of the one-and two-body hyperfine structure parameters were presented in table 6. These values should be compared with new results obtained by the precise studies carried out within the framework of this work.
The current results of the semi-empirical fine-and hyperfine structure analysis of the even levels of the neutral terbium atom are presented in tables 1-5.
The values of radial fine structure parameters, their statistical errors and the values obtained with the Cowan code [15,16] (HFR) are given in tables 1 and 2. The second-order contributions concerning electrostatically correlated spin-orbit interactions were included according to the procedure described in the work [5]. This means that the excitations of one electron from open 4f 8 -, 4f 9 -shells to empty n f shells, the excitations of one electron from closed n 0 d 10 shells into an open 5d-shell and the excitations of one electron from closed n 0 p 6 shells into an open 6p-shell, were taken into account.
The comparison of the experimental and calculated energy values and hfs A and B constants is shown in table 3. The complete version of this table, together with the predictions of the energy values and hfs constants for the levels up to approximately 28000 cm −1 is presented in supplementary material associated with this paper.
In our procedure, we used all the experimental data known so far, i.e., the values of 99 energy electronic levels, g j -Landé factors known for 66 energy levels 86 A and 84 B hyperfine structure constants. The energy and g J values were taken from the NIST Atomic Spectra Database [17], which is based primarily on the monograph of Martin et al. [18]. The experimentally determined hfs constants were taken from Childs [19][20][21], Furmann [8,9] and Stefanska [10,22]. In the fs-fit with 369 parameters, 24 of which were treated as free, we achieved a mean-square deviation of 37 cm −1 .
The first three columns in the table 3 present the values of experimental, calculated energy of electronic levels and the difference between them in cm −1 . The two main fine structure components with their percentages, are given in columns 4-7. In next columns, the calculated g J values are compared with the experimental ones. In columns 10 and 12 the experimental hyperfine constants A and B are listed together with their experimental uncertainties. The calculated A and B constants for all energy levels are listed in columns 11 and 13. We achieved mean errors for A constants σ(A) = 17 MHz and for B constants σ(B) = 36 MHz, respectively.
The hfs constants A and B for the energy levels in region about 23000 cm −1 were very helpful in the identification of J-quantum numbers and assignment of the spectroscopic description. The semi-empirical calculations of the hyperfine constants A and B showed that it was possible to clarify the configuration and designation of the energy levels in a wide energy range.
The comparison of the experimental and calculated hfs A and B constants [MHz] of the even-parity levels obtained in full and limited number of 4f 8 and 4f 9 -core states is contain in table 4. Table 1. Values of the intra-configuration fine structure parameters (cm −1 ); ( * ) denotes an fixed parameter, a denotes arbitrarily assumed value of the center of gravity of the configuration.

Parameter
Value HFR even configurations In our published earlier paper [10], following Sandars and Back theory [23] we used the one-body radial parameters a κk nl , b κk nl , where κk = 01, 12 for magnetic-dipole hfs interactions and κk = 02, 11, 13 for electric-quadrupole hfs interactions. The contributions from the second-order perturbation theory, so-called electrostatically correlated hyperfine interactions, concerned with the excitations of one electron from closed shells to an open shell: n 0 s → 5d, n 0 d → 5d and from an open 4f-shell to empty n f shells dependent on κk = 01, 12 or 02 were omitted. The above restrictions were made due to the huge size of the hyperfine structure matrix. Only "hfs core-polarization effects", i.e. the influence of the excitations of electrons from closed n 0 s shells to empty n s shells or to an open 6s-shell on the hyperfine structure, were taken into account.
The optimization of our computer procedures for generating the angular coefficient of the hyperfine structure matrix presented within this work, allowed the quantitative determination of one-and two-body contributions to the hyperfine structure.
The values of the one-and two-body hyperfine structure parameters (MHz) and effective radial integrals (a.u.) obtained from the experimental data for the even parity configurations of Tb I are include in table 5. The ratio of the one-and two-body parameters κk = 12 and κk = 01 was assumed to amount to 1. For the parameters including electrostatic integrals of the order t = 4 the ratio in relation to corresponding t = 2 parameters were set to 0.65071 (from Hartree-Fock calculations [24]). The contributions originating on excitations from closed n 0 d shells to an open 5d-shell and from an open 4f-shell to empty n f shells dependent on κk = 01, 12 or 02 were specified. As we wrote in our earlier works [6,10,25], in Sandars and Beck theory [23] the operator s and the radial parameter a 10 nl (where l > 0) represent relativistic effects in the hyperfine structure. In our method we assume that the parameter a 10 nl for l > 0 is equal to zero. We can make this assumption because, according to, e.g., Feneuille and Armstrong [26], Armstrong [27] and Lindgren and Morrisson [28] the relativistic effects and configuration interaction effects, concerning the excitation of electrons from the closed shells to the empty shells, have the same angular part. Thus, the above mentioned effects are inseparable and is not possible to determine those values independently in the least-squares procedure by use a 10 nl radial parameter.       Table 5. Values of the one-and two-body hyperfine structure parameters (MHz) and effective radial integrals (a.u.) obtained from the experimental data for the even parity configurations of Tb i; OHFS stands for "optimized Hartree-Fock-Slater" method.