# Solidification in a Supercomputer: From Crystal Nuclei to Dendrite Assemblages

- 4.8k Downloads
- 59 Citations

## Abstract

Thanks to the recent progress in high-performance computational environments, the range of applications of computational metallurgy is expanding rapidly. In this paper, cutting-edge simulations of solidification from atomic to microstructural levels performed on a graphics processing unit (GPU) architecture are introduced with a brief introduction to advances in computational studies on solidification. In particular, million-atom molecular dynamics simulations captured the spontaneous evolution of anisotropy in a solid nucleus in an undercooled melt and homogeneous nucleation without any inducing factor, which is followed by grain growth. At the microstructural level, the quantitative phase-field model has been gaining importance as a powerful tool for predicting solidification microstructures. In this paper, the convergence behavior of simulation results obtained with this model is discussed, in detail. Such convergence ensures the reliability of results of phase-field simulations. Using the quantitative phase-field model, the competitive growth of dendrite assemblages during the directional solidification of a binary alloy bicrystal at the millimeter scale is examined by performing two- and three-dimensional large-scale simulations by multi-GPU computation on the supercomputer, TSUBAME2.5. This cutting-edge approach using a GPU supercomputer is opening a new phase in computational metallurgy.

## Keywords

Graphic Processing Unit Grain Boundary Directional Solidification Solidification Microstructure Competitive Growth## Introduction

- (1)
The difficulty of direct (in situ) observation.

- (2)
The wide range of temporal and spatial scales.

- (3)
The need to consider multiple physics including fluid flow, thermal and solute diffusion.

Regarding the first difficulty, several pioneering works have achieved the in situ observation of solidification for transparent materials3,4 and for alloys by using synchrotron radiation x-rays.5, 6, 7 These studies provided considerable information on dendrite growth. However, it is not yet straightforward to directly observe the dynamics of solidification during actual production processes in general. Therefore, computational studies have contributed to clarifying the nature of solidification processes. In the early stage of research, Monte Carlo simulations with the Pott model8 were often performed to study the kinetics of grain growth,9,10 and this became a popular method for the study of solidification, recrystallization and other phenomena.11,12 The front tracking method13,14 and cellular automata15,16 have also been widely employed for simulations of dendritic growth. In 1993, Kobayashi succeeded in reproducing a complicated dendrite structure using the phase-field model.17 Since then, phase-field simulation18, 19, 20, 21, 22, 23 has become a major tool for the simulation of solidification. The main advantage of the phase-field model is that it is not necessary to explicitly track the position of a sharp interface in complex microstructural patterns. On the other hand, a long-standing issue regarding a quantitative aspect of the phase-field model remained unresolved until recently. This serious problem has been resolved by recent progress in the quantitative phase-field model.24, 25, 26, 27, 28, 29, 30, 31, 32, 33 Combined with the recent rapid progress in high-performance computational environments, large-scale phase-field simulations can now capture the competition between bundles of dendrites, including selection and regularity,34, 35, 36 which is filling the gaps in knowledge over a wide range of temporal and spatial scales. Moreover, the coupling of phase-field simulation with computational fluid dynamics is now capturing some aspects of multiple physics during solidification,37,38 which can be applied to fluid-dynamics-based phenomena such as the fragmentation of dendrite tips during solidification.38 The Lattice Boltzmann method39,40 is a promising numerical method for a large-scale fluid computation in solidification problem.

In combination with phase-field simulations, molecular dynamics simulations have contributed to the estimation of interfacial parameters such as the solid–liquid interfacial energy and the kinetic coefficient.41, 42, 43, 44, 45, 46, 47, 48, 49 In particular, the anisotropy in the interfacial parameters is a key factor determining the morphology of dendrite structures, although there are few reliable experimental values for the degree of anisotropy in the interfacial parameters. Therefore, an important role of molecular dynamics simulations is to provide interfacial parameters estimated from atomic-scale information for use in mesoscale simulations, which is a basic concept of multiscale modeling. Moreover, recent large-scale molecular dynamics simulations have captured the morphological dynamics of crystal growth with curved interfaces50, 51, 52 and multiple nucleation from an undercooled melt, which is followed by the formation of polycrystalline microstructures.53,54

Such progress in simulations of solidification is largely attributable to the rapid progress in high-performance computational environments. In particular, considerable benefit has been obtained from the high parallel efficiency of graphics processing units (GPUs). Large-scale simulations performed on GPU supercomputers have ranged from the nucleation and subsequent grain growth in a million-atom molecular dynamics simulation to the competitive growth of millimeter-size dendrite assemblages in a large-scale phase-field simulation. In this paper, cutting-edge simulations of solidification performed on a GPU supercomputer are introduced with a brief introduction to the current state of computational studies on solidification. Firstly, the spontaneous evolution of anisotropy in a solid nucleus investigated by million-atom molecular dynamics simulation is discussed in “Solidification in Large-Scale Molecular Dynamics Simulation” section. Investigation of the nucleation of crystal nuclei from an undercooled melt, which is an initial stage of solidification, is also outlined in the same section. In “Advances in Quantitative Computation of Solidification Microstructures” section, advances in quantitative computation of phase-field simulations are discussed with an examination of the convergence of the results of quantitative phase-field simulations. In “Large-Scale Phase-Field Simulation of Competitive Growth of Dendrite Assemblages” section, we report the competitive growth of dendrite assemblages during the directional solidification of a binary alloy bicrystal investigated by performing two- and three-dimensional large-scale simulations using the quantitative phase-field model by multi-GPU computation. We conclude with a discussion of the implications of these results for the future of computational metallurgy.

## Solidification in Large-Scale Molecular Dynamics Simulation

### Spontaneous Evolution of Anisotropy in a Solid Nucleus

Element (potential) | | Δ | | | | |
---|---|---|---|---|---|---|

Fe(bcc) (FS) | 2400 ± 10 [1811.0] | 0.22 [0.143] | 170 | 30.5 | 25.7 | – |

Fe(bcc) (ABCH) | 2358.7 ± 4.0 | 0.218 | 206 ± 10 | 32.5 ± 1.6 | 24.6 ± 1.4 | 25.6 ± 2.0 |

Fe(bcc) (MH(SA)) | 1772.0 ± 2.0 | 0.162 | 175 ± 11 | 45.8 ± 3.8 | 33.5 ± 1.0 | 31.8 ± 1.8 |

Fe(bcc) (pair) | 2311.8 ± 3.0 | 0.259 | 221 ± 14 | 30.8 ± 7.0 | 22.6 ± 4.0 | 23.1 ± 3.6 |

Fe(bcc) (MEAM) | 1807 | 0.134 | 184 ± 14 | – | – | – |

Fe(fcc) (ABCH) | 2251.0 ± 6.0 | 0.200 | 319 ± 12 | 24.0 ± 1.6 | 19.3 ± 2.2 | – |

Fe(fcc) (pair) | 2202.0 ± 7.0 | 0.212 | 311 ± 14 | – | – | – |

Ni (SMF) | 1820 [1726] | 0.187 [0.1776] | 310 ± 7 | – | – | – |

Ni (FBD) | 1714 | 0.179 | 284.7 ± 7.0 | 45 | 32 | 18 |

Ni (SC) | 1205 | 0.11 | 256 | – | – | – |

Ni (MEAM) | 1741.6 | 0.23 | 331.8 | |||

Cu (FBD) | 1279 [1356.6] | 0.116 [0.135] | 195.7 ± 5.6 | 46 | 27 | 19 |

Cu (MEAM) | 1319.6 | 0.15 | 255.0 | – | – | – |

Al (EA) | 930 [933.5] | 0.101 [0.109] | 149 | – | – | – |

Al (MEAM) | 924.9 | 0.12 | 172.6 | |||

Cr (SC) | 2232 [2180] | 0.16 [0.22] | 304 | – | – | – |

We have developed our own code for carrying out large-scale molecular dynamics simulations by single-GPU computing, which enables molecular dynamics simulations of 1 million atoms to be handled over a period of nanoseconds with a computational time of several days.56 Using this code with single-GPU computing, the spontaneous evolution of anisotropy in a solid nucleus during the solidification of iron was investigated by million-atom molecular dynamics simulation.52 As the simulation methodology, the Finnis–Sinclair potential,57 which is one of the established potentials for bcc metals, was employed for the interaction between iron atoms. A leapfrog method was used to integrate a classical equation of motion with a time step of 5.0 fs. A Berendsen thermostat58 was applied to control the temperature. The Andersen method59 was employed to control the pressure in each direction independently. The initial configuration was prepared by embedding an octagonal solid nucleus in an iron melt. The iron melt was obtained in advance by heating a bcc crystal of iron of size 53.4 × 53.4 × 4.3 nm^{3} (1,037,880 atoms) at 3500 K under a NVT constant condition. The solid nucleus was prepared as an octagonal cutout from the bcc crystal with four {100} facets and four {110} facets of side 1.78 nm. The solid nucleus was inserted into the center of the iron melt, while omitting all melt atoms located within 2.5 Å from solid atoms. After the quenching of the simulation cell at 10 K for 25 ps to fill the gap between the melt and solid atoms, the obtained initial configuration was isothermally undercooled at Δ*T* = 300 K. Note that the melting point of bcc Fe according to the Finnis–Sinclair potential is 2400 K,60 and therefore the undercooling temperature of 300 K corresponds to 2100 K.

*θ*from the

*x*-axis. In the figure showing the extracted information, the radius normalized by the average radius of 19.87 nm is plotted. It was confirmed that there are preferential growth directions at rotation angles of 0°, 90°, 180° and 270°, which correspond to the 〈100〉 direction. This is in agreement with the information in Table I, in which the kinetic coefficient of the 〈100〉 direction is larger than that of the 〈110〉 direction. The spontaneous evolution of anisotropy in a solid nucleus during solidification was achieved for the first time with the aid of the high processing ability of the GPU architecture.

### Homogeneous Nucleation and Subsequent Grain Growth

One of the remaining issues in the simulation of solidification is how to treat the nucleation.54 In existing phase-field simulations, the nuclei in the melt are given in advance with a random distribution or are formed forcibly on the basis of classical nucleation theory. On the other hand, it is possible in principle to achieve nucleation in molecular dynamics simulations when suitable conditions are given. For example, nucleation in a nanoscale liquid droplet has been achieved with relative ease under continuous cooling,61,62 which has been widely examined to study the size dependence of the melting point of metal nanoparticles.61, 62, 63, 64, 65, 66 However, generally, it is not yet straightforward to achieve multiple nucleation, which is essential for the formation of polycrystalline microstructures, since a broad range of temporal and spatial scales is required. Therefore, multiple nucleation in a large-scale system is usually achieved with the aid of inducing factors such as a high pressure53 and surface fluctuation.67

^{3}(1,037,880 atoms) was heated at 3500 K under a NVT (number of atoms, volume and temperature) constant condition to obtain an iron melt as the initial configuration. The prepared initial configuration was isothermally undercooled at Δ

*T*= 1000 K for 10,000 ps under zero pressure by a NPT (number of atoms, pressure and temperature) constant condition. The Finnis–Sinclair potential57 was employed for the interaction between iron atoms as in the above simulation. The periodic boundary condition was employed for all boundaries. Figure 2 shows snapshots of the atomic configuration during a consecutive simulation of nucleation, solidification and grain growth. Many nuclei are simultaneously nucleated before 150 ps and grow to form spherical grains in the melt. Other nuclei are continuously nucleated from the remaining melt. Later-nucleated grains fill the spaces between earlier-nucleated grains, and all the iron melt has solidified by 300 ps. After the solidification, the small grains gradually shrink and disappear whereas the large ones become larger, which is regarded as grain coarsening. Since the existence of grain boundaries yields excess grain boundary energy (approximately 0.5 J/m

^{2}to 2.0 J/m

^{2}60), such grain growth occurs in order to decrease the area of grain boundaries. It was also confirmed from the molecular dynamics simulation that the rate of grain coarsening is one order of magnitude slower than that of the solidification.

The incubation time until the first nucleation and the number of nuclei drastically change when the undercooling temperature is varied. It was confirmed that the incubation time as a function of temperature has a peak (i.e., nose shape) at the critical temperature, which is a characteristic shape of the time–temperature–transformation (TTT) curve.54 Therefore, it is considered that the nucleation observed in Fig. 2 is entirely thermally activated without any other inducing factor. The thermally activated nucleation in the million-atom molecular dynamics simulation has been investigated in detail elsewhere.54

## Advances in Quantitative Computation of Solidification Microstructures

### Quantitative Phase-Field Model

As seen above, recent atomic-scale simulations can capture the scale of microstructure evolution. On the other hand, the description and prediction of microstructural evolution during solidification have generally been theoretically and numerically tackled within the framework of a free-boundary problem, the underlying physics of which are solutal and thermal diffusion in the bulk, mass and energy conservation laws at the interface and the Gibbs–Thomson effects. One of the central issues in modeling microstructural processes is therefore the precise description of the interface dynamics consistent with the free-boundary problem. The phase-field model has emerged as a powerful tool for describing the microstructural evolution processes.18, 19, 20, 21, 22, 23 This is a diffuse interface approach, in which the interface is not sharp but diffuse, exhibiting non-zero thickness. The main advantage of this model is that it is not necessary to explicitly track the position of a sharp interface in complex microstructural patterns. The phase-field model has been applied to a variety of solidification processes,18, 19, 20, 21, 22 and its capability of affording a qualitative understanding of phenomena has generally been acknowledged. Despite this success, however, a long-standing issue regarding the quantitative aspect of the phase-field model remained unresolved until recently.

Phase-field models were developed in early works to reproduce the free-boundary problem of interest in the so-called sharp-interface limit, where the thickness of the diffuse interface *W* approaches zero. However, in practice, a prerequisite for this diffuse interface approach is to assign a finite value to *W*. A realistic value of *W* for the solid–liquid interface is typically a few nm; thus, a spatial resolution of Å order is required to describe a diffuse interface having a realistic thickness. This high spatial resolution limits the system size to extremely small, making it impossible to deal with problems at the microstructural scale. Therefore, *W* has to be increased by orders of magnitude from the realistic thickness. However, this increment, in turn, causes the unrealistic magnification of some physical effects associated with the diffuse interface, which precludes the quantitative computation of solidification microstructures.

This serious problem was resolved by Karma and Rappel.24,25 They put forward a model based on a new procedure called the thin-interface limit, in which *W* is taken to be smaller than any physical length appearing at the microstructural scale but much larger than the realistic thickness. This model is called the quantitative phase-field model since it enables quantitatively meaningful simulations.25 This model was later extended to deal with alloy solidification in a dilute binary alloy with zero diffusivity in the solid (one-sided model).26,27 Moreover, the quantitative phase-field model has been developed to describe two-phase solidification in binary alloys with zero diffusivity in the solids (one-sided model),28 alloy solidification with coupled heat and solute diffusion in dilute binary alloys having zero solutal diffusivity in the solid and equal thermal diffusivities in the solid and liquid (one-sided solute transport and symmetric heat transport),29 and isothermal solidification in multicomponent alloys with zero diffusivities in the solid (one-sided model).30 In addition, one of the present authors has recently developed quantitative phase-field models for the two-sided case. To be more specific, the models were developed for isothermal single-phase solidification,31 two-phase solidification in dilute binary alloys with an arbitrary value of the solid diffusivity,32 and single-phase solidification in multicomponent alloys with coupled solutal and thermal diffusion.33 These two-sided models enable us to describe the equilibrium solidification, the microsegregation, the motion of the solid–solid interface, and the solidification processes in practical alloy systems such as carbon steels, where the diffusion in the solid is not negligible. Quantitative phase-field models are being increasingly utilized for quantitative simulations of solidification phenomena.34,36,68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81

As mentioned above, significant progress has been made in quantitative phase-field modeling for alloy solidification. The accuracy of quantitative phase-field simulations is evaluated by observing the convergence behavior of the simulation results with decreasing *W*. It has been demonstrated that the convergence of the results in quantitative phase-field simulations is much faster than that of the results in the conventional phase-field model,31, 32, 33 which indicates that accurate results can be obtained using a large value for *W* in quantitative phase-field models. This is quite advantageous in terms of the computational cost because the computational time for a three-dimensional simulation using a finite difference method is proportional to *W* ^{−5}.69 Hence, the quantitative phase-field model enables highly accurate and large-scale computations of solidification microstructures. However, it should be pointed out that the value of *W* required to obtain well-converged results is strongly dependent on the solidification condition of interest. No criterion has yet been established regarding a suitable choice of *W*, and a convergence test is generally required for each solidification condition to ensure accurate and efficient computations. Therefore, it is desirable to obtain information to reduce the effort involved in the convergence test. This point is addressed below.

### Convergence of Outcome in Quantitative Phase-Field Simulation

*y*-direction with the periodic boundary condition applied in the

*x*-direction. The details of time evolution equations can be found in Ref. 36. The input parameters used in this study are listed in Table II. A temperature gradient and an initial undercooling were set to

*G*= 1000 K/m and

*u*

_{0}= (

*c*

_{l}−

*c*

_{0})/[(1 −

*k*)

*c*

_{0}] = −1, respectively, where

*c*

_{l}is the concentration in the liquid phase,

*c*

_{0}is the average concentration, and

*k*is the partition coefficient. We started with initial seeds periodically spaced by the targeted spacing. This spacing corresponds to the primary arm spacing

*λ*and it was set to about 150

*μ*m. By performing a moving frame simulation, we obtained steady-state values of tip undercooling, given by

*Ω*= 1 −

*y*

_{tip}/

*l*

_{T}, where

*y*

_{tip}is the position of the dendrite tip and

*l*

_{T}is the thermal diffusion length, and the curvature radius of the dendrite tip

*ρ*.

Input parameters employed in quantitative phase-field simulations for directional solidification of an Al-2mass%Cu alloy36

Melting temperature of pure Al ( | 933.25 K |

Liquidus slope ( | –620 (K/at. frac.) |

Partition coefficient ( | 0.14 |

Gibbs–Thomson coefficient ( | 2.4 × 10 |

Strength of anisotropy of solid–liquid interface ( | 0.02 |

Liquid diffusivity ( | 3 × 10 |

Solid diffusivity ( | 2 × 10 |

*V*= 500 and 50

*μ*m/s are shown in Fig. 3a and b, respectively. The horizontal axis is the interface thickness normalized by the chemical capillary length

*d*

_{0}. In each case,

*W*and

*ρ*fully converge with unique values when

*W*is small, which indicates excellent convergence behavior. However, the value of

*W*required for the convergence strongly depends on the pulling velocity,

*V*. The convergence for

*V*= 500 and 50

*μ*m/s starts to break down when

*W*/

*d*

_{0}< 60 and 200, respectively. This large difference in the convergence behavior indicates that the effort required to find a suitable value of

*W*strongly depends on the solidification condition of interest. One may suppose that the breakdown of convergence should be related to the onset of the unphysical magnification of the interface effects that always exists in conventional phase-field models constructed in the sharp-interface limit as described in the previous section.

According to Fig. 3a and b, the converged value of *ρ* is strongly dependent on the pulling velocity. Hereafter, the values of Ω and *ρ* calculated for the smallest value of *W* are regarded as the converged values and are, respectively, denoted by *Ω* _{c} and *ρ* _{c}. In Fig. 3, the values of *ρ* _{c}/*d* _{0} for *V* = 500 and 50 *μ*m/s are about 300 and 1000, respectively. Hence, from the comparison between Fig. 3a and b, one may speculate that the convergence of the simulation for a relatively coarse structure starts to break down at a relatively large *W*. We investigated the validity of this speculation in a quantitative manner. All the data shown in Fig. 3a and b are plotted in Fig. 3c, where *ρ* and *Ω* are normalized by *ρ* _{c} and *Ω* _{c}, respectively, on the *y*-axis and *W* is normalized by *ρ* _{c} on the *x*-axis. It can be seen that the convergence starts to break down when *W*/*ρ* _{c} ~ 0.2 in both cases. In other words, regardless of the solidification condition, the results fully converge as long as *W*/*ρ* _{c} ≤ 0.2. This is also supported by the results for the directional solidification of an impure succinonitrile alloy in Ref. 27.

Note that the steady-state profile of the phase-field in our model is given by *ϕ* = tanh[*r*/(2^{1/2} *W*)]. Here, *ϕ* is the phase-field, which takes a value of +1 (−1) in the solid (liquid) and continuously changes from -1 to +1 inside the interface, and *r* is the spatial coordinate in the direction normal to the interface. This solution is obtained for the boundary condition *ϕ* = ±1 at *r* → ±∞. In this model, the interface thickness cannot be well defined. In the above discussion, *W* was used as a measure of the interface thickness and is actually the length of the region for −0.34 < *ϕ* < 0.34. When the region for −0.95 < *ϕ* < 0.95 is considered, the thickness of this region *W*′ is about 5 *W*. Hence, the condition *W*/*ρ* _{c} ≤ 0.2 corresponds to *W*′ ≤ *ρ* _{c}. Within the framework of the diffuse interface approach, an accurate description of the size and morphology of microstructures is not possible when the interface thickness is set to larger than the minimum curvature radius of the interface appearing in the microstructure. The condition *W*′ ≤ *ρ* _{c} should originate from this fact. Namely, the breakdown of the convergence shown in Fig. 3c is not triggered by the onset of the unphysical magnification of interface effects and is actually a natural consequence of the limitation unique to the diffuse interface approach.

*V*/

*V*

_{c}and

*ρ*/

*ρ*

_{c}, respectively. Here,

*V*is the steady-state value of moving velocity of dendrite tip and

*V*

_{c}is the converged one, viz., the value calculated for the smallest value of

*W*. Datasets A and B are the results for isothermal solidification in binary alloys (Figs. 4 and 5 in Ref. 31). Datasets C and D are those for non-isothermal solidification in a binary alloy without and with diffusion in the solid (Figs. 2 and 3 in Ref. 32) and datasets E and F are the results for isothermal and non-isothermal solidification in a ternary alloy (Figs. 4 and 5 in Ref. 32), respectively. Importantly, all the data converge as long as

*W*/

*ρ*

_{c}≤ 0.2. Hence, the condition

*W*/

*ρ*

_{c}≤ 0.2 holds true in both directional and free dendritic growth. This fact will reduce the effort involved in convergence tests. Once the minimum curvature radius,

*ρ*

_{c}, of the growing phase(s) appearing during a microstructural evolution process is obtained from preliminary simulations, accurate and efficient computation can be conducted by assigning a value of about 0.2

*ρ*

_{c}to

*W*.

## Large-Scale Phase-Field Simulation of Competitive Growth of Dendrite Assemblages

### High-Performance Computation for the Phase-Field Method

The development of the quantitative phase-field model enables the use of a large interface thickness *W* or a large computational lattice. However, as shown in the previous section, the value of *W* is restricted by the curvature of the dendrite tip *ρ*. Therefore, dendrite growth simulations using the phase-field method have been limited to two-dimensional problems or three-dimensional simulations of a small number of dendrites. Actually, many solidification structures are formed through the interactions during the competitive growth of dendrite assemblages.1,2 Although the cellular automaton method has been widely used for polycrystal solidification simulations,15,16,82 and has been employed for large-scale solidification simulations,83,84 multiple-dendrite-growth simulation by the phase-field method is crucial for accurate prediction of the solidification microstructure. An adaptive mesh refinement technique, in which fine meshes are used only around the interface,85, 86, 87, 88, 89 can reduce the computational cost. However, its applicability to polycrystal solidification with a large interface area fraction is not flexible. Moreover, the development of the code for adaptive mesh refinement requires tremendous effort.

Under such circumstances, GPU computation has attracted the attention of many phase-field researchers because GPUs have been successfully used to increase the speed of phase-field computation.33,90 Moreover, parallel computation using multiple GPUs has the potential to capture realistic dendrite assemblages.22,35,91,92 Shimokawabe et al. achieved the first-ever petascale phase-field simulation of dendrite growth using 4000 GPUs on the TSUBAME2.0 supercomputer at the Tokyo Institute of Technology.92 Subsequently, the present authors and coworkers successfully performed a very-large-scale simulation of multiple dendritic competitive growth using 4000^{3} meshes.35

From the viewpoint of applications, understanding of the competitive growth of dendrite assemblages is essential to improve and control solidification microstructures. It is widely accepted that dendrites whose 〈100〉 preferential growth direction is almost parallel to the heat flow direction can continue to grow by stopping the growth of dendrites having a 〈100〉 crystallographic orientation that deviates from the heat flow direction.16,93,94 On the other hand, unusual dendrite selections, in which inclined dendrites overgrow dendrites growing in the heat flow direction, have recently been observed in the unidirectional solidification of a bicrystal sample.95,96 To clarify the mechanism of this unusual overgrowth, two-dimensional phase-field simulations have been performed.34,36,90 In the following subsections, we report the competitive growth of dendrite assemblages investigated by two- and three-dimensional simulations using the quantitative phase-field model, which were performed on multiple GPUs.

### Two-Dimensional Simulation of Competitive Growth

Figure 5 shows snapshots from a two-dimensional simulation of competitive dendrite growth during the directional solidification of a binary alloy bicrystal. The quantitative phase-field model for the solidification of a dilute binary alloy31 was used with the moving frame algorithm for directional solidification under a constant temperature gradient, *G*. The computational conditions were same as in Ref. 36 except for a computational domain of 3.072 × 1.152 mm^{2} (4096 × 1536 meshes) and 20 million computational steps (750 s) performed for Al-3mass%Cu with *V* = 100 *μ*m/s and *G* = 10 K/mm. The computation was performed within 1 day using eight GPUs. Two seeds were placed at the two bottom corners of the computational domain, where the left seed was the favorably oriented (FO) grain and the right seed was the unfavorably oriented (UO) grain with its 〈100〉 crystallographic orientation at angle of 10º from the heat flow direction. As shown in Fig. 5a, the solids cover the bottom surface and the dendrites grow in the heat flow direction. A grain boundary (GB) is formed at the collision point between the two grains, as shown in Fig. 5b, and steady-state competitive growth between the GB dendrites starts from 8 × 10^{5} steps (Fig. 5c). Here, FO and UO dendrites are labeled using “F” and “U”, respectively. At the GB, some UO dendrites are blocked by the FO dendrite, and then the UO dendrite overgrows the FO dendrite after the blocking. Finally, all the FO dendrites are overgrown after about 18 million steps. To overgrow the FO dendrites labeled F1–F7, 3, 3, 3, 3, 8, 9 and 1 UO dendrites are required, respectively. This means that, for example, the F1 dendrite blocks the growth of the U1 and U2 dendrites and is overgrown by the U3 dendrite, and the F2 dendrite blocks the U3 and U4 dendrites and is overgrown by the U5 dendrite. This difference is mainly caused by the difference in the dendrite arm spacing between the GB FO dendrite and the FO dendrite at its immediate left. As shown in Fig. 5c, the arm spacing of F5–F6 (197 *μ*m) and F6-F7 (324 *μ*m) is the largest compared to F1–F2 (175 *μ*m), F2–F3 (161 *μ*m), F3–F4 (182 *μ*m), and F4–F5 (179 *μ*m). In addition, the UO dendrite arm spacing also affects the overgrowth of FO dendrites. The F4 dendrite is overgrown by the U9 dendrite. The average UO dendrite arm spacing shown in Fig. 5c, or U1–U7, is 199 *μ*m. On the other hand, the average UO dendrite arm spacing shown in Fig. 5d, or U10–U16, is 296 *μ*m. Thus, the large difference in the number of UO dendrites needed to overgrow the FO dendrite is caused by both spacing of FO dendrites and UO dendrites. As shown in Fig. 5d, when the U9 dendrite approaches the F4 dendrite, both GB dendrites fall down and the F4 dendrite moves to the left. When the spacing between F4 and F5 reaches the minimum value in which the two dendrites can coexist,36 the F4 dendrite is overgrown by the U9 dendrite. Accordingly, the horizontal migration of the FO dendrite when the UO dendrite approaches to the FO dendrite is a key process in the unusual selection. The unusual overgrowth observed in the present simulation occurs less readily with increasing inclination angle of the UO dendrites90 and pulling velocity34 due to the reduced the solute interaction around the tips of the GB dendrites.

### Three-Dimensional Simulation of Competitive Growth

^{3}(1536 × 1536 × 1024 meshes) and a computation of 0.5 million steps (23.7 s) was performed. Except for the lattice size of Δ

*x*= 1

*μ*m and the inclination angle of UO grain of 20°, the computational conditions were the same as in the previous two-dimensional simulation. It took about half a day for this simulation to be performed using 512 GPUs of the TSUBAME2.5 supercomputer at Tokyo Institute of Technology, which is a practical computational time. At the beginning of the computation, as shown in Fig. 6a, the two grains spread along the bottom surface to form a fanlike shape, and many secondary arms grow in the heat flow direction. Figure 6b show that the two grains collide and a straight GB is formed because of the high density of arms. Dendrite selection subsequently occurs and the number of dendrites decreases as shown in Fig. 6c and d. Because the inclined dendrites grow in the 〈100〉 direction with increasing arm spacing,99, 100, 101, 102 the UO dendrites move toward the FO dendrites. As a result, the competition between dendrites becomes intense at the GB, and the shape of the GB becomes zigzag as shown in Fig. 6e. This zigzag GB is very similar to that observed experimentally.97,98

Here, we showed the very beginning stage of competitive growth. By continuing this simulation longer, we will be able to observe the detail competition between FO and UO dendrites in three-dimensional space in detail. In three-dimensional space, because the solute diffusion is possible in the three directions, we need a longer computational time than for the two-dimensional problem to see the unusual overgrowth phenomenon. Therefore, this is challenging topic even using a supercomputer. Nevertheless, the results will be available in the near future.

## Conclusion

Utilizing the high parallel efficiency of GPUs, cutting-edge simulations were performed to capture the nature of solidification from various viewpoints. From an atomic viewpoint, a million-atom molecular dynamics simulation revealed the spontaneous evolution of anisotropy in a solid nucleus embedded in an undercooled iron melt, in which fourfold symmetry was achieved naturally without the use of any empirical parameters. Homogeneous nucleation from an undercooled melt was achieved by another million-atom molecular dynamics simulation, in which multiple nuclei solidified to form a multigrain microstructure and grain coarsening occurred during 10 ns, according to the results of the calculation. Moreover, the convergence behavior in quantitative phase-field simulations has been discussed in detail. Such convergence enables the use of a large interface thickness in quantitative phase-field simulations. Using the quantitative phase-field model for the solidification of a dilute binary alloy, the competitive growth of dendrite assemblages during the directional solidification of a binary alloy bicrystal in a millimeter scale was examined by performing two- and three-dimensional large-scale simulations by multi-GPU computation. From the two-dimensional simulation, the mechanism of the unusual overgrowth phenomenon, in which dendrites inclined to the heat flow direction overgrow those growing in the heat flow direction during unidirectional solidification, was clarified. On the other hand, a zigzag grain boundary was formed during the competition between favorably and unfavorably oriented dendrites in the three-dimensional phase-field simulation.

^{3}(4096 × 4096 × 4096 meshes) for a total time period of more than 100 s (4 million computational steps) using 768 GPUs with 768 CPUs on the TSUBAME2.0, which is the largest simulation of dendrite growth ever to be reported to the best of our knowledge.

## Notes

### Acknowledgements

This work was supported by Grant-in-Aid for Scientific Research (B) (Nos. 25289266 and 25289006) from Japan Society for the Promotion of Science (JSPS), Japan; 22th (T.T and M.O) and 23th (Y.S) ISIJ Research Promotion Grants from the Iron and Steel Institute of Japan (ISIJ), the Strategic Programs for Innovative Research (SPIRE), MEXT; and the Computational Materials Science Initiative (CMSI), Japan. Part of this work was supported by the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN) and the High Performance Computing Infrastructure (HPCI) in Japan for the computational environment. Y.S. would like to thank Ms. Kanae Oguchi for support in the development of the source code for the GPU environment for the molecular dynamics simulations. T.T. also thanks Mr. Shinji Sakane, Prof. Takashi Shimokawabe, and Prof. Takayuki Aoki for the support in the large-scale simulations using the TSUBAME2.5 supercomputer.

## References

- 1.W. Kurz and D.J. Fisher,
*Fundamentals of Solidification*(Aedermannsdorf: Trans Tech Publications, 1998), pp. 1–16.Google Scholar - 2.J.A. Dantzig and M. Rappaz,
*Solidification*(Lausanne: EPFL Press, 2009), pp. 1–22.CrossRefGoogle Scholar - 3.K.A. Jackson and J.D. Hunt,
*Acta Metall.*13, 1212 (1965).CrossRefGoogle Scholar - 4.H. Esaka and W. Kurz,
*J. Cryst. Growth*72, 578 (1985).CrossRefGoogle Scholar - 5.R.H. Mathiesen, L. Arnberg, F. Mo, T. Weitkamp, and A. Snigirev,
*Phys. Rev. Lett.*83, 5062 (1999).CrossRefGoogle Scholar - 6.A. Bogno, H. Nguyen-Thi, A. Buffet, G. Reinhart, B. Billia, N. Mangelinck-Noël, N. Bergeon, J. Baruchel, and T. Schenk,
*Acta Mater.*59, 4356 (2011).CrossRefGoogle Scholar - 7.H. Yasuda, T. Nagira, M. Yoshiya, A. Sugiyama, N. Nakatsuka, M. Kiire, M. Uesugi, K. Uesugi, K. Umetani, and K. Kajiwara,
*IOP Conf. Ser.: Mater. Sci. Eng.*33, 012036 (2012).CrossRefGoogle Scholar - 8.R.B. Potts,
*Math. Proc.*48, 106 (1952).MathSciNetCrossRefGoogle Scholar - 9.D.J. Srolovitz, M.P. Anderson, G.S. Grest, and P.S. Sahni,
*Scr. Metall.*17, 241 (1983).CrossRefGoogle Scholar - 10.M.P. Anderson, D.J. Srolovitz, G.S. Grest, and P.S. Sahni,
*Acta Metall.*32, 783 (1984).CrossRefGoogle Scholar - 11.A.W. Godfrey and J.W. Martin,
*Philos. Mag. A*72, 737 (1995).CrossRefGoogle Scholar - 12.T. Koseki, H. Inoue, Y. Fukuda, and A. Nogami,
*Sci. Technol. Adv. Mater.*4, 183 (2003).CrossRefGoogle Scholar - 13.D. Juric and G. Tryggvason,
*J. Comput. Phys.*123, 127 (1996).MathSciNetCrossRefGoogle Scholar - 14.A. Jacot and M. Rappaz,
*Acta Mater.*50, 1909 (2002).CrossRefGoogle Scholar - 15.M. Rappaz and C.A. Gandin,
*Acta Metall. Mater.*41, 345 (1993).CrossRefGoogle Scholar - 16.C.A. Gandin and M. Rappaz,
*Acta Metall. Mater.*42, 2233 (1994).CrossRefGoogle Scholar - 17.R. Kobayashi,
*Phys. D*63, 410 (1993).CrossRefGoogle Scholar - 18.W.J. Boetinger, J.A. Warren, C. Beckermann, and A. Karma,
*Annu. Rev. Mater. Res.*32, 163 (2002).CrossRefGoogle Scholar - 19.L. Granasy, T. Pusztai, and J.A. Warren,
*J. Phys.: Condens. Matter.*16, R1205 (2004).Google Scholar - 20.H. Emmerich,
*Adv. Phys.*57, 1 (2008).CrossRefGoogle Scholar - 21.I. Steinbach,
*Model. Simul. Mater. Sci. Eng.*17, 073001 (2009).MathSciNetCrossRefGoogle Scholar - 22.T. Takaki,
*ISIJ Int.*54, 437 (2014).CrossRefGoogle Scholar - 23.S.G. Kim, W.T. Kim, and T. Suzuki,
*Phys. Rev. E*60, 7186 (1999).CrossRefGoogle Scholar - 24.A. Karma and W.-J. Rappel,
*Phys. Rev. E*53, R3017 (1996).CrossRefGoogle Scholar - 25.A. Karma and W.-J. Rappel,
*Phys. Rev. E*57, 4323 (1998).CrossRefGoogle Scholar - 26.A. Karma,
*Phys. Rev. Lett.*87, 115701 (2001).CrossRefGoogle Scholar - 27.B. Echebarria, R. Folch, A. Karma, and M. Plapp,
*Phys. Rev. E*70, 061604 (2004).CrossRefGoogle Scholar - 28.R. Folch and M. Plapp,
*Phys. Rev. E*72, 011602 (2005).MathSciNetCrossRefGoogle Scholar - 29.J.C. Ramirez, C. Beckermann, A. Karma, and H.-J. Diepers,
*Phys. Rev. E*69, 051607 (2004).CrossRefGoogle Scholar - 30.S.G. Kim,
*Acta Mater.*55, 4391 (2007).CrossRefGoogle Scholar - 31.M. Ohno and K. Matsuura,
*Phys. Rev. E*79, 031603 (2009).CrossRefGoogle Scholar - 32.M. Ohno and K. Matsuura,
*Acta Mater.*58, 5749 (2010).CrossRefGoogle Scholar - 33.M. Ohno,
*Phys. Rev. E*86, 051603 (2012).CrossRefGoogle Scholar - 34.J. Li, Z. Wang, Y. Wang, and J. Wang,
*Acta Mater.*60, 1478 (2012).CrossRefGoogle Scholar - 35.T. Takaki, T. Shimokawabe, M. Ohno, A. Yamanaka, and T. Aoki,
*J. Cryst. Growth*382, 21 (2013).CrossRefGoogle Scholar - 36.T. Takaki, M. Ohno, T. Shimokawabe, and T. Aoki,
*Acta Mater.*81, 272 (2014).CrossRefGoogle Scholar - 37.R. Tönhardt and G. Amberg,
*J. Cryst. Growth*213, 161 (2013).CrossRefGoogle Scholar - 38.T. Takaki, R. Rojas, M. Ohno, T. Shimokawabe, and T. Aoki,
*IOP Conf. Ser.: Mater. Sci. Eng.*(in press).Google Scholar - 39.S. Chen and G.D. Doolen,
*Annu. Rev. Fluid Mech.*30, 329 (1998).MathSciNetCrossRefGoogle Scholar - 40.X. Shan and H. Chen,
*Phys. Rev. E*47, 1815 (1993).CrossRefGoogle Scholar - 41.J.J. Hoyt, B. Sadigh, M. Asta, and S.M. Foiles,
*Acta Mater.*47, 3181 (1999).CrossRefGoogle Scholar - 42.M. Asta, J.J. Hoyt, and A. Karma,
*Phys. Rev. B*66, 100101 (2002).CrossRefGoogle Scholar - 43.J.R. Morris,
*Phys. Rev. B*66, 144104 (2002).CrossRefGoogle Scholar - 44.J.J. Hoyt, M. Asta, and A. Karma,
*Mater. Sci. Eng. R*41, 121 (2003).CrossRefGoogle Scholar - 45.D.Y. Sun, M. Asta, and J.J. Hoyt,
*Phys. Rev. B*69, 174103 (2004).CrossRefGoogle Scholar - 46.Y. Watanabe, Y. Shibuta, and T. Suzuki,
*ISIJ Int.*50, 1158 (2010).CrossRefGoogle Scholar - 47.R. Hashimoto, Y. Shibuta, and T. Suzuki,
*ISIJ Int.*51, 1664 (2011).CrossRefGoogle Scholar - 48.E. Asadi, M.A. Zaeem, S. Nouranian, and M.I. Baskes,
*Phys. Rev. B*91, 024105 (2015).CrossRefGoogle Scholar - 49.E. Asadi, M.A. Zaeem, S. Nouranian, and M.I. Baskes,
*Acta Mater.*86, 169 (2015).CrossRefGoogle Scholar - 50.Y. Shibuta, K. Oguchi, and T. Suzuki,
*ISIJ Int.*52, 2205 (2012).CrossRefGoogle Scholar - 51.M. Berghoff, M. Selzer, and B. Nestler,
*Sci. World J.*2013, 564272 (2013).CrossRefGoogle Scholar - 52.Y. Shibuta, K. Oguchi, and M. Ohno,
*Scr. Mater.*86, 20 (2014).CrossRefGoogle Scholar - 53.F.H. Streitz, J.N. Glosli, and M.V. Patel,
*Phys. Rev. Lett.*96, 225701 (2006).CrossRefGoogle Scholar - 54.Y. Shibuta and K. Oguchi, The University of Tokyo, Tokyo, T. Takaki: Kyoto Institute of Technology, Kyoto, and M. Ohno: Hokkaido University, Sapporo, unpublished research (2015).Google Scholar
- 55.Y. Shibuta, Y. Watanabe, and T. Suzuki,
*Chem. Phys. Lett.*475, 264 (2009).CrossRefGoogle Scholar - 56.K. Oguchi, Y. Shibuta, and T. Suzuki,
*J. Jpn. Inst. Met.*76, 462 (2012).CrossRefGoogle Scholar - 57.M.W. Finnis and J.E. Sinclair,
*Philos. Mag. A*50, 45 (1984).CrossRefGoogle Scholar - 58.H.J.C. Berendsen, J.P.M. Postma, W.F. van Gunsteren, A. DiNola, and J.R. Haak,
*J. Chem. Phys.*81, 3684 (1984).CrossRefGoogle Scholar - 59.H.C. Andersen,
*J. Chem. Phys.*72, 2384 (1980).CrossRefGoogle Scholar - 60.Y. Shibuta, S. Takamoto, and T. Suzuki,
*ISIJ Int.*48, 1582 (2008).CrossRefGoogle Scholar - 61.Y. Shibuta and T. Suzuki,
*Chem. Phys. Lett.*445, 265 (2007).CrossRefGoogle Scholar - 62.Y. Shibuta and T. Suzuki,
*J. Chem. Phys.*129, 144102 (2008).CrossRefGoogle Scholar - 63.Y. Qi, T. Çağin, W.L. Johnson, and W.A. Goddard III,
*J. Chem. Phys.*115, 385 (2001).CrossRefGoogle Scholar - 64.F. Ding, A. Rosén, S. Curtarolo, and K. Bolton,
*Appl. Phys. Lett.*88, 133110 (2006).CrossRefGoogle Scholar - 65.Y. Shibuta and T. Suzuki,
*Phys. Chem. Chem. Phys.*12, 731 (2010).CrossRefGoogle Scholar - 66.Y. Shibuta and T. Suzuki,
*Chem. Phys. Lett.*502, 82 (2011).CrossRefGoogle Scholar - 67.T. Li, D. Donadio, L.M. Ghiringhelli, and G. Galli,
*Nat. Mater.*8, 726 (2009).CrossRefGoogle Scholar - 68.M. Greenwood, M. Haataja, and N. Provatas,
*Phys. Rev. Lett.*93, 246101 (2004).CrossRefGoogle Scholar - 69.C.W. Lan and C.J. Shih,
*Phys. Rev. E*69, 031601 (2004).CrossRefGoogle Scholar - 70.J.C. Ramirez and C. Beckermann,
*Acta Mater.*53, 1721 (2005).CrossRefGoogle Scholar - 71.H. Emmerich and R. Siquieri,
*J. Phys.: Condens. Matter.*18, 11121 (2006).Google Scholar - 72.A. Parisi and M. Plapp,
*Acta Mater.*56, 1348 (2008).CrossRefGoogle Scholar - 73.J. Rosam, P.K. Jimack, and A.M. Mullis,
*Acta Mater.*56, 4559 (2008).CrossRefGoogle Scholar - 74.J. Rosam, P.K. Jimack, and A.M. Mullis,
*Phys. Rev. E*79, 030601 (2009).CrossRefGoogle Scholar - 75.K. Oguchi and T. Suzuki,
*ISIJ Int.*49, 1536 (2009).CrossRefGoogle Scholar - 76.M. Ohno and K. Matsuura,
*Acta Mater.*58, 6134 (2010).CrossRefGoogle Scholar - 77.M. Ohno and K. Matsuura,
*ISIJ Int.*50, 1879 (2010).CrossRefGoogle Scholar - 78.G. Boussinot, E.A. Brener, and D.E. Temkin,
*Acta Mater.*58, 1750 (2010).CrossRefGoogle Scholar - 79.B. Echebarria, A. Karma, and S. Gurevich,
*Phys. Rev. E*81, 021608 (2010).CrossRefGoogle Scholar - 80.S. Gurevich, A. Karma, M. Plapp, and R. Trivedi,
*Phys. Rev. E*81, 011603 (2010).CrossRefGoogle Scholar - 81.Z. Wang, J. Wang, J. Li, G. Yang, and Y. Zhou,
*Phys. Rev. E*84, 041604 (2011).CrossRefGoogle Scholar - 82.C.A. Gandin, T. Carozzani, H. Digonnet, S. Chen, and G. Guillemot,
*JOM*65, 1122 (2013).CrossRefGoogle Scholar - 83.M. Eshraghi, S.D. Felicelli, and B. Jelinek,
*J. Cryst. Growth*354, 129 (2012).CrossRefGoogle Scholar - 84.B. Jelinek, M. Eshraghi, S. Felicelli, and J.F. Peters,
*Comput. Phys. Commun.*185, 939 (2014).CrossRefGoogle Scholar - 85.N. Provatas, N. Goldenfeld, and J. Dantzig,
*Phys. Rev. Lett.*80, 3308 (1998).CrossRefGoogle Scholar - 86.N. Provatas, N. Goldenfeld, and J. Dantzig,
*J. Comput. Phys.*148, 265 (1999).MathSciNetCrossRefGoogle Scholar - 87.C.W. Lan, Y.C. Chang, and C.J. Shih,
*Acta Mater.*51, 1857 (2003).CrossRefGoogle Scholar - 88.T. Takaki, T. Fukuoka, and Y. Tomita,
*J. Cryst. Growth*283, 263 (2005).CrossRefGoogle Scholar - 89.Z. Guo and S.M. Xiong,
*Comput. Phys. Commun.*190, 89 (2015).MathSciNetCrossRefGoogle Scholar - 90.D. Tourret and A. Karma,
*Acta Mater.*82, 64 (2015).CrossRefGoogle Scholar - 91.A. Yamanaka, T. Aoki, S. Ogawa, and T. Takaki,
*J. Cryst. Growth*318, 40 (2011).CrossRefGoogle Scholar - 92.T. Shimokawabe, T. Aoki, T. Takaki, A. Yamanaka, A. Nukada, T. Endo, N. Maruyama, and S. Matsuoka,
*Proceedings of 2011 SC—International Conference for High Performance Computing, Networking, Storage and Analysis*, 1 (2011).Google Scholar - 93.D. Walton and B. Chalmers,
*Trans. Metall. Soc. AIME*215, 447 (1959).Google Scholar - 94.H. Esaka, M. Tamura, and K. Shinozuka,
*Mater. Trans.*44, 829 (2003).CrossRefGoogle Scholar - 95.N. D’Souza, M.G. Ardakani, A. Wagner, B.A. Shollock, and M. McLean,
*J. Mater. Sci.*37, 481 (2002).CrossRefGoogle Scholar - 96.Y.Z. Zhou, A. Volek, and N.R. Green,
*Acta Mater.*56, 2631 (2008).CrossRefGoogle Scholar - 97.Z. Liu, M. Lin, D. Yu, X. Zhou, Y. Gu, and H. Fu,
*Metall. Mater. Trans. A*44, 5113 (2013).CrossRefGoogle Scholar - 98.C. Yang, L. Liu, X. Zhao, N. Wang, J. Zhang, and H. Fu,
*J. Alloys Compd.*578, 577 (2013).CrossRefGoogle Scholar - 99.S. Akamatsu and T. Ihle,
*Phys. Rev. E*56, 4479 (1997).CrossRefGoogle Scholar - 100.A. Pocheau, J. Deschamps, and M. Georgelin,
*JOM*59–57, 71 (2007).CrossRefGoogle Scholar - 101.J. Deschamps, M. Georgelin, and A. Pocheau,
*Phys. Rev. E*78, 011605 (2008).CrossRefGoogle Scholar - 102.A. Pocheau, J. Deschamps, and M. Georgelin,
*Phys. Rev. E*81, 051608 (2010).CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.