Abstract
This paper describes the verification and validation (V&V) framework developed for the stochastic Particle-in-Cell, Direct Simulation Monte Carlo code Aleph. An ideal framework for V&V from the viewpoint of the authors is described where a physics problem is defined, and relevant physics models and parameters to the defined problem are assessed and captured in a Phenomena Identification and Ranking Table (PIRT). Numerous V&V examples guided by the PIRT for a simple gas discharge are shown to demonstrate the V&V process applied to a real-world simulation tool with the overall goal to demonstrably increase the confidence in the results for the simulation tool and its predictive capability. Although many examples are provided here to demonstrate elements of the framework, the primary goal of this work is to introduce this framework and not to provide a fully complete implementation, which would be a much longer document. Comparisons and contrasts are made to more usual approaches to V&V, and techniques new to the low-temperature plasma community are introduced. Specific challenges relating to the sufficiency of available data (e.g., cross sections), the limits of ad hoc validation approaches, the additional difficulty of utilizing a stochastic simulation tool, and the extreme cost of formal validation are discussed.
Graphic Abstract
Similar content being viewed by others
Data Availability Statement
This manuscript has no associated data or the data will not be deposited. [Authors’ comment:...].
References
L. Schwer, An overview of the PTC 60/V&V 10: guide for verification and validation in computational solid mechanics. Eng. Comput. 23, 245–252 (2007)
M. Turner, Workshop on the verification and validation of computer simulations in low temperature plasma physics. in Gaseous Electronics Conference (2012)
M. Turner, Verification of particle-in-cell simulations with Monte Carlo collisions. Plasma Sources Sci. Technol. 25, 054007 (2016)
M. Turner, A. Derzsi, Z. Donko, D. Eremin, S. Kelly, T. Lafleur, T. Mussenbrock, Simulation benchmarks for low-pressure plasmas: capacitive discharges. Phys. Plasmas 20, 013507 (2013)
B. Bagheri, J. Teunissen, U. Ebert, M. Becker, S. Chen, O. Ducasse, O. Eichwald, D. Loffhagen, A. Luque, D. Mihailova, J. Plewa, J. van Dijk, M. Yousfi, Comparison of six simulation codes for positive streamers in air. Plasma Sources Sci. Technol. 27, 095002 (2018)
M. Surendra, Radiofrequency discharge Benchmark model comparison. Plasma Sources Sci. Technol. 4, 56 (1995)
T. Trucano, M. Pilch, W. Oberkampf, On the role of code comparisons in verification and validation. Sandia National Laboratories SAND2003–2752, Albuquerque (2003)
J. Carlsson, A. Khrabrov, I. Kaganovich, T. Sommerer, D. Keating, Validation and bechmarking of two particle-in-cell codes for a glow discharge. Plasma Sources Sci. Technol. 26, 014003 (2016)
A. Derzsi, T. Lafleur, J. Booth, I. Korolov, Z. Donko, Experimental and simulation study of a capacitively coupled oxygen discharge dirven by tailored voltage waveforms. Plasma Sources Sci. Technol. 25, 015004 (2015)
A. Derzsi, B. Bruneau, A. Gibson, E. Johnson, D. O’connell, T. Gans, J. P. Booth, Z. Donko, , Power coupling mode transitions induced by tailored voltage waveforms in capacitive oxygen discharges. Plasma Sources Sci. Technol. 26, 034002 (2017)
R. Hood, B. Scheiner, S. Baalrud, M. Hopkins, E. Barnat, B. Yee, R. Merlino, F. Skiff, Ion flow and sheath structure near positively biased electrodes. Phys. Plasmas 23, 113503 (2016)
Z. Donko, A. Derzsi, P. Hartmann, S. Brandt, J. Schulze, B. Berger, M. Koepke, B. Bruneau, E. Johnson, T. Lafleur, J.P. Booth, A. Gibson, D. O’Connell, T. Gans, Experimental benchmark of kinetic simulations of capacitively coupled plasmas in molecular gases. Plasma Phys. Controll. Fusion 60, 014010 (2018)
M. Turner, Uncertainty and error in complex plasma chemistry models. Plasma Sources Sci. Technol. 24, 035027 (2015)
M. Turner, Uncertainty and sensitivity analysis in complex plasma chemistry models. Plasma Sources Sci. Technol. 25, 015003 (2016)
A. Fierro, C. Moore, B. Yee, M. Hopkins, Three-dimensional kinetic modeling of streamer propagation in a nitrogen/helium gas mixture. Plasma Sources Sci. Technol. 27, 105008 (2018)
A.S. Fierro, C. Moore, B. Scheiner, B.T. Yee, M.M. Hopkins, Radiation transport in kinetic simulations and the influence of photoemission on electron current in self-sustaining discharges. J. Phys. D Appl. Phys. 50, 065202 (2017)
C. Birdsall, A. Langdon, Plasma Physica via Computer Simulation (McGraw-Hill, New York, 2005)
G. Bird, Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Oxford University Press, Oxford, 1994)
M. Hopkins, P. Crozier, C. Moore, Comparison of Aleph and BOLSIG\(+\)Results for Electron-Nitrogen Chemistry. Sandia National Laboratories SAND2014-19653, Albuquerque (2014)
G. Radtke, L. Musson, K. Cartwright, Error Estimation for Aleph PIC Plasma Sheath Simulations. Sandia National Laboratories SAND2012-5195C, Albuqerque (2012)
C. Roark, P. Stoltz, Aleph Verification Simulations. Sandia National Laboratories SAND2012-5192C, Albuqerque (2012)
P. Crozier, P. Stewart, Arc Simulations with Aleph. Sandia National Laboratories SAND2010-6564C, Albuquerque (2010)
M. Bettencourt, Aleph Code Electrostatic Solver Verification. Sandia National Laboratories SAND2015-0339, Albuquerque (2015)
R. Hooper, S. Moore, Aleph Field Solver Challenge Problem Results Summary. Sandia National Laboratories SAND2015-0317, Albuqeruqe (2015)
H. Timko, P.S. Crozier, M.M. Hopkins, K. Matyash, R. Schneider, Why perform Code-to\_code comparisons: a vacuum Arc discharge simulation case study. Contrib. Plasma Phys. 52(4), 295–308 (2012)
J. Boerner, J. Pacheco, A. Grillet, Evaluation of the Aleph PIC code on Benchmark simulations, in Gaseous Electronics Conference, Bochum (2016)
T.J. Oliver, S.P. Nowlen, A Phenomena Identification and Ranking Table (PIRT) Exercise for Nuclear Power Plant Fire Modeling Applications. US Nuclear Regulatory Commission, NUREG/CR-6978 and SAND2008-3997P (2008)
M. Heroux, R. Bartlett, V. Howle, R. Hoekstra, J. Hu, T. Kolda, R. Lehoucq, K. Long, R. Pawlowski, E. Phipps, A. Salinger, H. Thornquist, R. Tuminaro, J. Willenbring, A. Williams, K. Stanley, An overview of the Trilinos project. ACM Trans. Math. Softw. 31, 13 (2005)
G. Radtke, N. Martin, C. Moore, A. Huang, K. Cartwright, Robust verification of stochastic codes, J. Comput. Phys. (2020) (submitted)
F. Riva, C. Beadle, P. Ricci, A methodology for the rigorous verification of particle-in-cell simulations. Phys. Plasmas 24, 055703 (2017)
F. Riva, P. Ricci, F. Halpern, S. Jolliet, J. Loizu, A. Mosetto, Verification methodology for plasma simulations and application to a scrape-off layer turbulence code. Phys. Plasmas 21, 062301 (2014)
W.L. Oberkampf, C.J. Roy, Verification and Validation in Scientific Computing (Cambridge University Press, Cambridge, 2012)
W. Rider, W. Witkowski, J. Kamm, T. Wildey, Robust verification analysis. J. Comput. Phys. 307, 146–163 (2016)
A. Bobylev, The exact solutions of Boltzmann’s equations. Dokl. Akad. Nauk. 225, 1296–1299 (1975)
M. Krook, T. Wu, Exact solutions of the Boltzmann equation. Phys. Fluids 20, 1589–1595 (1977)
G. Hagelaar, L. Pitchford, Solving the Boltzmann equation to obtain electron transport coefficients and rate coefficients for fluid models. Plasma Sources Sci. Technol. 14, 722 (2005)
W. Wagner, A convergence proof for Bird’s Direct Simulation Monte Carlo method for the Boltzmann equation. J. Stat. Phys. 66, 1011 (1992)
L.C. Pitchford, J.-P. Boeuf, SIGLO database. www.lxcat.net. Accessed 11 Nov 2013
A.V. Phelps, L.C. Pitchford, Anisotropic scattering of electrons by N2 and its effect on electron transport. Phyis. Rev. A 31(5), 2932–2949 (1985)
Z.M. Raspopović, S. Sakadžíc, S.A. Bzenić, Z.L. Petrović, Benchmark calculations for Monte Carlo simulations of electron transport. IEEE Trans. Plasma Sci. 27(5), 1241 (1999)
L. Pitchford, A. Phelps, Comparitive calculations of electron-swarm properties in N2 at moderate E/N values. Phys. Rev. A 25, 540–554 (1982)
D. Bohm, The Characteristics of Electrical Discharges in Magnetic Fields (MacGraw-Hill, New York, 1949), p. 77
S.D. Baalrud, C.C. Hegna, J.D. Callen, Instability-enhanced collisional friction can determine the Bohm criterion in multiple-ion-species plasmas. Phys. Rev. Lett. 103(20), 205002 (2009)
S.D. Baalrud, C.C. Hegna, Kinetic theory of the presheath and the Bohm criterion. Plasma Sources Sci. Technol. 20, 025013 (2011)
K.-U. Riemann, The Bohm criterion and sheath formation. J. Phys. D Appl. Phys. 24(4), 493 (1991)
R.L. Merlino, Understanding Langmuir probe current–voltage characteristics. Am. J. Phys. 75, 1078 (2007)
S.D. Baalrud, N. Hershkowitz, B. Longmier, Global nonambipolar flow: plasma confinement where all electrons are lost to one boundary and all positive ions to another boundary. Phys. Plasmas 14(4), 042109 (2007)
E.V. Barnat, G.R. Laity, S.D. Baalrud, Response of the plasma to the size of an anode electrode biased near the plasma potential. Phys. Plasmas 21(10), 103512 (2014)
M.M. Hopkins, B.T. Yee, S.D. Baalrud, E.V. Barnat, The onset of plasma potential locking. Phys. Plasmas 23(6), 063519 (2016)
J.P. Sheehan, Y. Raitses, N. Hershkowitz, I. Kaganovich, N.J. Fisch, A comparison of emissive probe techniques for electric potential measurements in a complex plasma. Phys. Plasmas 18, 073501 (2011)
E.V. Barnat, K. Frederickson, Two-dimensional mapping of electron densities and temperatures using laser-collisional induced fluorescence. Plasma Sources Sci. Technol. 19, 055015 (2010)
E. Barnat, A. Fierro, Ultrafast laser-collision-induced fluorescence in atmospheric pressure plasma. J. Phys. D Appl. Phys. 50, 14LT01 (2017)
B.T. Yee, B. Scheiner, S.D. Baalrud, E.V. Barnat, M.M. Hopkins, Electron presheaths: the outsized influence of positive boundaries on plasmas. Plasma Sources Sci. Technol. 26(2), 025009 (2017)
B. Scheiner, S.D. Baalrud, B.T. Yee, M.M. Hopkins, E.V. Barnat, Theory of the electron sheath and presheath. Phys. Plasmas 22, 123520 (2015)
B. Scheiner, S.D. Baalrud, M.M. Hopkins, B.T. Yee, E.V. Barnat, Particle-in-cell study of the ion-to-electron sheath transition. Phys. Plasmas 23, 083510 (2016)
N. Nakano, N. Shimura, Z.L. Petrović, T. Makabe, Simulation of rf glow discharges in SF6 by the relaxation continuum model: Physical structure and function of the narrow-gap reactive ion etcher. Phys. Rev. E 49, 4455–4465 (1994)
B. Scheiner, E.V. Barnat, S.D. Baalrud, M.M. Hopkins, B.T. Yee, Measurements of the fireball onset. Phys. Plasmas 25, 043513 (2018)
B. Scheiner, E.V. Barnat, S.D. Baalrud, M.M. Hopkins, B.T. Yee, Theory and simulation of anode spots in low pressure plasmas. Phys. Plasmas 24, 113520 (2017)
J. Krile, A. Neuber, H. Krompholz, Effects of UV illumination on surrace flashover under pulsed excitation. IEEE Trans. Plasma Sci. 36, 332–340 (2008)
G. Edmiston, A. Neuber, H. Krompholz, J. Krile, Seed electron production from O- ions under high-power microwave excitation. J. Appl. Phys. 103, 063303 (2008)
L. Christophorou, L. Pinnaduwage, Basic physics of gaseous dielectrics. IEEE Trans. Electr. Insul. 25, 55–74 (1990)
A. Sun, J. Teunissen, U. Ebert, The inception of pulsed discharges in air: simulations in background fields above and below breakdown. J. Phys. D Appl. Phys. 47, 445205 (2014)
M. Hopkins, A. Fierro, G. Nail, E. Barnat, The role of metastables in discharge evolution. in Gaseous Electronics Conference, Portland (2018)
A. Fierro, J. Stephens, S. Beeson, J. Dickens, A. Neuber, Discrete photon implementation for plasma simulations. Phys. Plasmas 23, 013506 (2016)
A. Fierro, E. Barnat, C. Moore, M. Hopkins, P. Clem, Kinetic simulation of a low-pressure helium discharge with comparison to experimental measurements. Plasma Sources Sci. Technol. 28, 055012 (2019). (In review)
A. Dogariu, B.M. Goldberg, S. O’Byrne, R.B. Miles, Species-independent femtosecond localized electric field measurement. Phys. Rev. Appl. 7(2), 024024 (2017)
S. Roy, J.R. Gord, A.K. Patnaik, Recent advances in coherent anti-Stokes Raman scattering spectroscopy: fundamental developments and applications in reacting flows. Prog. Energy Combust. Sci. 36(2), 280–306 (2010)
W.R. Lempert, S.P. Kearney, E.V. Barnat, Diagnostic study of four-wave-mixing-basedelectric-field measurements in high-pressure nitrogen plasmas. Appl. Optics 50(29), 5688–5694 (2011)
A.K. Patnaik, I. Adamovich, J.R. Gord, S. Roy, Recent advances in ultrafast-laser-based spectroscopy and imaging for reacting plasmas and flames. Plasma Sources Sci. Technol. 26(10), 103001 (2017)
H. Arnolds, M. Bonn, Ultrafast surface vibrational dynamics. Surf. Sci. Rep. 65, 45–66 (2010)
T. Kondo, T. Ito, Flipping water molecules at insulator/solution interface using an externally applied weak electric field. Appl. Phys. Lett. 104(10), 101601 (2014)
S.E. Sanders, H. Vanselous, P.B. Petersen, Water at surfaces with tunable surface chemistries. J. Phys. Condens. Matter 30(11), 113001 (2018)
J.P. Marangos, Development of high harmonic generation spectroscopy of organic molecules and biomolecules. J. Phys. B Atomic Mol. Opt. Phys. 49(13), 132001 (2016)
J. Plewa, O. Eichwald, O. Ducasse, P. Dessante, C. Jacobs, N. Renon, M. Yousfi, 3D streamers simulation in a pin to plane configuration using massively parallel computing. J. Phys. D Appl. Phys. 51, 095206 (2018)
J. Teunissen, U. Ebert, 3D PIC-MCC simulations of discharge inception around a shapre anode in nitrogen/oxygen mixtures. Plasma Sources Sci. Technol. 25, 044005 (2016)
A. Jindal, C. Moore, A. Fierro, M. Hopkins, 3D streamer evolution in an azimuthally swept pin-to-plane wedge geometry using a PIC-DSMC code, in 71st Gaseous Electronics Conference, (2018)
S. Zabelok, R. Arslanbekov, V. Kolobov, Adaptive kinetic-fluid solvers for heterogeneous computing architectures. J. Comput. Phys. 303, 455 (2015)
A. Fierro, J. Dickens, A. Neuber, Graphics prcessing unit accelerated three-dimensional model for the simulation of pulsed low-temperature plasmas. Phys. Plasmas 21, 123504 (2014)
R. Jambunathan, D. Levin, CHAOS: An octree-based PIC-DSMC code for modeling of electron kinetic properties in a plasma plume using MPI-CUDA parallelization. J. Comput. Phys. 373, 571–604 (2018)
B. Chaudhury, A. Gupta, H. Shah, S. Bhadani, Accelerated simulation of microwave breakdown in gases on Xeon Phi based cluster-application to self-organized plasma pattern formation. Comput. Phys. Commun. 229, 20–35 (2018)
P. Mertmann, D. Eremin, T. Mussenbrock, R. Brinkmann, P. Awakowicz, Fine-sorting one-dimensional particle-in-cell algorithm with Monte-Carlo collisions on a graphics processing unit. Comput. Phys. Commun. 182, 2161–2167 (2011)
J. Stephens, M. Abide, A. Fierro, A. Neuber, Practical considerations for modeling streamer discharges in air with radiation transport. Plasma Sources Sci. Technol. 27, 075007 (2018)
G. Chen, L. Chacon, D. Barnes, An energy- and charge-conserving, implicit, electrostatic particle-in-cell algorithm. J. Comput. Phys. 18, 7018–7036 (2011)
G. Lapenta, Exactly energy conserving semi-implicit particle in cell formulation. J. Comput. Phys. 334, 349–366 (2017)
R. Marskar, Adaptive multiscale methods for 3D streamer discharges in air. Plasma Res. Express 1, 015011 (2019)
T. Pointon, Second-order, exact charge conservation for electromagnetic particle-in-cell simulation in complex geometry. Comput. Phys. Commun. 179, 535–544 (2008)
J. Teunissen, U. Ebert, Afivo: A framework for quadtree/octree AMR with shared-memory parallelization and geometric multigrid methods. Comput. Phys. Commun. 233, 156–166 (2018)
V. Kolobov, R. Arslanbekov, V. Aristov, A. Frolova, S. Zebelok, Unified solver for rarefied and continuum flows with adaptive mesh and algorithm refinement. J. Comput. Phys. 223, 589 (2007)
R. Arslanbekov, V. Kolobov, Advaptive kinetic-fluid models for expanding plasmas. J. Phys. Conf. Ser. 1031, 012018 (2018)
C. Li, U. Ebert, W. Hundsdorfer, Spatially hybrid computations for streamer discharges with generic features of pulled fronts: I. Planar fronts. J. Comput. Phys. 229, 200–220 (2010)
Acknowledgements
This work was supported by the Department of Energy Office of Fusion Energy Sciences at the U.S. Department of Energy under contract No. DE- AC04-94SL85000 Support was also provided by Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program under project 209241. Additional support was provided by the Office of Defense Nuclear Nonproliferation Research and Development. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Author information
Authors and Affiliations
Contributions
Conceptualization, M.H. and E.B.; formal analysis, A.F., E.B., M.H., C.M., G.R., and B.Y.; funding acquisition, M.H. and E.B.; investigation, A.F., E.B., M.H., C.M., G.R., and B.Y.; visualization, A.F., E.B., M.H., C.M., G.R., and B.Y.; writing-original draft preparation, A.F., E.B., M.H., C.M., G.R., and B.Y.; writing-review and editing, A.F., E.B., M.H., C.M., G.R., and B.Y. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Rights and permissions
About this article
Cite this article
Fierro, A., Barnat, E., Hopkins, M. et al. Challenges and opportunities in verification and validation of low temperature plasma simulations and experiments. Eur. Phys. J. D 75, 151 (2021). https://doi.org/10.1140/epjd/s10053-021-00088-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1140/epjd/s10053-021-00088-6