Skip to main content
Log in

Challenges and opportunities in verification and validation of low temperature plasma simulations and experiments

  • Colloquium - Plasma Physics
  • Published:
The European Physical Journal D Aims and scope Submit manuscript

Abstract

This paper describes the verification and validation (V&V) framework developed for the stochastic Particle-in-Cell, Direct Simulation Monte Carlo code Aleph. An ideal framework for V&V from the viewpoint of the authors is described where a physics problem is defined, and relevant physics models and parameters to the defined problem are assessed and captured in a Phenomena Identification and Ranking Table (PIRT). Numerous V&V examples guided by the PIRT for a simple gas discharge are shown to demonstrate the V&V process applied to a real-world simulation tool with the overall goal to demonstrably increase the confidence in the results for the simulation tool and its predictive capability. Although many examples are provided here to demonstrate elements of the framework, the primary goal of this work is to introduce this framework and not to provide a fully complete implementation, which would be a much longer document. Comparisons and contrasts are made to more usual approaches to V&V, and techniques new to the low-temperature plasma community are introduced. Specific challenges relating to the sufficiency of available data (e.g., cross sections), the limits of ad hoc validation approaches, the additional difficulty of utilizing a stochastic simulation tool, and the extreme cost of formal validation are discussed.

Graphic Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data Availability Statement

This manuscript has no associated data or the data will not be deposited. [Authors’ comment:...].

References

  1. L. Schwer, An overview of the PTC 60/V&V 10: guide for verification and validation in computational solid mechanics. Eng. Comput. 23, 245–252 (2007)

    Article  Google Scholar 

  2. M. Turner, Workshop on the verification and validation of computer simulations in low temperature plasma physics. in Gaseous Electronics Conference (2012)

  3. M. Turner, Verification of particle-in-cell simulations with Monte Carlo collisions. Plasma Sources Sci. Technol. 25, 054007 (2016)

    Article  ADS  Google Scholar 

  4. M. Turner, A. Derzsi, Z. Donko, D. Eremin, S. Kelly, T. Lafleur, T. Mussenbrock, Simulation benchmarks for low-pressure plasmas: capacitive discharges. Phys. Plasmas 20, 013507 (2013)

    Article  ADS  Google Scholar 

  5. B. Bagheri, J. Teunissen, U. Ebert, M. Becker, S. Chen, O. Ducasse, O. Eichwald, D. Loffhagen, A. Luque, D. Mihailova, J. Plewa, J. van Dijk, M. Yousfi, Comparison of six simulation codes for positive streamers in air. Plasma Sources Sci. Technol. 27, 095002 (2018)

    Article  ADS  Google Scholar 

  6. M. Surendra, Radiofrequency discharge Benchmark model comparison. Plasma Sources Sci. Technol. 4, 56 (1995)

    Article  ADS  Google Scholar 

  7. T. Trucano, M. Pilch, W. Oberkampf, On the role of code comparisons in verification and validation. Sandia National Laboratories SAND2003–2752, Albuquerque (2003)

  8. J. Carlsson, A. Khrabrov, I. Kaganovich, T. Sommerer, D. Keating, Validation and bechmarking of two particle-in-cell codes for a glow discharge. Plasma Sources Sci. Technol. 26, 014003 (2016)

    Article  ADS  Google Scholar 

  9. A. Derzsi, T. Lafleur, J. Booth, I. Korolov, Z. Donko, Experimental and simulation study of a capacitively coupled oxygen discharge dirven by tailored voltage waveforms. Plasma Sources Sci. Technol. 25, 015004 (2015)

    Article  ADS  Google Scholar 

  10. A. Derzsi, B. Bruneau, A. Gibson, E. Johnson, D. O’connell, T. Gans, J. P. Booth, Z. Donko, , Power coupling mode transitions induced by tailored voltage waveforms in capacitive oxygen discharges. Plasma Sources Sci. Technol. 26, 034002 (2017)

  11. R. Hood, B. Scheiner, S. Baalrud, M. Hopkins, E. Barnat, B. Yee, R. Merlino, F. Skiff, Ion flow and sheath structure near positively biased electrodes. Phys. Plasmas 23, 113503 (2016)

    Article  ADS  Google Scholar 

  12. Z. Donko, A. Derzsi, P. Hartmann, S. Brandt, J. Schulze, B. Berger, M. Koepke, B. Bruneau, E. Johnson, T. Lafleur, J.P. Booth, A. Gibson, D. O’Connell, T. Gans, Experimental benchmark of kinetic simulations of capacitively coupled plasmas in molecular gases. Plasma Phys. Controll. Fusion 60, 014010 (2018)

    Article  ADS  Google Scholar 

  13. M. Turner, Uncertainty and error in complex plasma chemistry models. Plasma Sources Sci. Technol. 24, 035027 (2015)

    Article  ADS  Google Scholar 

  14. M. Turner, Uncertainty and sensitivity analysis in complex plasma chemistry models. Plasma Sources Sci. Technol. 25, 015003 (2016)

    Article  ADS  Google Scholar 

  15. A. Fierro, C. Moore, B. Yee, M. Hopkins, Three-dimensional kinetic modeling of streamer propagation in a nitrogen/helium gas mixture. Plasma Sources Sci. Technol. 27, 105008 (2018)

    Article  ADS  Google Scholar 

  16. A.S. Fierro, C. Moore, B. Scheiner, B.T. Yee, M.M. Hopkins, Radiation transport in kinetic simulations and the influence of photoemission on electron current in self-sustaining discharges. J. Phys. D Appl. Phys. 50, 065202 (2017)

    Article  ADS  Google Scholar 

  17. C. Birdsall, A. Langdon, Plasma Physica via Computer Simulation (McGraw-Hill, New York, 2005)

    Google Scholar 

  18. G. Bird, Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Oxford University Press, Oxford, 1994)

    Google Scholar 

  19. M. Hopkins, P. Crozier, C. Moore, Comparison of Aleph and BOLSIG\(+\)Results for Electron-Nitrogen Chemistry. Sandia National Laboratories SAND2014-19653, Albuquerque (2014)

  20. G. Radtke, L. Musson, K. Cartwright, Error Estimation for Aleph PIC Plasma Sheath Simulations. Sandia National Laboratories SAND2012-5195C, Albuqerque (2012)

  21. C. Roark, P. Stoltz, Aleph Verification Simulations. Sandia National Laboratories SAND2012-5192C, Albuqerque (2012)

  22. P. Crozier, P. Stewart, Arc Simulations with Aleph. Sandia National Laboratories SAND2010-6564C, Albuquerque (2010)

  23. M. Bettencourt, Aleph Code Electrostatic Solver Verification. Sandia National Laboratories SAND2015-0339, Albuquerque (2015)

  24. R. Hooper, S. Moore, Aleph Field Solver Challenge Problem Results Summary. Sandia National Laboratories SAND2015-0317, Albuqeruqe (2015)

  25. H. Timko, P.S. Crozier, M.M. Hopkins, K. Matyash, R. Schneider, Why perform Code-to\_code comparisons: a vacuum Arc discharge simulation case study. Contrib. Plasma Phys. 52(4), 295–308 (2012)

    Article  ADS  Google Scholar 

  26. J. Boerner, J. Pacheco, A. Grillet, Evaluation of the Aleph PIC code on Benchmark simulations, in Gaseous Electronics Conference, Bochum (2016)

  27. T.J. Oliver, S.P. Nowlen, A Phenomena Identification and Ranking Table (PIRT) Exercise for Nuclear Power Plant Fire Modeling Applications. US Nuclear Regulatory Commission, NUREG/CR-6978 and SAND2008-3997P (2008)

  28. M. Heroux, R. Bartlett, V. Howle, R. Hoekstra, J. Hu, T. Kolda, R. Lehoucq, K. Long, R. Pawlowski, E. Phipps, A. Salinger, H. Thornquist, R. Tuminaro, J. Willenbring, A. Williams, K. Stanley, An overview of the Trilinos project. ACM Trans. Math. Softw. 31, 13 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. G. Radtke, N. Martin, C. Moore, A. Huang, K. Cartwright, Robust verification of stochastic codes, J. Comput. Phys. (2020) (submitted)

  30. F. Riva, C. Beadle, P. Ricci, A methodology for the rigorous verification of particle-in-cell simulations. Phys. Plasmas 24, 055703 (2017)

    Article  ADS  Google Scholar 

  31. F. Riva, P. Ricci, F. Halpern, S. Jolliet, J. Loizu, A. Mosetto, Verification methodology for plasma simulations and application to a scrape-off layer turbulence code. Phys. Plasmas 21, 062301 (2014)

    Article  ADS  MATH  Google Scholar 

  32. W.L. Oberkampf, C.J. Roy, Verification and Validation in Scientific Computing (Cambridge University Press, Cambridge, 2012)

    MATH  Google Scholar 

  33. W. Rider, W. Witkowski, J. Kamm, T. Wildey, Robust verification analysis. J. Comput. Phys. 307, 146–163 (2016)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  34. A. Bobylev, The exact solutions of Boltzmann’s equations. Dokl. Akad. Nauk. 225, 1296–1299 (1975)

    ADS  MathSciNet  Google Scholar 

  35. M. Krook, T. Wu, Exact solutions of the Boltzmann equation. Phys. Fluids 20, 1589–1595 (1977)

    Article  ADS  MATH  Google Scholar 

  36. G. Hagelaar, L. Pitchford, Solving the Boltzmann equation to obtain electron transport coefficients and rate coefficients for fluid models. Plasma Sources Sci. Technol. 14, 722 (2005)

    Article  ADS  Google Scholar 

  37. W. Wagner, A convergence proof for Bird’s Direct Simulation Monte Carlo method for the Boltzmann equation. J. Stat. Phys. 66, 1011 (1992)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  38. L.C. Pitchford, J.-P. Boeuf, SIGLO database. www.lxcat.net. Accessed 11 Nov 2013

  39. A.V. Phelps, L.C. Pitchford, Anisotropic scattering of electrons by N2 and its effect on electron transport. Phyis. Rev. A 31(5), 2932–2949 (1985)

    Article  ADS  Google Scholar 

  40. Z.M. Raspopović, S. Sakadžíc, S.A. Bzenić, Z.L. Petrović, Benchmark calculations for Monte Carlo simulations of electron transport. IEEE Trans. Plasma Sci. 27(5), 1241 (1999)

    Article  ADS  Google Scholar 

  41. L. Pitchford, A. Phelps, Comparitive calculations of electron-swarm properties in N2 at moderate E/N values. Phys. Rev. A 25, 540–554 (1982)

    Article  ADS  Google Scholar 

  42. D. Bohm, The Characteristics of Electrical Discharges in Magnetic Fields (MacGraw-Hill, New York, 1949), p. 77

    Google Scholar 

  43. S.D. Baalrud, C.C. Hegna, J.D. Callen, Instability-enhanced collisional friction can determine the Bohm criterion in multiple-ion-species plasmas. Phys. Rev. Lett. 103(20), 205002 (2009)

    Article  ADS  Google Scholar 

  44. S.D. Baalrud, C.C. Hegna, Kinetic theory of the presheath and the Bohm criterion. Plasma Sources Sci. Technol. 20, 025013 (2011)

    Article  ADS  Google Scholar 

  45. K.-U. Riemann, The Bohm criterion and sheath formation. J. Phys. D Appl. Phys. 24(4), 493 (1991)

    Article  ADS  Google Scholar 

  46. R.L. Merlino, Understanding Langmuir probe current–voltage characteristics. Am. J. Phys. 75, 1078 (2007)

    Article  ADS  Google Scholar 

  47. S.D. Baalrud, N. Hershkowitz, B. Longmier, Global nonambipolar flow: plasma confinement where all electrons are lost to one boundary and all positive ions to another boundary. Phys. Plasmas 14(4), 042109 (2007)

    Article  ADS  Google Scholar 

  48. E.V. Barnat, G.R. Laity, S.D. Baalrud, Response of the plasma to the size of an anode electrode biased near the plasma potential. Phys. Plasmas 21(10), 103512 (2014)

    Article  ADS  Google Scholar 

  49. M.M. Hopkins, B.T. Yee, S.D. Baalrud, E.V. Barnat, The onset of plasma potential locking. Phys. Plasmas 23(6), 063519 (2016)

    Article  ADS  Google Scholar 

  50. J.P. Sheehan, Y. Raitses, N. Hershkowitz, I. Kaganovich, N.J. Fisch, A comparison of emissive probe techniques for electric potential measurements in a complex plasma. Phys. Plasmas 18, 073501 (2011)

    Article  ADS  Google Scholar 

  51. E.V. Barnat, K. Frederickson, Two-dimensional mapping of electron densities and temperatures using laser-collisional induced fluorescence. Plasma Sources Sci. Technol. 19, 055015 (2010)

    Article  ADS  Google Scholar 

  52. E. Barnat, A. Fierro, Ultrafast laser-collision-induced fluorescence in atmospheric pressure plasma. J. Phys. D Appl. Phys. 50, 14LT01 (2017)

    Article  Google Scholar 

  53. B.T. Yee, B. Scheiner, S.D. Baalrud, E.V. Barnat, M.M. Hopkins, Electron presheaths: the outsized influence of positive boundaries on plasmas. Plasma Sources Sci. Technol. 26(2), 025009 (2017)

    Article  ADS  Google Scholar 

  54. B. Scheiner, S.D. Baalrud, B.T. Yee, M.M. Hopkins, E.V. Barnat, Theory of the electron sheath and presheath. Phys. Plasmas 22, 123520 (2015)

    Article  ADS  Google Scholar 

  55. B. Scheiner, S.D. Baalrud, M.M. Hopkins, B.T. Yee, E.V. Barnat, Particle-in-cell study of the ion-to-electron sheath transition. Phys. Plasmas 23, 083510 (2016)

    Article  ADS  Google Scholar 

  56. N. Nakano, N. Shimura, Z.L. Petrović, T. Makabe, Simulation of rf glow discharges in SF6 by the relaxation continuum model: Physical structure and function of the narrow-gap reactive ion etcher. Phys. Rev. E 49, 4455–4465 (1994)

    Article  ADS  Google Scholar 

  57. B. Scheiner, E.V. Barnat, S.D. Baalrud, M.M. Hopkins, B.T. Yee, Measurements of the fireball onset. Phys. Plasmas 25, 043513 (2018)

    Article  ADS  Google Scholar 

  58. B. Scheiner, E.V. Barnat, S.D. Baalrud, M.M. Hopkins, B.T. Yee, Theory and simulation of anode spots in low pressure plasmas. Phys. Plasmas 24, 113520 (2017)

    Article  ADS  Google Scholar 

  59. J. Krile, A. Neuber, H. Krompholz, Effects of UV illumination on surrace flashover under pulsed excitation. IEEE Trans. Plasma Sci. 36, 332–340 (2008)

    Article  ADS  Google Scholar 

  60. G. Edmiston, A. Neuber, H. Krompholz, J. Krile, Seed electron production from O- ions under high-power microwave excitation. J. Appl. Phys. 103, 063303 (2008)

    Article  ADS  Google Scholar 

  61. L. Christophorou, L. Pinnaduwage, Basic physics of gaseous dielectrics. IEEE Trans. Electr. Insul. 25, 55–74 (1990)

    Article  Google Scholar 

  62. A. Sun, J. Teunissen, U. Ebert, The inception of pulsed discharges in air: simulations in background fields above and below breakdown. J. Phys. D Appl. Phys. 47, 445205 (2014)

    Article  Google Scholar 

  63. M. Hopkins, A. Fierro, G. Nail, E. Barnat, The role of metastables in discharge evolution. in Gaseous Electronics Conference, Portland (2018)

  64. A. Fierro, J. Stephens, S. Beeson, J. Dickens, A. Neuber, Discrete photon implementation for plasma simulations. Phys. Plasmas 23, 013506 (2016)

    Article  ADS  Google Scholar 

  65. A. Fierro, E. Barnat, C. Moore, M. Hopkins, P. Clem, Kinetic simulation of a low-pressure helium discharge with comparison to experimental measurements. Plasma Sources Sci. Technol. 28, 055012 (2019). (In review)

    Article  ADS  Google Scholar 

  66. A. Dogariu, B.M. Goldberg, S. O’Byrne, R.B. Miles, Species-independent femtosecond localized electric field measurement. Phys. Rev. Appl. 7(2), 024024 (2017)

    Article  ADS  Google Scholar 

  67. S. Roy, J.R. Gord, A.K. Patnaik, Recent advances in coherent anti-Stokes Raman scattering spectroscopy: fundamental developments and applications in reacting flows. Prog. Energy Combust. Sci. 36(2), 280–306 (2010)

    Article  Google Scholar 

  68. W.R. Lempert, S.P. Kearney, E.V. Barnat, Diagnostic study of four-wave-mixing-basedelectric-field measurements in high-pressure nitrogen plasmas. Appl. Optics 50(29), 5688–5694 (2011)

    Article  ADS  Google Scholar 

  69. A.K. Patnaik, I. Adamovich, J.R. Gord, S. Roy, Recent advances in ultrafast-laser-based spectroscopy and imaging for reacting plasmas and flames. Plasma Sources Sci. Technol. 26(10), 103001 (2017)

    Article  ADS  Google Scholar 

  70. H. Arnolds, M. Bonn, Ultrafast surface vibrational dynamics. Surf. Sci. Rep. 65, 45–66 (2010)

    Article  ADS  Google Scholar 

  71. T. Kondo, T. Ito, Flipping water molecules at insulator/solution interface using an externally applied weak electric field. Appl. Phys. Lett. 104(10), 101601 (2014)

    Article  ADS  Google Scholar 

  72. S.E. Sanders, H. Vanselous, P.B. Petersen, Water at surfaces with tunable surface chemistries. J. Phys. Condens. Matter 30(11), 113001 (2018)

    Article  ADS  Google Scholar 

  73. J.P. Marangos, Development of high harmonic generation spectroscopy of organic molecules and biomolecules. J. Phys. B Atomic Mol. Opt. Phys. 49(13), 132001 (2016)

    Article  ADS  Google Scholar 

  74. J. Plewa, O. Eichwald, O. Ducasse, P. Dessante, C. Jacobs, N. Renon, M. Yousfi, 3D streamers simulation in a pin to plane configuration using massively parallel computing. J. Phys. D Appl. Phys. 51, 095206 (2018)

    Article  ADS  Google Scholar 

  75. J. Teunissen, U. Ebert, 3D PIC-MCC simulations of discharge inception around a shapre anode in nitrogen/oxygen mixtures. Plasma Sources Sci. Technol. 25, 044005 (2016)

    Article  ADS  Google Scholar 

  76. A. Jindal, C. Moore, A. Fierro, M. Hopkins, 3D streamer evolution in an azimuthally swept pin-to-plane wedge geometry using a PIC-DSMC code, in 71st Gaseous Electronics Conference, (2018)

  77. S. Zabelok, R. Arslanbekov, V. Kolobov, Adaptive kinetic-fluid solvers for heterogeneous computing architectures. J. Comput. Phys. 303, 455 (2015)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  78. A. Fierro, J. Dickens, A. Neuber, Graphics prcessing unit accelerated three-dimensional model for the simulation of pulsed low-temperature plasmas. Phys. Plasmas 21, 123504 (2014)

    Article  ADS  Google Scholar 

  79. R. Jambunathan, D. Levin, CHAOS: An octree-based PIC-DSMC code for modeling of electron kinetic properties in a plasma plume using MPI-CUDA parallelization. J. Comput. Phys. 373, 571–604 (2018)

    Article  ADS  MathSciNet  Google Scholar 

  80. B. Chaudhury, A. Gupta, H. Shah, S. Bhadani, Accelerated simulation of microwave breakdown in gases on Xeon Phi based cluster-application to self-organized plasma pattern formation. Comput. Phys. Commun. 229, 20–35 (2018)

    Article  ADS  Google Scholar 

  81. P. Mertmann, D. Eremin, T. Mussenbrock, R. Brinkmann, P. Awakowicz, Fine-sorting one-dimensional particle-in-cell algorithm with Monte-Carlo collisions on a graphics processing unit. Comput. Phys. Commun. 182, 2161–2167 (2011)

    Article  ADS  Google Scholar 

  82. J. Stephens, M. Abide, A. Fierro, A. Neuber, Practical considerations for modeling streamer discharges in air with radiation transport. Plasma Sources Sci. Technol. 27, 075007 (2018)

    Article  ADS  Google Scholar 

  83. G. Chen, L. Chacon, D. Barnes, An energy- and charge-conserving, implicit, electrostatic particle-in-cell algorithm. J. Comput. Phys. 18, 7018–7036 (2011)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  84. G. Lapenta, Exactly energy conserving semi-implicit particle in cell formulation. J. Comput. Phys. 334, 349–366 (2017)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  85. R. Marskar, Adaptive multiscale methods for 3D streamer discharges in air. Plasma Res. Express 1, 015011 (2019)

    Article  ADS  Google Scholar 

  86. T. Pointon, Second-order, exact charge conservation for electromagnetic particle-in-cell simulation in complex geometry. Comput. Phys. Commun. 179, 535–544 (2008)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  87. J. Teunissen, U. Ebert, Afivo: A framework for quadtree/octree AMR with shared-memory parallelization and geometric multigrid methods. Comput. Phys. Commun. 233, 156–166 (2018)

    Article  ADS  Google Scholar 

  88. V. Kolobov, R. Arslanbekov, V. Aristov, A. Frolova, S. Zebelok, Unified solver for rarefied and continuum flows with adaptive mesh and algorithm refinement. J. Comput. Phys. 223, 589 (2007)

    Article  ADS  MATH  Google Scholar 

  89. R. Arslanbekov, V. Kolobov, Advaptive kinetic-fluid models for expanding plasmas. J. Phys. Conf. Ser. 1031, 012018 (2018)

    Article  Google Scholar 

  90. C. Li, U. Ebert, W. Hundsdorfer, Spatially hybrid computations for streamer discharges with generic features of pulled fronts: I. Planar fronts. J. Comput. Phys. 229, 200–220 (2010)

    Article  ADS  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the Department of Energy Office of Fusion Energy Sciences at the U.S. Department of Energy under contract No. DE- AC04-94SL85000 Support was also provided by Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program under project 209241. Additional support was provided by the Office of Defense Nuclear Nonproliferation Research and Development. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization, M.H. and E.B.; formal analysis, A.F., E.B., M.H., C.M., G.R., and B.Y.; funding acquisition, M.H. and E.B.; investigation, A.F., E.B., M.H., C.M., G.R., and B.Y.; visualization, A.F., E.B., M.H., C.M., G.R., and B.Y.; writing-original draft preparation, A.F., E.B., M.H., C.M., G.R., and B.Y.; writing-review and editing, A.F., E.B., M.H., C.M., G.R., and B.Y. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to M. Hopkins.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fierro, A., Barnat, E., Hopkins, M. et al. Challenges and opportunities in verification and validation of low temperature plasma simulations and experiments. Eur. Phys. J. D 75, 151 (2021). https://doi.org/10.1140/epjd/s10053-021-00088-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1140/epjd/s10053-021-00088-6

Navigation