Skip to main content

Advertisement

SpringerLink
Go to cart
  1. Home
  2. Environmental Fluid Mechanics
  3. Article
A framework for evaluating regional-scale numerical photochemical modeling systems
Download PDF
Your article has downloaded

Similar articles being viewed by others

Slider with three articles shown per slide. Use the Previous and Next buttons to navigate the slides or the slide controller buttons at the end to navigate through each slide.

On the mathematical modelling and data assimilation for air pollution assessment in the Tropical Andes

25 April 2020

O. L. Quintero Montoya, Elías D. Niño-Ruiz & Nicolás Pinel

Evaluation of EU air quality standards through modeling and the FAIRMODE benchmarking methodology

24 October 2018

Jonilda Kushta, Georgios K. Georgiou, … Jos Lelieveld

Evaluation of a new 12 km regional perturbed parameter ensemble over Europe

30 August 2021

Simon O. Tucker, Elizabeth J. Kendon, … James M. Murphy

Large-Eddy Simulation of the Atmospheric Boundary Layer

18 August 2020

Rob Stoll, Jeremy A. Gibbs, … Marc Calaf

Stochastically Perturbed Parameterizations for the Process-Level Representation of Model Uncertainties in the CMA Global Ensemble Prediction System

01 October 2022

Fei Peng, Xiaoli Li & Jing Chen

On the uncertainties introduced by land cover data in high-resolution regional simulations

06 August 2018

Alexander De Meij, G. Zittis & T. Christoudias

The challenges of air quality modelling when crossing multiple spatial scales

06 August 2019

Carlos Silveira, Joana Ferreira & Ana Isabel Miranda

The air pollution modelling system URBAIR: how to use a Gaussian model to accomplish high spatial and temporal resolutions

06 August 2021

A. P. Fernandes, S. Rafael, … M. Lopes

Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity

05 July 2018

Susannah M. Burrows, Aritra Dasgupta, … Yun Qian

Download PDF
  • Original Article
  • Open Access
  • Published: 05 March 2010

A framework for evaluating regional-scale numerical photochemical modeling systems

  • Robin Dennis1,
  • Tyler Fox2,
  • Montse Fuentes3,
  • Alice Gilliland1,
  • Steven Hanna4,
  • Christian Hogrefe5,
  • John Irwin6,
  • S. Trivikrama Rao1,
  • Richard Scheffe2,
  • Kenneth Schere1,
  • Douw Steyn7 &
  • …
  • Akula Venkatram8 

Environmental Fluid Mechanics volume 10, pages 471–489 (2010)Cite this article

  • 1685 Accesses

  • 141 Citations

  • 3 Altmetric

  • Metrics details

Abstract

This paper discusses the need for critically evaluating regional-scale (~200–2,000 km) three-dimensional numerical photochemical air quality modeling systems to establish a model’s credibility in simulating the spatio-temporal features embedded in the observations. Because of limitations of currently used approaches for evaluating regional air quality models, a framework for model evaluation is introduced here for determining the suitability of a modeling system for a given application, distinguishing the performance between different models through confidence-testing of model results, guiding model development and analyzing the impacts of regulatory policy options. The framework identifies operational, diagnostic, dynamic, and probabilistic types of model evaluation. Operational evaluation techniques include statistical and graphical analyses aimed at determining whether model estimates are in agreement with the observations in an overall sense. Diagnostic evaluation focuses on process-oriented analyses to determine whether the individual processes and components of the model system are working correctly, both independently and in combination. Dynamic evaluation assesses the ability of the air quality model to simulate changes in air quality stemming from changes in source emissions and/or meteorology, the principal forces that drive the air quality model. Probabilistic evaluation attempts to assess the confidence that can be placed in model predictions using techniques such as ensemble modeling and Bayesian model averaging. The advantages of these types of model evaluation approaches are discussed in this paper.

Download to read the full article text

Working on a manuscript?

Avoid the common mistakes

References

  1. Bachmann J (2007) Will the circle be unbroken: a history of the national ambient air quality standards. J Air Waste Manag Assoc 57: 652–697

    Article  CAS  Google Scholar 

  2. Frost GJ et al (2006) Effects of changing power plant NO x emissions on O3 in the eastern United States: proof of concept. J Geophys Res 111: D12306

    Article  CAS  Google Scholar 

  3. Gégo E, Gilliland A, Godowitch J, Rao ST, Porter PS, Hogrefe C (2008) Modeling analyses of the effects of changes in nitrogen oxides emissions from the electric power sector on ozone levels in the eastern United States. J Air Waste Manag Assoc 58: 580–588

    Article  CAS  Google Scholar 

  4. Otte TL et al (2005) Linking the Eta model with the community multiscale air quality (CMAQ) modeling system to build a national air quality forecasting system. Weather Forecast 43: 1648–1665

    Google Scholar 

  5. Mathur R (2008) Estimating the impact of the 2004 Alaskan forest fires on episodic particulate matter pollution over the eastern United States through assimilation of satellite-derived aerosol optical depths in a regional air quality model. J Geophys Res 113: D17302. doi:10.1029/2007JD009767

    Article  CAS  Google Scholar 

  6. Eder B et al (2009) A demonstration of the use of national air quality forecast guidance for developing local air quality index forecasts. Bull Am Meteorol Soc doi:10.1175/2009BAMS2734.1

  7. Hanna SR, Davis JM (2002) Evaluation of a photochemical grid model using estimates of concentration probability density functions. Atmos Environ 36: 1793–1798

    Article  CAS  Google Scholar 

  8. Fox DG (1981) Judging air quality model performance: a summary of the AMS workshop on dispersion model performance. Bull Am Meteorol Soc 62: 599–609

    Article  Google Scholar 

  9. Dabberdt WF et al (2004) Meteorological research needs for improved air quality forecasting. Bull Am Meteorol Soc 85: 563–586

    Article  Google Scholar 

  10. Steyn DG, Galmarini S (2008) Evaluating the predictive and explanatory value of atmospheric numerical models: between relativism and objectivism. Open Atmos Sci J 2: 38–45. doi:10.2174/1874282300802010038

    Article  Google Scholar 

  11. Irwin JS et al (2008) A procedure for inter-comparing the skill of regional-scale air quality model simulations of daily maximum 8-hr ozone concentrations. Atmos Environ 42: 5403–5412

    Article  CAS  Google Scholar 

  12. Weil JC, Sykes RI, Venkatram A (1992) Evaluating air quality models: review and outlook. J Appl Meteorol 31: 1121–1145

    Article  Google Scholar 

  13. Swall JL, Foley KM (2009) The impact of spatial correlation and incommensurability on model evaluation. Atmos Environ 43: 1204–1217

    Article  CAS  Google Scholar 

  14. Rao ST (2009) Environmental monitoring and modeling needs in the 21st century. EM Magazine, October

  15. Chang JC, Hanna SR (2004) Air quality model performance. Meteorol Atmos Phys 87: 167–196

    Article  Google Scholar 

  16. Rao ST, Zurbenko IG, Neagu R, Porter PS, Ku JY, Henry RF (1997) Space and time scales in ambient ozone data. Bull Am Meteorol Soc 78: 2153–2166

    Article  Google Scholar 

  17. Hogrefe C, Rao ST, Zurbenko IG, Porter PS (2000) Interpreting information in time series of ozone observations and model predictions relevant to regulatory policies in the eastern United States. Bull Am Meteorol Soc 81: 2083–2106

    Article  Google Scholar 

  18. Appel KW, Gilliland AB, Sarwar G, Gilliam RC (2007) Evaluation of the community multi-scale air quality (CMAQ) model version 4.5: sensitivities impacting model performance; part 1 Ozone. Atmos Environ 41(40): 9603–9615

    Article  CAS  Google Scholar 

  19. Gégo EL, Porter PS, Irwin JS, Hogrefe C, Rao ST (2005) Assessing the comparability of ammonium, nitrate and sulfate concentrations measured by three air quality monitoring networks. Pure Appl Geophys 162: 1919–1939

    Article  Google Scholar 

  20. Kang D, Mathur R, Rao ST, Yu S (2008) Bias-adjustment techniques for improving ozone air quality forecasts. J Geophys Res 113(D23308): 1–17

    Google Scholar 

  21. Appel KW, Bhave PV, Gilliland AB, Sarwar G, Roselle SJ (2008) Evaluation of the community multiscale air quality (CMAQ) model version 4.5: sensitivities impacting model performance; part II—particulate matter. Atmos Environ 42: 6057–6066

    Article  CAS  Google Scholar 

  22. Mathur R, Yu S, Kang D, Schere KL (2008) Assessment of the wintertime performance of developmental particulate matter forecasts with the Eta-community multiscale air quality modeling system. J Geophys Res 113: D02303. doi:10.1029/2007JD008580

    Article  CAS  Google Scholar 

  23. Morris RE, McNally DE, Tesche TW, Tonnesen G, Boylan JW, Brewer P (2005) Preliminary evaluation of the community multiscale air quality model for 2002 over the southeastern United States. J Air Waste Manag Assoc 55: 1694–1708

    CAS  Google Scholar 

  24. Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J Geophys Res 106(D7): 7183–7192

    Article  Google Scholar 

  25. Vautard R, Builtzes PHJ, Thunis P, Cuvelier C, Bedogni M, Bessagnet B, Honore C, Moussiopoulos N, Pirovano G, Schaap M, Stern R, Tarrason L, Wind P (2007) Evaluation and intercomparison of ozone and PM10 simulations by several chemistry transport models over four European cities within the CityDelta project. Atmos Environ 41: 173–188

    Article  CAS  Google Scholar 

  26. Stohl A, Hittenberger M, Wotawa G (1998) Validation of the Lagrangian particle dispersion model FLEXPART against large-scale tracer experiment data. Atmos Environ 32: 4245–4264

    Article  CAS  Google Scholar 

  27. Godowitch JM, Hogrefe C, Rao ST (2008) Diagnostic analyses of a regional air quality model: changes in modeled processes affecting ozone and chemical-transport indicators from NO_x point source emission reductions. J Geophys Res 113. doi:10.1029/2007JD009537

  28. Saltelli A, Tarantola S, Campolongo F, Ratto M (2004) Sensitivity analysis in practice, a guide to assessing scientific models. Wiley, New York

    Google Scholar 

  29. Cullen AC, Frey HC (1999) Probabilistic techniques in exposure assessment. A handbook for dealing with variability and uncertainty in models and inputs. Plenum Press, New York, 335 pp

  30. Biswas J, Rao ST (2001) Uncertainties in episodic ozone modeling stemming from uncertainties in the meteorological fields. J Appl Meteorol 40: 117–136

    Article  Google Scholar 

  31. Otte TL (2008) The impact of nudging in the meteorological model for retrospective air quality simulations. Part II: Evaluating collocated meteorological and air quality observations. J Appl Meteorol Climatol 47: 1868–1887

    Article  Google Scholar 

  32. Gilliland AB, Hogrefe C, Pinder RW, Godowitch JM, Foley KL, Rao ST (2008) Dynamic evaluation of regional air quality models: assessing changes in O3 stemming from changes in emissions and meteorology. Atmos Environ 42: 5110–5123

    Article  CAS  Google Scholar 

  33. Godowitch JM, Gilliland AB, Draxler R, Rao ST (2008) Modeling assessment of point source NO x emission reductions on ozone air quality in the eastern United States. Atmos Environ 42: 87–100

    Article  CAS  Google Scholar 

  34. Pinder RW, Adams PJ, Pandis SN, Gilliland AB (2006) Temporally resolved ammonia emission inventories: current estimates, evaluation tools, and measurement needs. J Geophys Res Atmos 111. doi:10.1029/2005JD006603

  35. Cohan DS, Hakami A, Hu Y, Russell AG (2005) Nonlinear response of ozone to emissions: source apportionment and sensitivity analysis. Environ Sci Technol 39: 6739–6748

    Article  CAS  Google Scholar 

  36. Napelenok SL, Cohan DS, Hu Y, Russell AG (2006) Decoupled direct 3D sensitivity analysis for particulate matter (DDM-3D/PM). Atmos Environ 40: 6112–6121

    Article  CAS  Google Scholar 

  37. Seaman NL (2000) Meteorological modeling for air-quality assessments. Atmos Environ 34: 2231–2259

    Article  CAS  Google Scholar 

  38. Bhave PV, Pouliot GA, Zheng M (2007) Diagnostic model evaluation for carbonaceous PM2.5 using organic markers measured in the southeastern U.S. Environ Sci Technol 41: 1577–1583

    Article  CAS  Google Scholar 

  39. Napelenok SL, Pinder RW, Gilliland AB, Martin RV (2008) A method for evaluating spatially-resolved NO x emissions using Kalman filter inversion, direct sensitivities, and space-based NO2 observations. Atmos Chem Phys 8: 6469–6499

    Article  Google Scholar 

  40. Chow JC (2003) Introduction to special topic: weekend and weekday differences in ozone levels. J Air Waste Manag Assoc 53: 771

    Google Scholar 

  41. Godowitch JM, Pouliot G, Rao ST (2009) On the use of a dynamic evaluation approach to assess multi-year change in modeled and observed urban nitrogen oxide concentrations. In: Steyn DG, Rao ST (eds) Proceedings of the 30th NATO/SPS international technical meeting on air pollution, modeling and its application, San Francisco, CA

  42. Environmental Protection Agency (2008) NOx budget trading program, EPA-430-R-08-008, 62 pp

  43. Porter PS, Rao ST, Zurbenko IG, Dunker AM, Wolff GT (2001) Ozone air quality over North America: part II—An analysis of trend detection and attribution techniques. J Air Waste Manag Assoc 51: 283–306

    CAS  Google Scholar 

  44. Camalier L, Cox W, Dolwick P (2007) The effects of meteorology on ozone in urban areas and their use in assessing ozone trends. Atmos Environ 41: 7127–7137

    Article  CAS  Google Scholar 

  45. Lewellen WS, Sykes RI, Parker SF (1985) An evaluation technique which uses the prediction of both concentration mean and variance. In: Proceedings of the DOE/AMS air pollution model evaluation workshop, Savannah river lab report number DP-1701-1, section 2, 24 pp

  46. Irwin JS, Rao ST, Petersen WB, Turner DB (1987) Relating error bounds for maximum concentration estimates to diffusion meteorology uncertainty. Atmos Environ 21: 1927–1937

    Article  CAS  Google Scholar 

  47. Galmarini S et al (2004) Ensemble dispersion forecasting, part I: concept approach, and indicators. Atmos Environ 38: 4607–4617

    Article  CAS  Google Scholar 

  48. Galmarini S et al (2004) Ensemble dispersion forecasting, part II: application and evaluation. Atmos Environ 38: 4619–4632

    Article  CAS  Google Scholar 

  49. Dabberdt WF, Miller E (2000) Uncertainty, ensembles and air quality dispersion modeling: applications and challenges. Atmos Environ 34: 4667–4673

    Article  CAS  Google Scholar 

  50. Delle Monache L, Deng X, Zhou Y, Stull R (2006) Ozone ensemble forecasts: 1 A new ensemble design. J Geophys Res 111: D05307. doi:10.1029/2005JD006310

    Article  Google Scholar 

  51. Mallet V, Sportisse B (2006) Ensemble-based air quality forecasts: a multimodel approach applied to ozone. J Geophys Res 111: D18302. doi:10.1029/2005JD006675

    Article  CAS  Google Scholar 

  52. McKeen S et al (2005) Assessment of an ensemble of seven real-time ozone forecasts over eastern North America during the summer of 2004. J Geophys Res 110: D21307. doi:10.1029/2005JD005858

    Article  CAS  Google Scholar 

  53. van Loon M et al (2007) Evaluation of long-term ozone simulations from seven regional air quality models and their ensemble. Atmos Environ 41: 2083–2097

    Article  CAS  Google Scholar 

  54. Riccio A, Giunta G, Galmarini S (2007) Seeking for the rational basis of the median model: the optimal combination of multi-model ensemble results. Atmos Chem Phys 7: 6085–6098

    Article  CAS  Google Scholar 

  55. Warner TT, Sheu R-S, Bowers JF, Sykes RI, Dodd GC, Henn DS (2002) Ensemble simulations with coupled atmospheric dynamic and dispersion models: illustrating uncertainties in dosage simulations. J Appl Meteorol 41: 488–504

    Article  Google Scholar 

  56. Hogrefe C, Rao ST (2001) Demonstrating attainment of the air quality standards: integration of observations and model predictions into the probabilistic framework. J Air Waste Manag Assoc 51: 1060–1072

    CAS  Google Scholar 

  57. Jones JM, Hogrefe C, Henry RF, Ku J-Y, Sistla G (2005) An assessment of the sensitivity and reliability of the relative reduction factor (RRF) approach in the development of 8-hr ozone attainment plans. J Air Waste Manag Assoc 55: 13–19

    CAS  Google Scholar 

  58. Hogrefe C, Civerolo KL, Hao W, Ku JY, Zalewsky EE, Sistla G (2008) Rethinking the assessment of photochemical modeling systems in air quality planning applications. J Air Waste Manag Assoc. doi:10.3155-1047-3289.58.8.1086

  59. Raftery AE, Gneiting T, Balabdaoui F, Palokowski M (2005) Using Bayesian model averaging to calibrate ensembles. Mon Weather Rev 133: 1155–1174

    Article  Google Scholar 

  60. Pinder RW, Gilliam RC, Appel KW, Napelenok SL, Gilliland AB (2009) Efficient probabilistic estimates of surface ozone concentration using an ensemble of model configurations and direct sensitivity calculations. Environ Sci Technol 43: 2388–2393

    Article  CAS  Google Scholar 

  61. Foley K, Pinder R, Napelenok S (2008) New directions in air quality model evaluation: probabilistic model evaluation. Poster presented at the CMAS Conference, Chapel Hill, NC. Available at http://www.cmascenter.org/conference/2008/agenda.cfm

  62. Rao ST, Schere KS, Galmarini S, Steyn DG (2009) AQMEII: air quality model evaluation international initiative. In: Proceedings of the 30th NATO/SPS international technical meeting on air pollution modeling and its application, San Francisco

Download references

Acknowledgements

Although this paper has been subjected to the U.S. Environmental Protection Agency review and approved for publication, it does not necessarily reflect the views and policies of the Agency.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Author information

Authors and Affiliations

  1. Atmospheric Modeling and Analysis Division, National Exposure Research Laboratory, US Environmental Protection Agency, Research Triangle Park, NC, 27711, USA

    Robin Dennis, Alice Gilliland, S. Trivikrama Rao & Kenneth Schere

  2. Air Quality Assessment Division, Office of Air Quality Planning and Standards, US Environmental Protection Agency, Research Triangle Park, NC, 27711, USA

    Tyler Fox & Richard Scheffe

  3. Department of Statistics, North Carolina State University, Raleigh, NC, 27695, USA

    Montse Fuentes

  4. Hanna Consultants, Kennebunkport, ME, 04046, USA

    Steven Hanna

  5. NYS Department of Environmental Conservation, Bureau of Air Quality Analysis and Research, Albany, NY, 12233, USA

    Christian Hogrefe

  6. John S. Irwin and Associates, Raleigh, NC, 27615, USA

    John Irwin

  7. Department of Earth and Ocean Sciences, The University of British Columbia, Vancouver, BC, V6T1Z4, Canada

    Douw Steyn

  8. Department of Mechanical Engineering, University of California, Riverside, CA, 92521, USA

    Akula Venkatram

Authors
  1. Robin Dennis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tyler Fox
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Montse Fuentes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alice Gilliland
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Steven Hanna
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Hogrefe
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. John Irwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. S. Trivikrama Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Richard Scheffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Kenneth Schere
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Douw Steyn
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Akula Venkatram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Trivikrama Rao.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Dennis, R., Fox, T., Fuentes, M. et al. A framework for evaluating regional-scale numerical photochemical modeling systems. Environ Fluid Mech 10, 471–489 (2010). https://doi.org/10.1007/s10652-009-9163-2

Download citation

  • Received: 21 August 2009

  • Accepted: 16 December 2009

  • Published: 05 March 2010

  • Issue Date: August 2010

  • DOI: https://doi.org/10.1007/s10652-009-9163-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Air quality model
  • Photochemical model
  • Model evaluation
  • Performance evaluation
Download PDF

Working on a manuscript?

Avoid the common mistakes

Advertisement

Over 10 million scientific documents at your fingertips

Switch Edition
  • Academic Edition
  • Corporate Edition
  • Home
  • Impressum
  • Legal information
  • Privacy statement
  • Your US state privacy rights
  • How we use cookies
  • Your privacy choices/Manage cookies
  • Accessibility
  • FAQ
  • Contact us
  • Affiliate program

Not affiliated

Springer Nature

© 2023 Springer Nature Switzerland AG. Part of Springer Nature.