Abstract
This paper discusses the need for critically evaluating regional-scale (~200–2,000 km) three-dimensional numerical photochemical air quality modeling systems to establish a model’s credibility in simulating the spatio-temporal features embedded in the observations. Because of limitations of currently used approaches for evaluating regional air quality models, a framework for model evaluation is introduced here for determining the suitability of a modeling system for a given application, distinguishing the performance between different models through confidence-testing of model results, guiding model development and analyzing the impacts of regulatory policy options. The framework identifies operational, diagnostic, dynamic, and probabilistic types of model evaluation. Operational evaluation techniques include statistical and graphical analyses aimed at determining whether model estimates are in agreement with the observations in an overall sense. Diagnostic evaluation focuses on process-oriented analyses to determine whether the individual processes and components of the model system are working correctly, both independently and in combination. Dynamic evaluation assesses the ability of the air quality model to simulate changes in air quality stemming from changes in source emissions and/or meteorology, the principal forces that drive the air quality model. Probabilistic evaluation attempts to assess the confidence that can be placed in model predictions using techniques such as ensemble modeling and Bayesian model averaging. The advantages of these types of model evaluation approaches are discussed in this paper.
References
Bachmann J (2007) Will the circle be unbroken: a history of the national ambient air quality standards. J Air Waste Manag Assoc 57: 652–697
Frost GJ et al (2006) Effects of changing power plant NO x emissions on O3 in the eastern United States: proof of concept. J Geophys Res 111: D12306
Gégo E, Gilliland A, Godowitch J, Rao ST, Porter PS, Hogrefe C (2008) Modeling analyses of the effects of changes in nitrogen oxides emissions from the electric power sector on ozone levels in the eastern United States. J Air Waste Manag Assoc 58: 580–588
Otte TL et al (2005) Linking the Eta model with the community multiscale air quality (CMAQ) modeling system to build a national air quality forecasting system. Weather Forecast 43: 1648–1665
Mathur R (2008) Estimating the impact of the 2004 Alaskan forest fires on episodic particulate matter pollution over the eastern United States through assimilation of satellite-derived aerosol optical depths in a regional air quality model. J Geophys Res 113: D17302. doi:10.1029/2007JD009767
Eder B et al (2009) A demonstration of the use of national air quality forecast guidance for developing local air quality index forecasts. Bull Am Meteorol Soc doi:10.1175/2009BAMS2734.1
Hanna SR, Davis JM (2002) Evaluation of a photochemical grid model using estimates of concentration probability density functions. Atmos Environ 36: 1793–1798
Fox DG (1981) Judging air quality model performance: a summary of the AMS workshop on dispersion model performance. Bull Am Meteorol Soc 62: 599–609
Dabberdt WF et al (2004) Meteorological research needs for improved air quality forecasting. Bull Am Meteorol Soc 85: 563–586
Steyn DG, Galmarini S (2008) Evaluating the predictive and explanatory value of atmospheric numerical models: between relativism and objectivism. Open Atmos Sci J 2: 38–45. doi:10.2174/1874282300802010038
Irwin JS et al (2008) A procedure for inter-comparing the skill of regional-scale air quality model simulations of daily maximum 8-hr ozone concentrations. Atmos Environ 42: 5403–5412
Weil JC, Sykes RI, Venkatram A (1992) Evaluating air quality models: review and outlook. J Appl Meteorol 31: 1121–1145
Swall JL, Foley KM (2009) The impact of spatial correlation and incommensurability on model evaluation. Atmos Environ 43: 1204–1217
Rao ST (2009) Environmental monitoring and modeling needs in the 21st century. EM Magazine, October
Chang JC, Hanna SR (2004) Air quality model performance. Meteorol Atmos Phys 87: 167–196
Rao ST, Zurbenko IG, Neagu R, Porter PS, Ku JY, Henry RF (1997) Space and time scales in ambient ozone data. Bull Am Meteorol Soc 78: 2153–2166
Hogrefe C, Rao ST, Zurbenko IG, Porter PS (2000) Interpreting information in time series of ozone observations and model predictions relevant to regulatory policies in the eastern United States. Bull Am Meteorol Soc 81: 2083–2106
Appel KW, Gilliland AB, Sarwar G, Gilliam RC (2007) Evaluation of the community multi-scale air quality (CMAQ) model version 4.5: sensitivities impacting model performance; part 1 Ozone. Atmos Environ 41(40): 9603–9615
Gégo EL, Porter PS, Irwin JS, Hogrefe C, Rao ST (2005) Assessing the comparability of ammonium, nitrate and sulfate concentrations measured by three air quality monitoring networks. Pure Appl Geophys 162: 1919–1939
Kang D, Mathur R, Rao ST, Yu S (2008) Bias-adjustment techniques for improving ozone air quality forecasts. J Geophys Res 113(D23308): 1–17
Appel KW, Bhave PV, Gilliland AB, Sarwar G, Roselle SJ (2008) Evaluation of the community multiscale air quality (CMAQ) model version 4.5: sensitivities impacting model performance; part II—particulate matter. Atmos Environ 42: 6057–6066
Mathur R, Yu S, Kang D, Schere KL (2008) Assessment of the wintertime performance of developmental particulate matter forecasts with the Eta-community multiscale air quality modeling system. J Geophys Res 113: D02303. doi:10.1029/2007JD008580
Morris RE, McNally DE, Tesche TW, Tonnesen G, Boylan JW, Brewer P (2005) Preliminary evaluation of the community multiscale air quality model for 2002 over the southeastern United States. J Air Waste Manag Assoc 55: 1694–1708
Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J Geophys Res 106(D7): 7183–7192
Vautard R, Builtzes PHJ, Thunis P, Cuvelier C, Bedogni M, Bessagnet B, Honore C, Moussiopoulos N, Pirovano G, Schaap M, Stern R, Tarrason L, Wind P (2007) Evaluation and intercomparison of ozone and PM10 simulations by several chemistry transport models over four European cities within the CityDelta project. Atmos Environ 41: 173–188
Stohl A, Hittenberger M, Wotawa G (1998) Validation of the Lagrangian particle dispersion model FLEXPART against large-scale tracer experiment data. Atmos Environ 32: 4245–4264
Godowitch JM, Hogrefe C, Rao ST (2008) Diagnostic analyses of a regional air quality model: changes in modeled processes affecting ozone and chemical-transport indicators from NO_x point source emission reductions. J Geophys Res 113. doi:10.1029/2007JD009537
Saltelli A, Tarantola S, Campolongo F, Ratto M (2004) Sensitivity analysis in practice, a guide to assessing scientific models. Wiley, New York
Cullen AC, Frey HC (1999) Probabilistic techniques in exposure assessment. A handbook for dealing with variability and uncertainty in models and inputs. Plenum Press, New York, 335 pp
Biswas J, Rao ST (2001) Uncertainties in episodic ozone modeling stemming from uncertainties in the meteorological fields. J Appl Meteorol 40: 117–136
Otte TL (2008) The impact of nudging in the meteorological model for retrospective air quality simulations. Part II: Evaluating collocated meteorological and air quality observations. J Appl Meteorol Climatol 47: 1868–1887
Gilliland AB, Hogrefe C, Pinder RW, Godowitch JM, Foley KL, Rao ST (2008) Dynamic evaluation of regional air quality models: assessing changes in O3 stemming from changes in emissions and meteorology. Atmos Environ 42: 5110–5123
Godowitch JM, Gilliland AB, Draxler R, Rao ST (2008) Modeling assessment of point source NO x emission reductions on ozone air quality in the eastern United States. Atmos Environ 42: 87–100
Pinder RW, Adams PJ, Pandis SN, Gilliland AB (2006) Temporally resolved ammonia emission inventories: current estimates, evaluation tools, and measurement needs. J Geophys Res Atmos 111. doi:10.1029/2005JD006603
Cohan DS, Hakami A, Hu Y, Russell AG (2005) Nonlinear response of ozone to emissions: source apportionment and sensitivity analysis. Environ Sci Technol 39: 6739–6748
Napelenok SL, Cohan DS, Hu Y, Russell AG (2006) Decoupled direct 3D sensitivity analysis for particulate matter (DDM-3D/PM). Atmos Environ 40: 6112–6121
Seaman NL (2000) Meteorological modeling for air-quality assessments. Atmos Environ 34: 2231–2259
Bhave PV, Pouliot GA, Zheng M (2007) Diagnostic model evaluation for carbonaceous PM2.5 using organic markers measured in the southeastern U.S. Environ Sci Technol 41: 1577–1583
Napelenok SL, Pinder RW, Gilliland AB, Martin RV (2008) A method for evaluating spatially-resolved NO x emissions using Kalman filter inversion, direct sensitivities, and space-based NO2 observations. Atmos Chem Phys 8: 6469–6499
Chow JC (2003) Introduction to special topic: weekend and weekday differences in ozone levels. J Air Waste Manag Assoc 53: 771
Godowitch JM, Pouliot G, Rao ST (2009) On the use of a dynamic evaluation approach to assess multi-year change in modeled and observed urban nitrogen oxide concentrations. In: Steyn DG, Rao ST (eds) Proceedings of the 30th NATO/SPS international technical meeting on air pollution, modeling and its application, San Francisco, CA
Environmental Protection Agency (2008) NOx budget trading program, EPA-430-R-08-008, 62 pp
Porter PS, Rao ST, Zurbenko IG, Dunker AM, Wolff GT (2001) Ozone air quality over North America: part II—An analysis of trend detection and attribution techniques. J Air Waste Manag Assoc 51: 283–306
Camalier L, Cox W, Dolwick P (2007) The effects of meteorology on ozone in urban areas and their use in assessing ozone trends. Atmos Environ 41: 7127–7137
Lewellen WS, Sykes RI, Parker SF (1985) An evaluation technique which uses the prediction of both concentration mean and variance. In: Proceedings of the DOE/AMS air pollution model evaluation workshop, Savannah river lab report number DP-1701-1, section 2, 24 pp
Irwin JS, Rao ST, Petersen WB, Turner DB (1987) Relating error bounds for maximum concentration estimates to diffusion meteorology uncertainty. Atmos Environ 21: 1927–1937
Galmarini S et al (2004) Ensemble dispersion forecasting, part I: concept approach, and indicators. Atmos Environ 38: 4607–4617
Galmarini S et al (2004) Ensemble dispersion forecasting, part II: application and evaluation. Atmos Environ 38: 4619–4632
Dabberdt WF, Miller E (2000) Uncertainty, ensembles and air quality dispersion modeling: applications and challenges. Atmos Environ 34: 4667–4673
Delle Monache L, Deng X, Zhou Y, Stull R (2006) Ozone ensemble forecasts: 1 A new ensemble design. J Geophys Res 111: D05307. doi:10.1029/2005JD006310
Mallet V, Sportisse B (2006) Ensemble-based air quality forecasts: a multimodel approach applied to ozone. J Geophys Res 111: D18302. doi:10.1029/2005JD006675
McKeen S et al (2005) Assessment of an ensemble of seven real-time ozone forecasts over eastern North America during the summer of 2004. J Geophys Res 110: D21307. doi:10.1029/2005JD005858
van Loon M et al (2007) Evaluation of long-term ozone simulations from seven regional air quality models and their ensemble. Atmos Environ 41: 2083–2097
Riccio A, Giunta G, Galmarini S (2007) Seeking for the rational basis of the median model: the optimal combination of multi-model ensemble results. Atmos Chem Phys 7: 6085–6098
Warner TT, Sheu R-S, Bowers JF, Sykes RI, Dodd GC, Henn DS (2002) Ensemble simulations with coupled atmospheric dynamic and dispersion models: illustrating uncertainties in dosage simulations. J Appl Meteorol 41: 488–504
Hogrefe C, Rao ST (2001) Demonstrating attainment of the air quality standards: integration of observations and model predictions into the probabilistic framework. J Air Waste Manag Assoc 51: 1060–1072
Jones JM, Hogrefe C, Henry RF, Ku J-Y, Sistla G (2005) An assessment of the sensitivity and reliability of the relative reduction factor (RRF) approach in the development of 8-hr ozone attainment plans. J Air Waste Manag Assoc 55: 13–19
Hogrefe C, Civerolo KL, Hao W, Ku JY, Zalewsky EE, Sistla G (2008) Rethinking the assessment of photochemical modeling systems in air quality planning applications. J Air Waste Manag Assoc. doi:10.3155-1047-3289.58.8.1086
Raftery AE, Gneiting T, Balabdaoui F, Palokowski M (2005) Using Bayesian model averaging to calibrate ensembles. Mon Weather Rev 133: 1155–1174
Pinder RW, Gilliam RC, Appel KW, Napelenok SL, Gilliland AB (2009) Efficient probabilistic estimates of surface ozone concentration using an ensemble of model configurations and direct sensitivity calculations. Environ Sci Technol 43: 2388–2393
Foley K, Pinder R, Napelenok S (2008) New directions in air quality model evaluation: probabilistic model evaluation. Poster presented at the CMAS Conference, Chapel Hill, NC. Available at http://www.cmascenter.org/conference/2008/agenda.cfm
Rao ST, Schere KS, Galmarini S, Steyn DG (2009) AQMEII: air quality model evaluation international initiative. In: Proceedings of the 30th NATO/SPS international technical meeting on air pollution modeling and its application, San Francisco
Acknowledgements
Although this paper has been subjected to the U.S. Environmental Protection Agency review and approved for publication, it does not necessarily reflect the views and policies of the Agency.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Dennis, R., Fox, T., Fuentes, M. et al. A framework for evaluating regional-scale numerical photochemical modeling systems. Environ Fluid Mech 10, 471–489 (2010). https://doi.org/10.1007/s10652-009-9163-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10652-009-9163-2
Keywords
- Air quality model
- Photochemical model
- Model evaluation
- Performance evaluation