Evaluating Bioassessment Designs and Decision Thresholds Using Simulation Techniques

  • Craig D. SnyderEmail author
  • Nathaniel P. Hitt
  • David R. Smith
  • Jonathan P. Daily


Natural resource managers face numerous choices when developing bioassessment programs but seldom have the opportunity to compare the performance of alternative designs. As a result, managers often lack a basis for establishing decision thresholds based on their objectives for evaluating resource condition, accounting for uncertainty, and controlling costs. In this chapter, we illustrate how simulation techniques may be used to optimize bioassessment decision thresholds and sampling designs with a case study of benthic macroinvertebrate communities in Shenandoah National Park, USA. We evaluated the effects of sampling effort (6 levels) and taxonomic resolution (family vs. genus) on the sensitivity of a commonly used index of stream condition (Macroinvertebrate Biotic Integrity Index, MBII) to classify resource condition as affected by ecological change. We computed expected utility values to compare decision thresholds, which integrated statistical power and differential risk tolerance for misclassification (i.e., type I and II error rates). Our analysis revealed important differences among bioassessment designs. MBII sensitivity increased with sampling effort, but improvements were modest across the highest sampling levels. Genus-level assessments were generally most sensitive to ecological change, even though precision increased at the family level due to decreased variation in reference communities. However, the sensitivity-cost relationship revealed no single, optimal combination of taxonomic resolution and sampling effort. Rather, we found that for a given cost, equivalent sensitivities could be obtained from larger samples at the family-level or smaller samples at the genus level. An analysis of expected utility demonstrated that the optimal decision threshold depends on prior probability of resource condition, i.e., reference, early warning, or impaired. We conclude that simulation methods provide a flexible approach to evaluate and optimize bioassessment designs and decision thresholds based on objective-specific utility values.


Bioassessment design Metric sensitivity-cost tradeoffs Simulation Benthic macroinvertebrates Sampling effort Taxonomic resolution Decision thresholds 


  1. Andersen, T., J. Carstensen, E. Hernández-García, and C. M. Duarte. 2009. Ecological thresholds and regime shifts: Approaches to identification. Trends in Ecology and Evolution 24:49–57.CrossRefGoogle Scholar
  2. Bailey, R. C., R. H. Norris, and T. B. Reynoldson. 2001. Taxonomic resolution of benthic macroinvertebrate communities in bioassessments. Journal of the North American Benthological Society 20:280–286.CrossRefGoogle Scholar
  3. Brown, M. T., and M. B. Vivas. 2005. Landscape development intensity index. Environmental Monitoring and Assessment 101:289–309.CrossRefGoogle Scholar
  4. Burton, J., and J. Gerritsen. 2003. A stream condition index for Virginia non-coastal streams. Washington, D.C.: U. S. Environmental Protection Agency.Google Scholar
  5. Cade, B. S., and B. R. Noon. 2003. A gentle introduction to quantile regression for ecologists. Frontiers in Ecology and Evolution 1:412–420.CrossRefGoogle Scholar
  6. Cao, Y., and C. P. Hawkins. 2005. Simulating biological impairment to evaluate the accuracy of ecological indicators. Journal of Applied Ecology 42:954–965.CrossRefGoogle Scholar
  7. Capitulo, A. R., M. Tangorra, and C. Ocon. 2001. Use of benthic macroinvertebrates to assess the biological status of Pampean streams in Argentina. Aquatic Ecology 35:109–119.CrossRefGoogle Scholar
  8. Clarke, R. T., M. T. Furse, R. J. M. Gunn, J. M. Winder, and J. F. Wright. 2002. Sampling variation in macroinvertebrate data and implications for river quality indices. Freshwater Biology 47:1735–1751.CrossRefGoogle Scholar
  9. Clarke, R. T., A. Lorenz, L. Sandin, A. Schmidt-Kloiber, J. Strackbein, N. T. Kneebone, and P. Haase. 2006. Effects of sampling and sub-sampling variation using the STAR-AQEM sampling protocol on the precision of macroinvertebrate metrics. Hydrobiologia 566:441–459.CrossRefGoogle Scholar
  10. Courtemanch, D. L. 1996. Commentary on the subsampling procedures used for rapid bioassessments. Journal of the North American Benthological Society 15:381–385.CrossRefGoogle Scholar
  11. Dale, V.H., and S.C. Beyeler 2001. Challenges in the development and use of ecological indicators. Ecological Indicators 1:3–10.Google Scholar
  12. Field, S. A., A. J. Tyre, N. Jonzen, J. R. Rhodes, and H. P. Possingham. 2004. Minimizing the cost of environmental management decisions by optimizing statistical thresholds. Ecology Letters 7:669–675.CrossRefGoogle Scholar
  13. Gerritsen, J., J. Burton, and M.T. Barbour 2000. A stream condition index for West Virginia wadeable streams. Tetra Tech, Inc., Owings Mills, Maryland. Prepared for West Virgiia Department of Environmental Protection, Charleston, N.C. http://www.dep.state.WV.US/Docs/536WV-Index.pdf.
  14. Gregory, R. S., and R. L. Keeney. 2002. Making smarter environmental management decisions. Journal of the American Water Resources Association 38:1601–1612.CrossRefGoogle Scholar
  15. Growns, J. E., B. C. Chessman, J. E. Jackson, and D. G. Ross. 1997. Rapid assessment of Australian rivers using macroinvertebrates: Cost and efficiency of six methods of sample processing. Journal of the North American Benthological Society 16:682–693.CrossRefGoogle Scholar
  16. Hauer, F. R., and V. H. Resh. 2006. Macroinvertebrates. In Methods in stream ecology, eds F. R. Hauer and G. A. Lambert, 435–463. San Diego: Elsevier.Google Scholar
  17. Hawkins, C. P., R. H. Norris, J. N. Hogue, and J. W. Feminella. 2000. Development and evaluation of predictive models for measuring the biological integrity of streams. Ecological Applications 10:1456–1477.CrossRefGoogle Scholar
  18. Hawkins, C. P., Y. Cao, and B. Roper. 2010. Methods of predicting reference condition biota affects the performance and interpretation of ecological indices. Freshwater Biology 55:1066–1085.CrossRefGoogle Scholar
  19. Hering, D., O. Moog, L. Sandin, and P. F. Verdonschot. 2004. Overview and application of the AQEM assessment system. Hydrobiologia 516:1–20.CrossRefGoogle Scholar
  20. Herlihy, A. T., D. P. Larsen, S. G. Paulsen, N. S. Urquhart, and B. J. Rosenbaum. 2000. Designing a spatially bananced, randomized site selection process for regional stream surveys: The EMAP mid-Atlantic pilot study. Environmental Monitoring and Assessment 63:95–113.CrossRefGoogle Scholar
  21. Hitt, N. P., and P. L. Angermeier. 2008. River-stream connectivity affects fish bioassessment performance. Environmental Management 42:132–150.CrossRefGoogle Scholar
  22. Keeney, R. L. 1992. Value-focused thinking, a path to creative decision-making. Cambridge: Harvard University Press.Google Scholar
  23. Kennen, J. G., K. Rvia-Murray, and K. M. Beaulieu. 2010. Determining hydrologic factors that influence stream macroinvertebrate assemblages in the northeastern US. Ecohydrology 3:88–106.Google Scholar
  24. Kery, M., J. A. Royle, M. Plattner, and R. M. Dorazio. 2009. Species richness and occupancy estimation in communities subject to temporal emigration. Ecology 90:1279–1290.CrossRefGoogle Scholar
  25. King, R. S., and C. J. Richardson. 2002. Evaluating subsampling approaches and macroinvertebrate taxonomic resolution for wetland bioassessment. Journal of the North American Benthological Society 21:150–171.CrossRefGoogle Scholar
  26. Klemm, D. J., K. A. Blocksom, W. T. Thoeny, F. A. Fulk, A. T. Herlihy, P. R. Kaufmann, and S. M. Cormier. 2002. Methods development and use of macroinvertebrates as indicators of ecological conditions for streams in the mid-Atlantic Highlands region. Environmental Monitoring and Assessment 78:169–212.CrossRefGoogle Scholar
  27. Klemm, D. J., K. A. Blocksom, F. A. Fulk, A. T. Herlihy, R. M. Hughes, P. R. Kaufmann, D. V. Peck, J. L. Stoddard, W. T. Thoeny, M. B. Griffith, and W. S. Davis. 2003. Development and evaluation of a macroinvertebrate biotic integrity index (MBII) for regionally assessing mid-Atlantic highlands streams. Environmental Management 31:656–669.CrossRefGoogle Scholar
  28. Koenker, R. 2010. quantreg: Quantile regression. R package version 4.50. Last accessed on July 11, 2011.
  29. Koenker, R., and K. F. Hallock. 2001. Quantile regression. Journal of Economic Perspectives 15:143–156.CrossRefGoogle Scholar
  30. Lane, C. R., and M. T. Brown. 2006. Energy-based land use predictors of proximal factors and benthic diatom composition in Florida freshwater marshes. Environmental Monitoring and Assessment 117:433–450.CrossRefGoogle Scholar
  31. Lazorchak, J. M., D. J. Klemm, and D. V. Peck, eds. 1998. Environmental monitoring and assessment program—surface waters: Field operations and methods for measuring ecological condition of wadeable streams. EPA 620/R-94/004F. Cincinnati: U.S. Environmental Protection Agency.Google Scholar
  32. Lenat, D. R. 1993. A biotic index for the southeastern United States: Derivation and list of tolerance values, with criteria for assigning water quality ratings. Journal of the North American Benthological Society 12:279–290.CrossRefGoogle Scholar
  33. Lenat, D. R., and V. H. Resh. 2001. Taxonomy and stream ecology—the benefits of genus- and species-level identifications. Journal of the North American Benthological Society 20:287–298.CrossRefGoogle Scholar
  34. Lynch, D. D. 1987. Hydrologic conditions and trends in Shenandoah National Park, Virginia, 1983–1984. Water Resources Investigation Report 87–4131. Richmond: US Geological Survey.Google Scholar
  35. Mack, J. J. 2006. Landscape as a predictor of wetland condition: An evaluation of the landscape development intensity index (LDI) with a large reference wetland dataset from Ohio. Environmental Monitoring and Assessment 120:221–241.CrossRefGoogle Scholar
  36. MacKenzie, D. I., and W. L. Kendall. 2002. How should detection probability be incorporated into estimates of relative abundance? Ecology 83:2387–2397.CrossRefGoogle Scholar
  37. Manly, B. F. 2008. Statistics for environmental science and management. 2nd ed. Boca Raton: Chapman and Hall/CRC.Google Scholar
  38. Marchant, R. 2007. The use of taxonomic distinctness to assess environmental disturbance of insect communities from running water. Freshwater Biology 52:1634–1645.CrossRefGoogle Scholar
  39. Marshall, M., and B. Piekielek. 2007. Eastern Rivers and Mountains Network ecological monitoring plan. Natural Resource Report NPS/ERMN/NRR-2007/017.Google Scholar
  40. Marshall, J. C., A. L. Steward, and B. D. Harch. 2006. Taxonomic resolution and quantification of freshwater macroinvertebrate samples from an Australian dryland river: The benefits and costs of using abundance data. Hydrobiologia 572:171–194.CrossRefGoogle Scholar
  41. Martin, J., M. C. Runge, J. D. Nichols, B. C. Lubow, and W. L. Kendall. 2009. Structured decision making as a conceptual framework to identify thresholds for conservation management. Ecological Applications 19:1079–1090.CrossRefGoogle Scholar
  42. Mazor, R. D., T. B. Reynoldson, D. M. Rosenberg, and V. H. Resh. 2006. Effects of biotic assemblage, classification, and assessment method on bioassessment performance. Canadian Journal of Fisheries and Aquatic Sciences 63:394–411.CrossRefGoogle Scholar
  43. McCarrondand, E., and R. Frydenburg. 2005. Using the human disturbance gradient to develop bioassessment procedures in estuaries. Pages 481–492 In Estuarine indicators, ed S. A. Bortone, 481–492. Boca Raton: CRC Press.Google Scholar
  44. McCune, B., and J. B. Grace. 2002. Analysis of ecological communities. Geneden Beach: MJM Software Design.Google Scholar
  45. McElravy, E. P., G. A. Lamberti, and V. H. Resh. 1989. Year-to-year variation in the aquatic macroinvertebrate fauna of a northern California stream. Journal of the North American Benthological Society 8:51–63.CrossRefGoogle Scholar
  46. Merritt, R. W., and K. W. Cummins. 1996. An introduction to the aquatic insects of North America. 3rd ed. Dubuque: Kendall/Hunt Publishing Co.Google Scholar
  47. Moeykens, M. D., and J. R. Voshell, Jr. 2002. Studies of benthic macroinvertebrates for the Shenandoah National Park Long-term monitoring systems: Statistical analysis of LTEMS Aquatic Dataset from 1986 to 2000 on water chemistry, habitat, and macroinvertebrates. Virginia Technical Resport No. 208–11-110A-007–374-1. Blacksburg: Virginia Tech University.Google Scholar
  48. Moulton, S. R., II, J. G. Kennen, R. M. Goldstein, and J. A. Hambrook. 2002. Revised protocols for sampling algae, invertebrate, and fish communities in the National Water-Quality Assessment Program. U.S. Geological Survey Open-File Report 02–150.Google Scholar
  49. Morse, J. C., Y. J. Bae, G. Munkhjargal, N. Sangpradub, K. Tanida, T. S. Vshivhova, B. Wang, L. Yang, and C. M. Yule. 2007. Freshwater biomonitoring with macroinvertebrates in east Asia. Frontiers in Ecology and the Environment 5:33–42.CrossRefGoogle Scholar
  50. New, T. R. 1996. Taxonomic focus and quality control in insect surveys for biodiversity conservation. Australian Journal of Entomology 35:97–106.CrossRefGoogle Scholar
  51. Nichols, J. D., and B. K. Williams. 2006. Monitoring for conservation. Trends in Ecology and Evolution 21:668–673.CrossRefGoogle Scholar
  52. Nichols, S. J., W. A. Robinson, and R. H. Norris. 2006. Sample variability influences on the precision of predictive bioassessment. Hydrobiologia 572: 215–233.CrossRefGoogle Scholar
  53. Odum, H. T. 1988. Self-organization, transformity, and information. Science 242:1132–1139.CrossRefGoogle Scholar
  54. Odum, H. T. 1996. Environmental accounting: Energy and decision making. New York: Wiley.Google Scholar
  55. Ollis, D. J., H. F. Dallas, K. J. Esler, and C. Boucher. 2006. Bioassessment of the ecological integrity of river ecosystems using aquatic macroinvertebrates: An overview with a focus on South Africa. African Journal of Aquatic Sciences 31:205–227.CrossRefGoogle Scholar
  56. Ostermiller, J. D., and C. P. Hawkins. 2004. Effects of sampling error on bioassessments of stream ecosystems: Application to RIVPACS-type models. Journal of the North American Benthological Society 23:363–382.CrossRefGoogle Scholar
  57. Plafkin, J. L., M. T. Barbour, K. D. Porter, S. K. Gross, and R. M. Hughes. 1989. Rapid bioassessment protocols for use in streams and rivers, benthic macroinvertebrates and fish. Washington, D.C.: U. S. Environmental Protection Agency.Google Scholar
  58. Pond, G.J., M.E. Passmore, F.A. Borsuk, L. Reynolds, and C.J. Rose. 2008. Downstream effects of mountaintop coal mining: comparing biological conditions using family- and genus-level macroinvertebrate bioassessment tools. Journal of the North American Benthological Society 27:717–737.Google Scholar
  59. Purcell, A. H., D. W. Bressler, M. J. Paul, M. T. Barbour, E. T. Rankin, J. L. Carter, and V. H. Resh. 2009. Assessment tools for urban catchments: Developing biological indicators based on benthic macroinvertebrates. Journal of the American Water Resources Association 45:306–319.CrossRefGoogle Scholar
  60. Qian, S. S., R. S. King, and C. J. Richardson. 2003. Two statistical methods for the detection of environmental thresholds. Ecological Modelling 166:87–97.CrossRefGoogle Scholar
  61. Quinn, J. M., and C. W. Hickey. 1990. The magnitude of the effects of substrate particle size, recent flooding, and catchment development on benthic invertebrates in 88 New Zealand rivers. New Zealand Journal of Marine and Freshwaters 24:411–427.CrossRefGoogle Scholar
  62. Reynoldson, T. B., R. H., Norris, V. H. Resh, K. E. Day, and D. M. Rosenberg. 1997. The reference condition: A comparison of multimetric and multivariate approaches to assess water quality impairment using benthic macroinvertebrates. Journal of the North American Benthological Society 16:833–852.CrossRefGoogle Scholar
  63. Roy, A. H., A. D. Rosemond, M. J. Paul, D. S. Leigh, and J. B. Wallace. 2003. Stream macroinvertebrate response to catchment urbanization (Georgia, USA). Freshwater Biology 48:329–346.CrossRefGoogle Scholar
  64. Scharf, F. S., F. Juanes, and M. Sutherland. 1998. Inferring ecological relationships from the edges of scatter diagrams: Comparison of regression techniques. Ecology 79: 448–460.CrossRefGoogle Scholar
  65. Snyder, C. D., J. A. Young, R. Villella, and D. P. Lemarie. 2003. Influences of upland and riparian land use patterns on stream biotic integrity. Landscape Ecology 18:647–664.CrossRefGoogle Scholar
  66. Sonderegger, D. L., H. Wang, W. H. Clements, and B. R. Noon. 2009. Using SiZer to detect thresholds in ecological data. Frontiers in Ecology and the Environment 7:190–195.CrossRefGoogle Scholar
  67. Sovell, L. A., and B. Vondracek. 1999. Evaluation of the fixed-count method for Rapid Bioassessment Protocol III with benthic macroinvertebrate metrics. Journal of the North American Benthological Society 18:420–426.CrossRefGoogle Scholar
  68. Summers, J. K., and K. E. Tonnessen. 1998. Linking monitoring and effects research: EMAP’s intensive site network program. Environmental Monitoring and Assessment 51:369–380.CrossRefGoogle Scholar
  69. Van Sickle, J., C. P. Hawkins, D. P. Larsen, and A. T. Herlihy. 2005. A null model for the expected macroinvertebrate assemblage in streams. Journal of the North American Benthological Society 24:178–191.CrossRefGoogle Scholar
  70. Van Sickle, J., D. P. Larsen, and C. P. Hawkins. 2007. Exclusion of rare taxa affects performance of the O/E index in bioassessments. Journal of the North American Benthological Society 26:319–331.CrossRefGoogle Scholar
  71. Vinson, M. R., and C. P. Hawkins. 1996. Effects of sampling area and subsampling procedures on comparisons of taxa richness among streams. Journal of the North American Benthological Society 15:392–399.CrossRefGoogle Scholar
  72. Wagner, R., T. Dapper, and H. H. Schmidt. 2000. The influence of environmental variables on the abundance of aquatic insects: A comparison of ordination and artificial neural networks. Hydrobiologia 422/423:143–152CrossRefGoogle Scholar
  73. Waite, I. R., A. t. Herlihy, D. P. Larsen, and D. J. Klemm. 2000. Comparing strengths of geographic and nongeographic classifications of stream benthic macroinvertebrates in the mid-Atlantic Highlands, USA. Journal of the North American Benthological Society 19:429–441.CrossRefGoogle Scholar
  74. Waite, I. R., A. T. Herlihy, D. P. Larsen, N. S. Urquhart, and D. J. Klemm. 2004. The effects of macroinvertebrate taxonomic resolution in large landscape bioassessments: An example from the mid-Atlantic Highlands, U.S.A. Freshwater Biology 49:474–489.CrossRefGoogle Scholar
  75. Wang, L., J. Lyons, P. Kanehl, and R. Gatti. 1997. Influences of watershed land use on habitat quality and biotic integrity in Wisconsin streams. Fisheries 22:6–12.CrossRefGoogle Scholar
  76. Yoder, C. O., and E. T. Rankin. 1998. The role of biological indicators in a state water quality management process. Environmental Monitoring and Assessment 51:61–88.CrossRefGoogle Scholar
  77. Young, J. A., J., G. Fleming, W. Cass, and C. Lea. 2009. Vegetation of Shenandoah National Park in Relation to Environmental Gradients, Version 2.0. Technical Report NPS/NER/NRTR—2009/142. Philadelphia: National Park Service.Google Scholar
  78. Yuan, L. L. 2004. Assigning macroinvertebrate tolerance classifications using generalized additive models. Freshwater Biology 49:662–677.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2014

Authors and Affiliations

  • Craig D. Snyder
    • 1
    Email author
  • Nathaniel P. Hitt
    • 1
  • David R. Smith
    • 1
  • Jonathan P. Daily
    • 1
  1. 1.Leetown Science Center, Aquatic Ecology BranchU.S. Geological SurveyKearneysvilleUSA

Personalised recommendations