Skip to main content
SpringerLink
Log in

Performance evaluation of problematic samples: a robust nonparametric approach for wastewater treatment plants

  • Original Research
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

This paper explores robust unconditional and conditional nonparametric approaches to support performance evaluation in problematic samples. Real-world assessments often face critical problems regarding available data, as samples may be relatively small, with high variability in the magnitude of the observed indicators and contextual conditions. This paper explores the possibility of mitigating the impact of potential outlier observations and variability in small samples using a robust nonparametric approach. This approach has the advantage of avoiding unnecessary loss of relevant information, retaining all the decision-making units of the original sample. We devote particular attention to identifying peers and targets in the robust nonparametric approach to guide improvements for underperforming units. The results are compared with a traditional deterministic approach to highlight the proposed method's benefits for problematic samples. This framework's applicability in internal benchmarking studies is illustrated with a case study within the wastewater treatment industry in Portugal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Andersen, P., & Petersen, N. C. (1993). A procedure for ranking efficient units in data envelopment analysis. Management Science, 39(10), 1261–1264.

    Article  Google Scholar 

  • Aragon, Y., Daouia, A., & Thomas-Agnan, C. (2005). Nonparametric frontier estimation: A conditional quantile-based approach. Econometric Theory, 21(2), 358–389.

    Article  Google Scholar 

  • Avkiran, N. K., & Rowlands, T. (2008). How to better identify the true managerial performance: State of the art using DEA. Omega, 36(2), 317–324.

    Article  Google Scholar 

  • Barros, C. P. (2006). Productivity Growth in the Lisbon Police Force. Public Organization Review, 6(1), 21–35.

    Article  Google Scholar 

  • Barros, C. P. (2008). Efficiency analysis of hydroelectric generating plants: A case study for Portugal. Energy Economics, 30(1), 59–75.

    Article  Google Scholar 

  • Camanho, A. S., & Dyson, R. G. (1999). Efficiency, size, benchmarks and targets for bank branches: An application of data envelopment analysis. Journal of the Operational Research Society, 50(9), 903–915.

    Article  Google Scholar 

  • Camanho, A. S., Portela, M. C., & Vaz, C. B. (2009). Efficiency analysis accounting for internal and external non-discretionary factors. Computers & Operations Research, 36(5), 1591–1601.

    Article  Google Scholar 

  • Castellet, L., & Molinos-Senante, M. (2016). Efficiency assessment of wastewater treatment plants: A data envelopment analysis approach integrating technical, economic, and environmental issues. Journal of Environmental Management, 167, 160–166.

    Article  Google Scholar 

  • Cazals, C., Florens, J. P., & Simar, L. (2002). Nonparametric frontier estimation: A robust approach. Journal of Econometrics, 106(1), 1–25.

    Article  Google Scholar 

  • Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444.

    Article  Google Scholar 

  • D’Inverno, G., Carosi, L., Romano, G., & Guerrini, A. (2018). Water pollution in wastewater treatment plants: An efficiency analysis with undesirable output. European Journal of Operational Research, 269(1), 24–34.

    Article  Google Scholar 

  • Daouia, A., & Simar, L. (2007). Nonparametric efficiency analysis: A multivariate conditional quantile approach. Journal of Econometrics, 140(2), 375–400.

    Article  Google Scholar 

  • Daraio, C., & Simar, L. (2005). Introducing environmental variables in nonparametric frontier models: A probabilistic approach. Journal of Productivity Analysis, 24(1), 93–121.

    Article  Google Scholar 

  • Daraio, C., & Simar, L. (2007). Conditional nonparametric frontier models for convex and nonconvex technologies: A unifying approach. Journal of Productivity Analysis, 28(1), 13–32.

    Article  Google Scholar 

  • De Witte, K., & Kortelainen, M. (2013). What explains the performance of students in a heterogeneous environment? Conditional efficiency estimation with continuous and discrete environmental variables. Applied Economics, 45(17), 2401–2412.

    Article  Google Scholar 

  • De Witte, K., & Marques, R. C. (2010a). Incorporating heterogeneity in non-parametric models: A methodological comparison. International Journal of Operational Research, 9(2), 188–204.

    Article  Google Scholar 

  • De Witte, K., & Marques, R. C. (2010b). Influential observations in frontier models, a robust non-oriented approach to the water sector. Annals of Operations Research, 181(1), 377–392.

    Article  Google Scholar 

  • De Witte, K., & Schiltz, F. (2018). Measuring and explaining organizational effectiveness of school districts: Evidence from a robust and conditional Benefit-of-the-Doubt approach. European Journal of Operational Research, 267(3), 1172–1181.

    Article  Google Scholar 

  • Dong, X., Zhang, X., & Zeng, S. (2017). Measuring and explaining eco-efficiencies of wastewater treatment plants in China: An uncertainty analysis perspective. Water Research, 112, 195–207.

    Article  Google Scholar 

  • Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2), 245–259.

    Article  Google Scholar 

  • Farrell, M. J. (1957). The measurement of productive efficiency. Journal of the Royal Statistical Society: Series A (general), 120(3), 253–281.

    Article  Google Scholar 

  • Fuentes, R., Torregrosa, T., & Ballenilla, E. (2015). Conditional order-m efficiency of wastewater treatment plants: The role of environmental factors. Water, 7(10), 5503–5524.

    Article  Google Scholar 

  • Fusco, E., Vidoli, F., & Rogge, N. (2020). Spatial directional robust Benefit of the Doubt approach in presence of undesirable output: An application to Italian waste sector. Omega, 94, 102053.

    Article  Google Scholar 

  • Golany, B., & Thore, S. (1997). Restricted best practice selection in DEA: An overview with a case study evaluating the socio-economic performance of nations. Annals of Operations Research, 73, 117–140.

    Article  Google Scholar 

  • Gómez, T., Gémar, G., Molinos-Senante, M., Sala-Garrido, R., & Caballero, R. (2017). Assessing the efficiency of wastewater treatment plants: A double-bootstrap approach. Journal of Cleaner Production, 164, 315–324.

    Article  Google Scholar 

  • Guerrini, A., Romano, G., Mancuso, F., & Carosi, L. (2016). Identifying the performance drivers of wastewater treatment plants through conditional order-m efficiency analysis. Utilities Policy, 42, 20–31.

    Article  Google Scholar 

  • Hernández-Sancho, F., Molinos-Senante, M., & Sala-Garrido, R. (2011). Energy efficiency in Spanish wastewater treatment plants: A non-radial DEA approach. Science of the Total Environment, 409(14), 2693–2699.

    Article  Google Scholar 

  • Hernández-Sancho, F., & Sala-Garrido, R. (2009). Technical efficiency and cost analysis in wastewater treatment processes: A DEA approach. Desalination, 249(1), 230–234.

    Article  Google Scholar 

  • Lavigne, C., De Jaeger, S., & Rogge, N. (2019). Identifying the most relevant peers for benchmarking waste management performance: A conditional directional distance Benefit-of-the-Doubt approach. Waste Management, 89, 418–429.

    Article  Google Scholar 

  • Lee, C. Y., & Cai, J. Y. (2020). LASSO variable selection in data envelopment analysis with small datasets. Omega, 91, 102019.

    Article  Google Scholar 

  • Li, Q., & Racine, J. S. (2007). Nonparametric econometrics: Theory and practice. Princeton University Press.

    Google Scholar 

  • Longo, S., Hospido, A., Lema, J. M., & Mauricio-Iglesias, M. (2018). A systematic methodology for the robust quantification of energy efficiency at wastewater treatment plants featuring data envelopment analysis. Water Research, 141, 317–328.

    Article  Google Scholar 

  • Lorenzo-Toja, Y., Vázquez-Rowe, I., Chenel, S., Marín-Navarro, D., Moreira, M. T., & Feijoo, G. (2015). Eco-efficiency analysis of Spanish WWTPs using the LCA+ DEA method. Water Research, 68, 651–666.

    Article  Google Scholar 

  • Moldan, B., Janoušková, S., & Hák, T. (2012). How to understand and measure environmental sustainability: Indicators and targets. Ecological Indicators, 17, 4–13.

    Article  Google Scholar 

  • Molinos-Senante, M., Gémar, G., Gómez, T., Caballero, R., & Sala-Garrido, R. (2016). Eco-efficiency assessment of wastewater treatment plants using a weighted Russell directional distance model. Journal of Cleaner Production, 137, 1066–1075.

    Article  Google Scholar 

  • Molinos-Senante, M., Hernández-Sancho, F., Mocholí-Arce, M., & Sala-Garrido, R. (2014). Economic and environmental performance of wastewater treatment plants: Potential reductions in greenhouse gases emissions. Resource and Energy Economics, 38, 125–140.

    Article  Google Scholar 

  • Molinos-Senante, M., Hernandez-Sancho, F., & Sala-Garrido, R. (2014). Benchmarking in wastewater treatment plants: A tool to save operational costs. Clean Technologies and Environmental Policy, 16(1), 149–161.

    Article  Google Scholar 

  • Omrani, H., Alizadeh, A., Emrouznejad, A., & Teplova, T. (2021). Data envelopment analysis model with decision makers’ preferences: a robust credibility approach. Annals of Operations Research, 1–38. https://doi.org/10.1007/s10479-021-04262-2

  • Omrani, H., Alizadeh, A., Emrouznejad, A., & Teplova, T. (2022). A robust credibility DEA model with fuzzy perturbation degree: An application to hospitals performance. Expert Systems with Applications, 189, 116021.

    Article  Google Scholar 

  • Rogge, N., & De Jaeger, S. (2013). Measuring and explaining the cost efficiency of municipal solid waste collection and processing services. Omega, 41(4), 653–664.

    Article  Google Scholar 

  • Roháčová, V. (2015). A DEA based approach for optimization of urban public transport system. Central European Journal of Operations Research, 23(1), 215–233.

    Article  Google Scholar 

  • Ruiz, J. L., & Sirvent, I. (2019). Performance evaluation through DEA benchmarking adjusted to goals. Omega, 87, 150–157.

    Article  Google Scholar 

  • Sadjadi, S. J., & Omrani, H. (2008). Data envelopment analysis with uncertain data: An application for Iranian electricity distribution companies. Energy Policy, 36(11), 4247–4254.

    Article  Google Scholar 

  • Sala-Garrido, R., Hernández-Sancho, F., & Molinos-Senante, M. (2012). Assessing the efficiency of wastewater treatment plants in an uncertain context: A DEA with tolerances approach. Environmental Science & Policy, 18, 34–44.

    Article  Google Scholar 

  • Sala-Garrido, R., Molinos-Senante, M., & Hernández-Sancho, F. (2011). Comparing the efficiency of wastewater treatment technologies through a DEA metafrontier model. Chemical Engineering Journal, 173(3), 766–772.

    Article  Google Scholar 

  • Sala-Garrido, R., Molinos-Senante, M., & Hernández-Sancho, F. (2012). How does seasonality affect water reuse possibilities? An efficiency and cost analysis. Resources, Conservation and Recycling, 58, 125–131.

    Article  Google Scholar 

  • Schiltz, F., De Witte, K., & Mazrekaj, D. (2020). Managerial efficiency and efficiency differentials in adult education: A conditional and bias-corrected efficiency analysis. Annals of Operations Research, 288(2), 529–546.

    Article  Google Scholar 

  • Silva, C., & Rosa, M. J. (2015). Energy performance indicators of wastewater treatment: A field study with 17 Portuguese plants. Water Science and Technology, 72(4), 510–519.

    Article  Google Scholar 

  • Simar, L. (2003). Detecting outliers in frontier models: A simple approach. Journal of Productivity Analysis, 20(3), 391–424.

    Article  Google Scholar 

  • Simar, L., & Wilson, P. W. (1998). Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models. Management Science, 44(1), 49–61.

    Article  Google Scholar 

  • United Nations (2015). Transforming our world: the 2030 agenda for sustainable development. General Assembly Resolution A/RES/70/1. New York: United Nations.

  • Vaz, C. B., Camanho, A. S., & Guimarães, R. C. (2010). The assessment of retailing efficiency using network data envelopment analysis. Annals of Operations Research, 173(1), 5–24.

    Article  Google Scholar 

  • Wheelock, D. C., & Wilson, P. (2003). Robust nonparametric estimation of efficiency and technical change in US commercial banking. Federal Reserve Bank of St. Louis Working Paper Series, (2003–037).

  • Wilson, P. W. (1993). Detecting outliers in deterministic nonparametric frontier models with multiple outputs. Journal of Business & Economic Statistics, 11(3), 319–323.

    Google Scholar 

  • Wilson, P. W. (1995). Detecting influential observations in data envelopment analysis. Journal of Productivity Analysis, 6(1), 27–45.

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to acknowledge the financial support of Project "More Sustainable WWTPs”. This project is financed by the Portuguese water company Águas do Centro Litoral (AdCL). The authors are especially grateful to all the support provided by Tiago Braga and Paulo Leitão from AdCL. The authors also wish to gratefully acknowledge Flávia Barbosa for the computational assistance in building the DEA model with peers restricted, and to Kristof De Witte and Laura Carosi for their valuable comments and suggestions on a previous version of the paper. The first author also wishes to acknowledge the financial support from FCT—Fundação para a Ciência e a Tecnologia (Portuguese national funding agency for science, research and technology), given through the Grant PD/BD/142815/2018 respecting the Doctoral Program on Sustainable Energy Systems by MIT-Portugal. The fourth author also gratefully acknowledges financial support from Research Foundation-Flanders, FWO (Postdoctoral Fellowship 12U0219N).

Author information

Authors and Affiliations

  1. Faculdade de Engenharia, Universidade do Porto, Porto, Portugal

    Alda A. Henriques, Ana S. Camanho & Pedro Amorim

  2. Águas do Centro Litoral (AdP Group), Coimbra, Portugal

    Milton Fontes

  3. Faculty of Economics and Business, KU Leuven, Leuven, Belgium

    Giovanna D’Inverno

  4. AdP Energias and Instituto Superior de Engenharia do Porto, Porto, Portugal

    Jaime Gabriel Silva

Authors
  1. Alda A. Henriques
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Milton Fontes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ana S. Camanho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giovanna D’Inverno
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pedro Amorim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jaime Gabriel Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ana S. Camanho.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 187 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Henriques, A.A., Fontes, M., Camanho, A.S. et al. Performance evaluation of problematic samples: a robust nonparametric approach for wastewater treatment plants. Ann Oper Res 315, 193–220 (2022). https://doi.org/10.1007/s10479-022-04629-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-022-04629-z

Keywords

Search

Navigation