Health Care Management Science

, Volume 22, Issue 2, pp 197–214 | Cite as

Multimethodology applied to the evaluation of Healthcare in Brazilian municipalities

  • Marcos Pereira Estellita Lins
  • Sergio Orlando Antoun NettoEmail author
  • Maria Stella de Castro Lobo


The integration of quantitative indicators with qualitative descriptions of context is a noticeable demand from many different scientific disciplines, since it contributes to linking theoretical and practical approaches to problem solving. Amongst them are the problem structuring methods, systems thinking and multimethodology. This work presents a mixed quantitative and qualitative methodological approach to aid formulation and structuring of performance measurement of health care in 5565 Brazilian municipalities. Data mining and data envelopment analysis (DEA) are applied in the context of conceptual mapping, thus shedding light on both quantitative and qualitative factors that influence health performance. Our aim is to propose a methodology for performance indicators to support health care policy making in Brazil, using quantitative indicators. However, the approach does not lose track of the role of important qualitative factors in the attribution of meaning to performance assessments. The methodological and analytical results can strengthen mutual understanding by analysts and stakeholders of the problem at hand. Quantitative results allow inefficient municipalities to understand the causes of their overall efficiency in terms of particular low partial DEA efficiencies combined with high deathrates.


Data envelopment analysis (DEA) Multimethodology Concept map Data mining Public health 



We gratefully acknowledge the grants from CNPq (process 304324/2013-2) and FAPERJ/CNE in support of this study.


  1. 1.
    Amado CAF, Dyson RG (2008) On comparing the performance of primary care providers. Eur J Oper Res 185:915–932. CrossRefGoogle Scholar
  2. 2.
    Amado CAF, Dyson RG (2009) Exploring the use of DEA for formative evaluation in primary diabetes care: an application to compare English practices. J Oper Res Soc 60:1469–1482. CrossRefGoogle Scholar
  3. 3.
    Bougnol ML, Dulá JH, Estellita Lins MP, Moreira da Silva AC (2010) Enhancing standard performance practices with DEA. Omega 38:33–45. CrossRefGoogle Scholar
  4. 4.
    Chilingerian JA, Sherman D (2004) Health care applications from hospitals to physicians; from productive efficiency to quality Frontiers. In: Cooper WW, Seiford LM, Zhu J (eds) Handbook on data envelopment analysis. Kluwer Academic Publishers, BostonGoogle Scholar
  5. 5.
    Cooper WW, Seiford LM, Tone K (2007) Data envelopment analysis – a comprehensive text with models, application, references and DEA solver. Springer, BerlinGoogle Scholar
  6. 6.
    Franceschini F, Galetto M, Maisano D, Mastrogiacomo L (2008) Properties of performance indicators in operations management - a reference framework. Int J Product Perform Manag 57(2):137–155. CrossRefGoogle Scholar
  7. 7.
    George J, Cowan J (1999) A handbook of techniques for formative evaluation. Routledge, AbingdonGoogle Scholar
  8. 8.
    Gerring J (2007) Case study research principles and practices. Boston University/ Cambridge University Press, Boston/CambridgeGoogle Scholar
  9. 9.
    Gerring J (2011) The case study: What it is and what it does. The Oxford handbook of political science. In: Robert E. Goodin (eds). Print Publication Date: Jul 2011 Published online: Sep 2013 ISBN: 9780199604456.
  10. 10.
    Golany B, Roll Y (1989) An application procedure for DEA. Omega. Int J Manag Sci 17:237–250Google Scholar
  11. 11.
    Goldschmidt R, Passos E (2005) Data mining: um guia prático. Elsevier, Rio de JaneiroGoogle Scholar
  12. 12.
    Habermas J (1998) The inclusion of the other. studies in political theory. MIT Press, 1992. Moral consciousness and communicative action (studies in contemporary German social thought). MIT Press, BostonGoogle Scholar
  13. 13.
    Hauser JR, Katz GM (2008) Metrics: You are what you measure! Accessed 04 Dec 2017
  14. 14.
    Hruschka ER, Ebecken NFF (2001) A genetic algorithm for cluster analysis. Intelligent Data Analysis 7(1):15–25Google Scholar
  15. 15.
    IDEAL (2015) Interactive Data Envelopment Analysis, developed by Prof Marcos Lins from the Federal University of Rio de Janeiro (available
  16. 16.
    Jackson MC (2001) Critical systems thinking and practice. Eur J Oper Res 128:233–244. CrossRefGoogle Scholar
  17. 17.
    Jain AK, Dubes RC (1988) Algorithms for clustering data. Prentice Hall, Englewood CliffsGoogle Scholar
  18. 18.
    Kirky MW (2000) Operations research trajectories: the Anglo-American experience from the 1930s to 1960s. Oper Res 48:661–670CrossRefGoogle Scholar
  19. 19.
    Koopmans TC (1951) Analysis of production as an efficient combination of activities. In: Koopmans TC (ed) Activity analysis of production and allocation, Cowles commission. Wiley, New York, pp 33–97Google Scholar
  20. 20.
    Le Menestrel LM, Van Wassenhove LN (2004) Ethics outside, within, or beyond OR models? Eur J Oper Res 153:477–474CrossRefGoogle Scholar
  21. 21.
    Lins ME, Sollero MKV, Calôba LG, da Silva ACM (2007a) Integrating the regulatory and utility firm perspectives, when measuring the efficiency of electricity distribution. Eur J Oper Res 181:1413–1424. CrossRefGoogle Scholar
  22. 22.
    Lins MPE, Lobo MSC, Fiszman R, Silva ACM, Ribeiro VJP (2007b) O Uso da Análise Envoltória de Dados – DEA - para Avaliação de Hospitais Universitários Brasileiros. Revista Ciência e Saúde Coletiva 12(4):985998Google Scholar
  23. 23.
    Lins ME, Oliveira LB, da Silva ACM, Rosa LP, Pereira AO (2011) Performance assessment of alternative energy resources in Brazilian power sector using data envelopment analysis. Renew Sustain Energy Rev 16:898–903CrossRefGoogle Scholar
  24. 24.
    Lobo MSC, Lins MPE (2010) Epistemic dialog between health services and operations research. Pesquisa Operacional 30(2):371–386. CrossRefGoogle Scholar
  25. 25.
    Lobo MSC, Ozcan Y, Lins MPE, Silva ACM, Fiszman R (2010) Financing reform and productivity change in Brazilian teaching hospitals: Malmquist approach. CEJOR 18:141–152. CrossRefGoogle Scholar
  26. 26.
    MacQueen J (1967) Some methods for classification and analysis of multivariate observations. In: Le Cam LM, Neyman J (eds) Proceedings of the fifth Berkeley symposium on mathematical statistics and probability. Volume I: statistics. University of California Press, Berkeley, pp 281–297Google Scholar
  27. 27.
    Mingers (2006) Realizing systems thinking – knowledge and action in management science. Springer, BerlinGoogle Scholar
  28. 28.
    Mingers J (2010) Multimethodology. Wiley encyclopedia of operations research and management science. Hoboken, WileyGoogle Scholar
  29. 29.
    Mingers J, Brocklesby J (1997) Multimethodology: towards a framework for mixing methodologies. Omega 25(5):489–509 (21) 985-998CrossRefGoogle Scholar
  30. 30.
    Mingers J, Gill A (1997) Multimethodology: the theory and practice of combining management science methodologies. Wiley, ChichesterGoogle Scholar
  31. 31.
    Morgan G (1998) Images of organization. Berrett-Koehler Publishers, OaklandGoogle Scholar
  32. 32.
    Novak JD (1998) Learning, creating, and using knowledge: concept maps as facilitative tools in schools and corporations. Lawrence Erbaum Associates, Mahwah, N.J., Lawrence Erlbaum & AssocGoogle Scholar
  33. 33.
    Nunamaker TR (1983) Measuring routine nursing service efficiency: a comparison of cost per patient day and data envelopment analysis models. Health Serv Res 18(2 Pt 1):183–208Google Scholar
  34. 34.
    O´Neill L, Rauner M, Heidenberger K, Kraus M (2008) A cross-national comparison and taxonomy of DEA-based hospital efficiency studies. Socio Econ Plan Sci 2008 42(3):158–189CrossRefGoogle Scholar
  35. 35.
    Okada A, Buckingham S, Sherborne T (2014) Knowledge Cartography: software tools and mapping techniques. 2nd edition. Springer-Verlag, LondonGoogle Scholar
  36. 36.
    Ozcan YA (2008) Health care benchmarking and performance evaluation: an assessment using data envelopment analysis (DEA). International series in operations research and management science. Springer, Berlin. CrossRefGoogle Scholar
  37. 37.
    Rosenhead J, Mingers J (2001) Rational Analysis for a Problematic World Revisited: Problem Structuring Methods for Complexity, Uncertainty and Conflict, 2nd Edition. Wiley, ChichesterGoogle Scholar
  38. 38.
    Ruiz-Moreno L, Sonzogno MC, Batista SHSE, Batista NA (2007) Mapa conceitual: ensaiando critérios de análise. Ciência Educação 13:453–463. CrossRefGoogle Scholar
  39. 39.
    Scheel H (2001) Undesirable outputs in efficiency valuations. Eur J Oper Res 132:400–410. CrossRefGoogle Scholar
  40. 40.
    Seiford LM, Zhu J (2002) Modeling undesirable factors in efficiency evaluation. Eur J Oper Res 142:16–20. CrossRefGoogle Scholar
  41. 41.
    Sherman HD (1984) Hospital efficiency measurement and evaluation. Med Care 22(10):922–939. CrossRefGoogle Scholar
  42. 42.
    Sherman HD, Zhu J (2006) Service productivity management: improving service performance using data envelopment Analysis (DEA). Springer, BostonGoogle Scholar
  43. 43.
    Sneath PHA, Sokal RR (1973) Numerical taxonomy. Freeman, San FranciscoGoogle Scholar
  44. 44.
    Suchman AL, Sluyter DJ, Williamson PR (2011) Leading change in healthcare – transforming organizations using complexity, positive psychology and relationship-centered care. Radcliffe Publishing, AbingdonGoogle Scholar
  45. 45.
    Tan P-N, Steinbach M, Kumar V (2009) Introduction to Data Mining. Pearson, Addison WesleyGoogle Scholar
  46. 46.
    Tashakkori A, Tedolie C (eds) (2003) Handbook of mixed methods in social and behavioral research. Sage, Thousand OaksGoogle Scholar
  47. 47.
    Vekiri I (2002) What is the value of graphical displays in learning? Psychol Rev 14(3):261–312Google Scholar
  48. 48.
    Vidal RVV (2006) Operational research: a multidisciplinary field. Pesquisa Operacional 26(1):69–90. CrossRefGoogle Scholar
  49. 49.
    Waszak C, Sines MC (2003) Mixed methods in psychological research. In: Tashakkori A, Teddlie C (Eds.) Handbook of mixed methods in social and behavioral research. Sage, Thousand Oaks, pp 537–576Google Scholar
  50. 50.
    Weis SM, Indurkhya N (1999) Predict data mining. Morgan Kaufmann Publishers, Inc., BurlingtonGoogle Scholar
  51. 51.
    WEKA (2012) Waikato environment for knowledge analysis. Software under GPL license, produced by the University of Waikato in New Zealand in the Java language, available at
  52. 52.
    Wu J, Liang L, Song M (2010) Performance based clustering for benchmarking of container ports: an application of DEA and cluster analysis technique. Int J Comput Intell Syst 3(6):709–722Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Marcos Pereira Estellita Lins
    • 1
  • Sergio Orlando Antoun Netto
    • 2
    Email author
  • Maria Stella de Castro Lobo
    • 3
  1. 1.Federal University of the State of Rio de Janeiro, UNIRIO, Centro de Ciências Exatas e TecnologiaRio de JaneiroBrazil
  2. 2.Faculdade de EngenhariaState University of Rio de JaneiroRio de JaneiroBrazil
  3. 3.University Hospital Clementino Fraga FilhoRio de JaneiroBrazil

Personalised recommendations