Analysis of the Methodologies for Evaluation of E-Government Policies

  • Dalibor Stanimirovic
  • Tina Jukic
  • Janja Nograsek
  • Mirko Vintar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7443)


Methodologies for evaluation of e-government policies do not provide enough valuable information to policy makers in conducting quality planning of e-government initiatives. Consequently, user acceptance of e-government services is below government anticipations, while the expected effects in terms of reducing costs and increasing the effectiveness of public administration are still in early stages. Paper presents an overview of existing methodologies for evaluation of e-government policies, identifies characteristics of recent evaluations and conceptualizes a theoretical framework for their comparative analysis. Analysis of more than 50 evaluation methodologies offers an insight into the current evaluation practice, enables detection of its deficiencies as well as their mitigation and could facilitate a significant contribution to more evidence-based evaluation of e-government policies.


e-government policy evaluation methodology evaluation and development level comparative analysis 


  1. 1.
    Silcock, R.: What is E-government? Parliamentary Affairs 54, 88–101 (2001)CrossRefGoogle Scholar
  2. 2.
    OECD. Organization for Economic Co-operation and Development: E-government: Analysis Framework and Methodology, Paris (2001)Google Scholar
  3. 3.
    Yildiz, M.: Examining the Motivations for E-Government from an Institutional Theory Perspective: Evidence from Turkey. In: The National Conference on Digital Government Research, Boston (2003)Google Scholar
  4. 4.
    Capgemini: Smarter, Faster, Better eGovernment, 8th Benchmark Measurement. European Commission Directorate General for Information Society and Media (2009)Google Scholar
  5. 5.
    Capgemini, Rand, DTi, IDC, Sogeti: Digitizing Public Services in Europe: Putting ambition into action. Report of the 9th Measurement. European Commission (2010)Google Scholar
  6. 6.
    Capgemini, IDC, Rand Europe, Sogeti and DTi: eGovernment Benchmark Pilot on Open Government and Transparency. European Commission (2011)Google Scholar
  7. 7.
    OECD. Organization for Economic Co-operation and Development: Focus on Citizens: Public Engagement for Better Policy and Services, Paris (2009)Google Scholar
  8. 8.
    OECD. Organization for Economic Co-operation and Development: Rethinking e-Government Services: User-centred approaches, Paris (2009)Google Scholar
  9. 9.
    OECD. Organization for Economic Co-operation and Development: E-Government Indicators: Proposal for a new Framework and Methodology, Paris (2010)Google Scholar
  10. 10.
    Kunstelj, M., Jukić, T., Vintar, M.: Analysing the Demand Side of E-Government: What Can We Learn From Slovenian Users? In: Wimmer, M.A., Scholl, J., Grönlund, Å. (eds.) EGOV 2007. LNCS, vol. 4656, pp. 305–317. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  11. 11.
    Kim, C.K.: A Cross-national Analysis of Global E-government. Public Organization Review 7, 317–329 (2007)CrossRefGoogle Scholar
  12. 12.
    Vintar, M., Nograsek, J.: How much can we trust different e-government surveys? The case of Slovenia. Information Polity 15, 199–213 (2010)Google Scholar
  13. 13.
    Dutta, S.: Estonia: A Sustainable Success in Networked Readiness? The Global Information Technology Report 2006–2007. Palgrave Macmillan, Hampshire (2007)Google Scholar
  14. 14.
    UN. United Nations: United Nations E-Government Survey, Leveraging e-government at a time of financial and economic crisis. Department of Economic and Social Affairs (2010) Google Scholar
  15. 15.
    WEF. World Economic Forum: The Global Information Technology Report 2010–2011. Transformations 2.0 (2011)Google Scholar
  16. 16.
    Bannister, F.: The curse of the benchmark: an assessment of the validity and value of e-government comparisons. Int. Review of Administrative Sciences 73, 171–188 (2007)CrossRefGoogle Scholar
  17. 17.
    Janssen, D., Rotthier, S., Snijkers, K.: If you measure it, they will score: An assessment of international eGovernment benchmarking. Information Polity 9, 121–130 (2004)Google Scholar
  18. 18.
    AGIMO. Australian Government Information Office: Demand and Value Assessment Methodology - DAM & VAM. Commonwealth of Australia (2004)Google Scholar
  19. 19.
    RSO SpA & LUISS Management: eGEP - eGovernment Economics Project: Expenditure Study Final Version. DG Information Society and Media, Milano (2006) Google Scholar
  20. 20.
    UN. United Nations: United Nations E-Government Survey,From E-Government to Connected Governance. Department of Economic and Social Affairs, Division for Public Administration and Development Management, New York (2008) Google Scholar
  21. 21.
    Accenture: From e-Government to e-Governance: Using new technologies to strengthen relationships with citizens. Institute for Health and Public Service Value, Dublin (2009)Google Scholar
  22. 22.
    West, D.M.: Improving Technology Utilization in Electronic Government around the World. Center for Public Policy, Brown University, Rhode Island (2008)Google Scholar
  23. 23.
    Dadayan, L.: Measuring Return on Government IT Investments. In: Proceedings of the 13th European Conference on Information Technology Evaluation, Genoa (2006)Google Scholar
  24. 24.
    Dutta, S., Mia, I. (eds.): WEF, World Economic Forum: The Global Information Technology Report 2009-2010, ICT for Sustainability (2010)Google Scholar
  25. 25.
    EIU. Economist Intelligence Unit:Digital economy rankings 2010 (2010)Google Scholar
  26. 26.
    ADAE: MAREVA methodology guide: Analysis of the value of ADELE projects. Fourth High Level Seminar on Measuring and Evaluating E-Government, Dubai (2007)Google Scholar
  27. 27.
    Rothig, P.: WiBe 4.0 Methodology. Economic Efficiency Assessments in Particular with Regard to the Use of Information & Communication Technology (2010)Google Scholar
  28. 28.
    Scholl, H.J.: Electronic Government: Information Management Capacity, Organizational Capabilities, and the Sourcing Mix. Government Information Quarterly 23, 73–96 (2006)CrossRefGoogle Scholar
  29. 29.
    Greaver, M.F.: Strategic Outsourcing: A Structured Approach to Outsourcing Decisions and Initiatives. AMACOM, New York (1999)Google Scholar
  30. 30.
    Kern, T., Willcocks, L.P., Van Heck, E.: The winner’s curse in IT outsourcing: Strategies for avoiding relational trauma. California Management Review 44, 47–69 (2002)Google Scholar
  31. 31.
    Lacity, M.C., Willcocks, L.P.: Information Systems and Outsourcing. Studies in Theory and Practice. Palgrave Macmillan (2009)Google Scholar
  32. 32.
    Klievink, B., Janssen, M.: Realizing joined-up government – Dynamic capabilities and stage models for transformation. Government Information Quarterly 6, 275–284 (2009)CrossRefGoogle Scholar
  33. 33.
    van Veenstra, A.F., Janssen, M., Tan, Y.-H.: Towards an Understanding of E-Government Induced Change – Drawing on Organization and Structuration Theories. In: Wimmer, M.A., Chappelet, J.-L., Janssen, M., Scholl, H.J. (eds.) EGOV 2010. LNCS, vol. 6228, pp. 1–12. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  34. 34.
    Schedler, K., Schmidt, B.: Managing the e-government organization. International Public Management Review 5, 1–20 (2004)Google Scholar
  35. 35.
    Scholl, H.J.: E-government: A Special Case of ICT-enabled Business Process Change. In: 36th Hawaii International Conference on System Sciences (2003)Google Scholar
  36. 36.
    Leitner, C., Kreuzeder, M.: Organisational Changes, Skills and the Role of Leadership Required by eGovernment. In: Wimmer, M.A., Traunmüller, R., Grönlund, Å., Andersen, K.V. (eds.) EGOV 2005. LNCS, vol. 3591, pp. 210–217. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  37. 37.
    Martin, B., Byrne, J.: Implementing e-Government: widening the lens. RMIT University, Australia, Melbourne (2003)Google Scholar
  38. 38.
    Bavec, C., Vintar, M.: What Matters in the Development of the E-Government in the EU? In: Wimmer, M.A., Scholl, J., Grönlund, Å. (eds.) EGOV 2007. LNCS, vol. 4656, pp. 424–435. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  39. 39.
    Singh, H., Das, A., Joseph, D.: Country-Level Determinants of E-Government Maturity. Communications of the Association for Information System 20, 632–648 (2007)Google Scholar
  40. 40.
    OECD.Organization for Economic Co-operation and Development: E-government and Administrative Simplification: Good Governance for Development in Arab Countries, Dubai (2007)Google Scholar
  41. 41.
    Hamilton, B.A.: Building a Methodology for Measuring the Value of E-Services. USA Social Security Administration (2002)Google Scholar
  42. 42.
    Kunstelj, M., Dečman, M.: Metrika na področju spremljanja razvoja e-uprave. In: Vintar, M., Grad, J. (eds.) Euprava: Izbrane Razvojne Perspective, Ljubljana, pp. 37–60 (2004)Google Scholar
  43. 43.
    Kunstelj, M., Vintar, M.: Evaluating the progress of e-government development: A critical analysis. Information Polity 9, 131–148 (2004)Google Scholar
  44. 44.
    Taylor, H.: Critical risks in outsourced IT projects: the intractable and the unforeseen. Communications of the ACM 49, 75–79 (2006)CrossRefGoogle Scholar
  45. 45.
    Jensen, P.H.: Public Sector Outsourcing Contracts - The Impact of Uncertainty, Incentives and Transaction Costs on Contractual Relationships. VDM Verlag, Saarbrücken (2007)Google Scholar
  46. 46.
    Linder, J.C.: Outsourcing for Radical Change: A Bold Approach to Enterprise Transformation. AMACOM, New York (2004)Google Scholar
  47. 47.
    Power, M.J., Desouza, K., Bonifazi, C.: The Outsourcing Handbook: How to Implement a Successful Outsourcing Process. Kogan Page, Chicago (2006)Google Scholar
  48. 48.
    O’Donnell, O., Boyle, R., Timonen, V.: Transformational aspects of E-government in Ire-land: Issues to be addressed. Electronic Journal of e-Government 1, 23–32 (2003)Google Scholar
  49. 49.
    Griffin, D., Foster, A., Halpin, E.: Joined-up E-government: an exploratory study of UK local government progress. Journal of Information Science and Technology 1, 58–83 (2004)Google Scholar
  50. 50.
    Layne, K., Lee, J.: Developing fully functional E-government: A four stage model. Government Information Quarterly 18, 122–136 (2001)CrossRefGoogle Scholar
  51. 51.
    Elnaghi, M., Alshawi, S., Missi, F.: A Leadership Model for e-Government Transformation. In: European and Mediterranean Conference on Information Systems (2007)Google Scholar
  52. 52.
    Ho, A.T.K.: Reinventing Local Governments and the E-Government Initiative. Public Administration Review 62, 434–444 (2002)CrossRefGoogle Scholar
  53. 53.
    Norris, P.: Digital Divide: Civic Engagement, Information Poverty and the Internet. Cambridge University Press, Cambridge (2001)CrossRefGoogle Scholar
  54. 54.
    Slevin, J.: The Internet and Society. Polity Press, Cambridge (2000)Google Scholar
  55. 55.
    Anttiroiko, A.V.: Toward the European Information Society. Communications of the ACM 44, 31–36 (2001)CrossRefGoogle Scholar
  56. 56.
    Thurlow, C., Lengel, L., Tomic, A.: Computer mediated communication: Social interac-tion and the internet. Sage Publications, Thousand Oaks (2004)Google Scholar
  57. 57.
    Shim, D.C., Eom, T.H.: E-Government and Anti-Corruption: Empirical Analysis of International Data. International Journal of Public Administration 31, 298–316 (2008)CrossRefGoogle Scholar
  58. 58.
    Tolbert, C.J., Mossberger, K.: The Effects of E-Government on Trust and Confidence in Government. Public Administration Review 66, 354–370 (2006)CrossRefGoogle Scholar
  59. 59.
    Parent, M., Vandebeek, C.A., Gemino, A.C.: Building Citizen Trust through e-Government. Government Information Quarterly 22, 720–736 (2005)CrossRefGoogle Scholar
  60. 60.
    Bolgherini, S.: The Technology Trap and the Role of Political and Cultural Variables: A Critical Analysis of the E-Government Policies. Review of Policy Research 24, 259–275 (2007)CrossRefGoogle Scholar
  61. 61.
    Lor, P.J., Britz, J.J.: Is a knowledge society possible without freedom of access to information? Journal of Information Science 33, 387–397 (2007)CrossRefGoogle Scholar
  62. 62.
    Mahalik, D.K.: Outsourcing in e-Governance: A Multi Criteria Decision Making Approach. Journal of Administration & Governance 5, 24–35 (2010)Google Scholar
  63. 63.
    CIO Council: Value Measuring Methodology: How-To-Guide, Washington (2002) Google Scholar
  64. 64.
    OECD: Guide to measuring the information society. Directorate for science, technology and industry, Working party on indicators for the information society, France, Paris (2005)Google Scholar
  65. 65.
    Rama Rao, T.P., Venkata Rao, V., Bhatnagar, S.C., Satyanarayana, S.J.: E-Governance As-sessment Frameworks (EAF Version 2.0). Government of India (2004)Google Scholar
  66. 66.
    UNESCO: E-government Tool Kit for Developing Countries. Asia Pacific Regional Bureau for Communication and Information, New Delhi (2005)Google Scholar
  67. 67.
    Baudu, S., Dzhumalieva, S.: Value Assessment Tool - VAST. Licensed by the European Commission under EUPL. Directorate General for Informatics, DIGIT (2010)Google Scholar
  68. 68.
    Republic of Korea: e-Government in Korea. Ministry of Public Administration and Security. OECD e-Leaders Meeting 2010, Belgium, Brussels (2010)Google Scholar
  69. 69.
    Devadoss, P.R., Pan, S.L., Huang, J.C.: Structurational analysis of e-government initia-tives: a case study of SCO. Decision Support Systems 34, 253–269 (2002)CrossRefGoogle Scholar
  70. 70.
    Moon, M.J.: The Evolution of E-Government among Municipalities: Rhetoric or Reality? Public Administration Review 62, 424–433 (2002)CrossRefGoogle Scholar
  71. 71.
    Chen, Y.C., Perry, J.: Outsourcing for E-Government: Managing for Success. Public Performance & Management Review 26, 404–421 (2003)CrossRefGoogle Scholar
  72. 72.
    Štemberger, M.I., Jaklič, J.: Towards E-government by business process change – A methodology for public sector. International Journal of Information Management 27, 221–232 (2007)CrossRefGoogle Scholar
  73. 73.
    Pollitt, C.: Technological Change: a central yet neglected feature of public administration. In: NISPA 2010, Slovenia, Ljubljana (2010)Google Scholar
  74. 74.
    Deloitte & Ipsos Belgium: Benchmarking deployment of eHealth services III (2011)Google Scholar
  75. 75.
    European Commission, Capgemini, & IDC: eGovernment Benchmarking in 2011 (2011)Google Scholar
  76. 76.
    Wauters, P., Nijskens, M., Tiebout, J. (Capgemini):The User Challange, Benchmarking The Supply Of Online Public Services (2007)Google Scholar
  77. 77.
    Meyer, I., Hüsing, T., Didero, M., Korte, W.B. (Empirica): eHealth Benchmarking (2009)Google Scholar
  78. 78.
    Weehuizen, R., van Oranje, C.: Innovative and adaptive pan-European services for citizens in 2010 and beyond. DG Information Society & Media (2007)Google Scholar
  79. 79.
    Boyer-Wright, K.M., Kottemann, J.E.: An Empirical Assessment of Common Fundamentals in National E-Readiness Frameworks. Journal of Global Information Technology Management 12, 55–74 (2009)Google Scholar
  80. 80.
    Picci, L.: The quantitative evaluation of the Economic Impact of E-Government: A structural Modelling Approach. Information Economics and Policy 18, 107–123 (2006)CrossRefGoogle Scholar
  81. 81.
    Rhee, D.Y.: The Study on Determinants of E-Government Evolution. Journal of Korean Association for Regional Information Society 12, 183–201 (2009)Google Scholar
  82. 82.
    Schwester, R.W.: Socio-Demographic Determinants of E-government Adoption: An Examination of Major U.S. Cities. Journal of Public Management and Social Policy 16, 21–32 (2010)Google Scholar
  83. 83.
    Serrano-Cinca, C., Rueda-Tomás, M., Portillo-Tarragona, P.: Determinants of E-Government Extension. Online Information Review 33, 476–498 (2009)CrossRefGoogle Scholar
  84. 84.
    Gallego-Álvarez, I., Rodríguez-Domínguez, L., García-Sánchez, I.M.: Are Determining Factors of Municipal E-Government Common to a Worldwide Municipal View? An Intra-Country Comparison. Government Information Quarterly 27, 423–430 (2010)CrossRefGoogle Scholar
  85. 85.
    Holzer, M., Kim, S.T.: Digital Governance in Municipalities Worldwide, A Longitudinal Assessment of Municipal Websites Throughout the World. New Jersey (2007)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  • Dalibor Stanimirovic
    • 1
  • Tina Jukic
    • 1
  • Janja Nograsek
    • 1
  • Mirko Vintar
    • 1
  1. 1.Faculty of AdministrationUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations