Comparative Evaluation Systems

  • Mario CocciaEmail author
  • Igor Benati
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-31816-5_1210-1

Synonyms

Definition

Comparative evaluation system is a systematic process for data collection, measurement, and analysis of the characteristics of different entities to generate a final rating and support decision-making processes of stakeholders for specific goals.

Introduction

Evaluation of a single object and/or subject in general does not provide complete information for decision-making. In order to support a rational decision, comparative evaluation could be necessary. Vartiainen (2002) argues that comparative evaluation refers to a cognitive process in which the evaluation and the findings of the evaluation process are set within a comparative framework. This approach establishes an assessment on the evaluand (the subject of an evaluation), comparing it with objects of the same category (cf., Cooksy 1999). To put it differently, the attribution of value is in relative terms, through comparison with similar subjects and/or objects.

Therefore, a comparative evaluation system (CES) is a stable set of techniques and tools to compare different units (individuals, organizations, countries, etc.) over time and space. In particular, a CES is a systematic process for data collection, measurement, and analysis of the characteristics of different entities to generate a final rating and support the decision-making processes of stakeholders for specific goals. The theoretical approach of CES has practical applications in different fields. Figure 1 shows the basic elements and steps of a CES.
Fig. 1

Elements and steps of a comparative evaluation system (CES)

A well-known CES is the rating system for the analysis of units, ranking and subsequent comparative assessment of their quality and performance-rating- (cf., Vartiainen 2002; Coccia and Benati 2018a, b). In particular, a rating system can be applied to assess the performance of individuals, organizations, and/or countries in achieving different objectives (e.g., career advancement for employees, funding for universities, international loans for countries, etc.). In the public administration, rating systems or CESs can be useful to provide information to assess performance, transparency level, and other characteristics of public organizations and/or their employees. Rating systems can pursue different objectives depending on the entity under study. As far as individuals are concerned, rating systems can be applied to personnel appraisal or recruiting. For organizations, rating systems can be aimed, for instance, at verifying the quality of the provision of services to citizens, the efficiency of public organizations, etc. Finally, for countries, rating systems can be directed to assess socioeconomic factors, such as democratization, general creditworthiness, etc. Figure 2 shows some examples of the objectives of rating systems for different entities (individuals, organizations, and countries). Although Comparative Evaluation Systems (CES) are the general category and rating systems are a main element of this set, in this contribution for the sake of simplicity, CES and rating system are used as interchangeable concepts.
Fig. 2

Entities and objectives of comparative evaluation systems (CESs)

Different typologies of rating systems for individuals, organizations, and countries are described in next paragraphs. For the sake of clarity, some practical examples are provided that, of course, are not a comprehensive overview of all CES applications in the public administration.

Rating Systems for Individuals

Rating systems of individuals can be adopted to assess skills or performance in all situations where they may be useful to achieve organizational objectives. In the public administration, rating systems, as CESs, are used both for recruitment processes and for career advancement. The individual performance of an employee is assessed by managers or external auditors against that of colleagues or previous activities carried out by the same employee (Llorens et al. 2018). An example of rating system of individuals is that implemented by the National Research Foundation (NRF) of South Africa. The NRF rating system provides a comparative evaluation of researchers with the goal of funding research activities in all fields (NRF 2018a). Its ratings are based on research outputs of researchers, considering their quality and impact as perceived by national and international peer reviewers (NRF 2018b). In particular, the reviewers analyze the quality of research outputs over previous years, the impact of research outputs on the field, the national and international standing of the researcher, and coherence of the researcher’s work. The NRF rating system incorporates five categories, as shown in Table 1.
Table 1

National Research Foundation (NRF) of South Africa rating system for researchers

Category

Description

A

Researchers who are unequivocally recognized by their peers as leading international scholars in their field for the high quality and impact of their recent research outputs.

B

Researchers who enjoy considerable international recognition by their peers for the high quality and impact of their recent research outputs.

C

Established researchers with a sustained recent record of productivity in the field who are recognized by their peers as having:

 - Produced a body of quality work, the core of which has coherence and attests to ongoing engagement with the field

 - Demonstrated the ability to conceptualize problems and apply research methods to investigating them.

P

Young researchers (normally younger than 35 years of age), who have held the doctorate or equivalent qualification for less than 5 years at the time of application and who, on the basis of exceptional potential demonstrated in their published doctoral work and/or their research outputs in their early postdoctoral careers, are considered likely to become future international leaders in their field.

Y

Young researchers (40 years or younger), who have held a doctorate or equivalent qualification for less than 5 years at the time of application, and who are recognized as having the potential to establish themselves as researchers within a 5-year period after evaluation, based on their performance and productivity of quality research outputs during their doctoral studies and/or early postdoctoral careers.

Source: NRF 2018a, b

Rating System for Organizations

The rating systems of public organizations are mainly intended for performance assessment of organizational behavior, quality of services provided to users, etc. They can be applied to different public organizations, such as hospitals, schools, local transport providers, universities, public research organizations, etc. (cf., Coccia 2005; Coccia and Rolfo 2010, 2013). The goal is to identify the best organizations in the provision of a service and to rank the various entities in terms of effectiveness and/or efficiency. An interesting example of rating system or CES for organizations is that of US hospitals (Hospital Compare 2018a, b). The overall rating of the Hospital Compare project summarizes up to 57 quality measures into a single star-based rating for each hospital. The methodology uses a statistical model known as latent variable model to calculate scores for seven categories:
  1. 1.

    Mortality

     
  2. 2.

    Safety of care

     
  3. 3.

    Readmission

     
  4. 4.

    Patient experience

     
  5. 5.

    Effectiveness of care

     
  6. 6.

    Timeliness of care

     
  7. 7.

    Efficient use of medical imaging

     
Each category (1–7) has a subsystem of indicators. For instance, the measures for category 7 “efficient use of medical imaging” are:
  • Outpatients with low-back pain who had an MRI (magnetic resonance imaging) without trying recommended treatments first, such as physical therapy

  • Outpatient CT (computed tomography) scans of the abdomen that were “combination” (double) scans

  • Outpatient CT scans of the chest that were “combination” (double) scans

  • Outpatients who got cardiac imaging stress tests before low-risk outpatient surgery

  • Outpatients with brain CT scans who got a sinus CT scan at the same time

These indicators are collected over the same period of time and used to calculate the rating for each category (Hospital Compare 2018b). A summary score is then calculated for each hospital by taking the weighted average of these group scores. The overall rating, ranging from 1 to 5 stars, shows how well each hospital performed, on average, compared to other hospitals in the USA. The more stars, the better a hospital performed across the available quality measures. This rating system or CES helps people choose the best hospital to treat their specific condition or solve health problems. Figure 3 shows the steps of this star rating methodology.
Fig. 3

Star rating methodology for US hospitals. Source: Hospital Quality Star Rating 2018

Rating System for Countries

Finally, rating systems or CES can also be applied to countries. An interesting example is the comparative evaluation of democracy level performed by Freedom House (2018), which is an independent organization dedicated to the expansion of freedom and democracy around the world. The information is useful for international investors, policy makers of international institutions, etc. Data for rating systems are collected by means of surveys measuring freedom according to two broad categories: political rights and civil liberties. Political rights enable people to freely participate in the political process, including the right to vote for distinct alternatives in legitimate elections, compete for public office, join political parties and organizations, and elect representatives who have a decisive impact on public policies and are accountable to the electorate. Civil liberties concern freedom of expression and belief, associational and organizational rights, rule of law, and personal autonomy without interference by the state. In particular, the rating process is based on a checklist of 10 questions on political rights and 15 questions on civil liberties. The questions on political rights are grouped into three subcategories: Electoral process (3 questions), political pluralism and participation (4), and functioning of government (3). The questions on civil liberties are grouped into four subcategories: freedom of expression and belief (4 questions), associational and organizational rights (3), rule of law (4), and personal autonomy and individual rights (4). The survey operates under the assumption that freedom for all peoples is best achieved in liberal democratic societies. General characteristics of each political rights and civil liberties rating are as follows (Freedom House 2018).

Political Rights

Rating of 1 – Countries and territories with a rating of 1 enjoy a wide range of political rights, including free and fair elections. Candidates who are elected actually rule, political parties are competitive, the opposition plays an important role and enjoys real power, and minority groups have reasonable self-government or can participate in the government through informal consensus.

Rating of 2 – Countries and territories with a rating of 2 have slightly weaker political rights than those with a rating of 1 because of such factors as some political corruption, limits on the functioning of political parties and opposition groups, and foreign or military influence on politics.

Ratings of 3, 4, 5 – Countries and territories with a rating of 3, 4, or 5 include those that moderately protect almost all political rights to those that more strongly protect some political rights while less strongly protecting others. The same factors that undermine freedom in countries with a rating of 2 may also weaken political rights in those with a rating of 3, 4, or 5, but to an increasingly greater extent at each successive rating.

Rating of 6 – Countries and territories with a rating of 6 have very restricted political rights. They are ruled by one-party or military dictatorships, religious hierarchies, or autocrats. They may allow a few political rights, such as some representation or autonomy for minority groups, and a few are traditional monarchies that tolerate political discussion and accept public petitions.

Rating of 7 – Countries and territories with a rating of 7 have few or no political rights because of severe government oppression, sometimes in combination with civil war. They may also lack an authoritative and functioning central government and suffer from extreme violence or warlord rule that dominates political power.

Civil Liberties

Rating of 1 – Countries and territories with a rating of 1 enjoy a wide range of civil liberties, including freedom of expression, assembly, association, education, and religion. They have an established and generally fair system of the rule of law (including an independent judiciary), allow free economic activity, and tend to strive for equality of opportunity for everyone, including women and minority groups.

Rating of 2 – Countries and territories with a rating of 2 have slightly weaker civil liberties than those with a rating of 1 because of such factors as some limits on media independence, restrictions on trade union activities, and discrimination against minority groups and women.

Ratings of 3, 4, 5 – Countries and territories with a rating of 3, 4, or 5 include those that moderately protect almost all civil liberties to those that more strongly protect some civil liberties while less strongly protecting others. The same factors that undermine freedom in countries with a rating of 2 may also weaken civil liberties in those with a rating of 3, 4, or 5, but to an increasingly greater extent at each successive rating.

Rating of 6 – Countries and territories with a rating of 6 have very restricted civil liberties. They strongly limit the rights of expression and association and frequently hold political prisoners. They may allow a few civil liberties, such as some religious and social freedoms, some highly restricted private business activity, and some open and free private discussion.

Rating of 7 – Countries and territories with a rating of 7 have few or no civil liberties. They allow virtually no freedom of expression or association, do not protect the rights of detainees and prisoners, and often control or dominate most economic activity.

The survey includes both analytical reports and the above numerical ratings – on a scale from 1 to 7 – for 195 countries and 14 territories in relation to political rights and civil liberties. The arithmetic mean of these two ratings determines an overall status of “Free” “Partly Free” or “Not Free” as follows: countries scoring between 1.0 and 2.5 are considered Free, those scoring 3.0–5.0 are Partly Free, and countries with a score from 5.5 to 7.0 are Not Free (cf., Fig. 4). The two ratings are connected, in the sense that, without a well-developed civil society, it is difficult, if not impossible, to have an environment supportive of political rights. Consequently, there is no country in the survey with a rating of 6 or 7 for civil liberties and, at the same time, a rating of 1 or 2 for political rights. Figure 4 shows the results of this methodology over a sample of countries (Freedom House 2018).
Fig. 4

Rating system of Freedom House Index for democracy in a sample of countries. Source: Freedom House 2018

Conclusion

Comparative evaluation systems (CESs) are an important approach in public administration and other research fields, because they can provide vital information to stakeholders for supporting decision-making processes to achieve specific objectives. Rating systems are the most widespread typology of CES.

In general, rating systems – applied to individuals, organizations, and countries – can be represented by a sequential process comprising four main stages: data collection, measurement of characteristics, ranking, and rating of entities, as described in Fig. 5.
Fig. 5

Sequential process of Rating Systems or CES

More specifically, Fig. 6 shows the construction of a rating system or CES that starts with a question or problem of evaluation, such as the performance of the best employees, the best organizations in providing services, or the countries with the best rule of law. The second step is data collection and measurement of characteristics concerning similar entities. After that, data analysis with specific techniques and methods provides a score for each entity to create a ranking of all entities. The last step is the rating (comparative evaluation) of entities to generate homogenous categories. The outputs are provided to stakeholders for decision-making processes supporting specific goals.
Fig. 6

Linear model of a rating systems

Final ratings can be expressed by numbers (1, 2, 3, …) or letters (A, B, C, D) or a combination of numbers/letters/symbols (A1, A2, A+, BB−, etc.) in increasing and/or decreasing order. They have some general properties:
  • Entities of higher rank have better performance (in terms of output, quality and/or good governance) than entities of lower rank. In general, the first numbers or letters in the ranking (e.g., A, A+, etc.) indicate the best performance.

  • If different entities have the same rating, then they have similar characteristics and/or performance.

  • Finally, it is possible to deduce the inventory of characteristics that are absent in the lowest rank entities by comparing them with the highest rank entities.

Overall, CES is a systematic process for data collection, measurement, and analysis of the characteristics of different entities to generate a final rating and support the decision-making processes of stakeholders to achieve specific goals. However, CES is a complex task because of manifold factors affecting entities that can change over time and space. To conclude, a comprehensive CES, affected by economic, social, psychological, anthropological, and perhaps biological factors of the entities under study, is a nontrivial exercise and can generate tentative results that need to be continuously verified over time.

Cross-References

References

  1. Coccia M (2005) A Scientometric model for the assessment of scientific research performance within public institutes. Scientometrics 65(3):307–321.  https://doi.org/10.1007/s11192-005-0276-1
  2. Coccia M, Rolfo S (2010) New entrepreneurial behaviour of public research organizations: opportunities and threats of technological services supply. Int J Serv Technol Manage 13(1/2):134–151.  https://doi.org/10.1504/IJSTM.2010.029674
  3. Coccia M, Rolfo S (2013) Human resource management and organizational behavior of public research institutions. Int J Public Admin 36(4):256–268.  https://doi.org/10.1080/01900692.2012.756889
  4. Coccia M, Benati I (2018a) Comparative studies Chapter 1197-1. In: Farazmand A (ed) Global encyclopedia of public administration, public policy, and governance –section bureaucracy. Springer International Publishing AG, Cham, pp 1–7.  https://doi.org/10.1007/978-3-319-31816-5_1197-1CrossRefGoogle Scholar
  5. Coccia M, Benati I (2018b) Comparative models of inquiry (Chapter 1199-1). In: Farazmand A (ed) Global encyclopedia of public administration, public policy, and governance. Springer International Publishing AG, Cham, pp 1–5. ISBN: 978-3-319-31816-5Google Scholar
  6. Cooksy LJ (1999) The meta-evaluand: the evaluation of project TEAMS. Am J Eval 20(1):123–136.  https://doi.org/10.1016/S1098-2140(99)80117-9CrossRefGoogle Scholar
  7. Freedom House (2018) Methodology. https://freedomhouse.org/report/freedom-world-2012/methodology. Accessed 10 June 2018
  8. Hospital Compare (2018a) The Official U.S. government site for medicare. https://www.medicare.gov/hospitalcompare/About/Hospital-overall-ratings.html. Accessed 18 June 2018
  9. Hospital Compare (2018b) The Official U.S. government site for medicare. https://www.medicare.gov/hospitalcompare/Data/Hospital-overall-ratings-calculation.html. Accessed 18 June 2018
  10. Hospital Quality Star Rating (2018) Hospital Quality Star Rating on Hospital Compare Public Input Period. Enhancements of the overall hospital quality star rating, Prepared by: Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE). https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/Downloads/Hospital-Compare-PC-2017.pdf. Accessed 18 June 2018
  11. Llorens JJ, Klingner DE, Nalbandian J (2018) Public personnel management. Contexts and strategies. Routledge, New YorkGoogle Scholar
  12. NRF (2018a) The National Research Foundation (NRF). http://www.nrf.ac.za/. Accessed 25 June 2018
  13. NRF (2018b) Reviews and evaluation (RE). http://www.nrf.ac.za/division/re/about. Accessed 25 June 2018
  14. Vartiainen P (2002) On the principles of comparative evaluation. Evaluation 8(3):359–371CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.CNR–National Research Council of ItalyTorinoItaly
  2. 2.Arizona State UniversityTempeUSA