Comparative Evaluation Systems
Comparative evaluation system is a systematic process for data collection, measurement, and analysis of the characteristics of different entities to generate a final rating and support decision-making processes of stakeholders for specific goals.
Evaluation of a single object and/or subject in general does not provide complete information for decision-making. In order to support a rational decision, comparative evaluation could be necessary. Vartiainen (2002) argues that comparative evaluation refers to a cognitive process in which the evaluation and the findings of the evaluation process are set within a comparative framework. This approach establishes an assessment on the evaluand (the subject of an evaluation), comparing it with objects of the same category (cf., Cooksy 1999). To put it differently, the attribution of value is in relative terms, through comparison with similar subjects and/or objects.
Different typologies of rating systems for individuals, organizations, and countries are described in next paragraphs. For the sake of clarity, some practical examples are provided that, of course, are not a comprehensive overview of all CES applications in the public administration.
Rating Systems for Individuals
National Research Foundation (NRF) of South Africa rating system for researchers
Researchers who are unequivocally recognized by their peers as leading international scholars in their field for the high quality and impact of their recent research outputs.
Researchers who enjoy considerable international recognition by their peers for the high quality and impact of their recent research outputs.
Established researchers with a sustained recent record of productivity in the field who are recognized by their peers as having:
- Produced a body of quality work, the core of which has coherence and attests to ongoing engagement with the field
- Demonstrated the ability to conceptualize problems and apply research methods to investigating them.
Young researchers (normally younger than 35 years of age), who have held the doctorate or equivalent qualification for less than 5 years at the time of application and who, on the basis of exceptional potential demonstrated in their published doctoral work and/or their research outputs in their early postdoctoral careers, are considered likely to become future international leaders in their field.
Young researchers (40 years or younger), who have held a doctorate or equivalent qualification for less than 5 years at the time of application, and who are recognized as having the potential to establish themselves as researchers within a 5-year period after evaluation, based on their performance and productivity of quality research outputs during their doctoral studies and/or early postdoctoral careers.
Rating System for Organizations
Safety of care
Effectiveness of care
Timeliness of care
Efficient use of medical imaging
Outpatients with low-back pain who had an MRI (magnetic resonance imaging) without trying recommended treatments first, such as physical therapy
Outpatient CT (computed tomography) scans of the abdomen that were “combination” (double) scans
Outpatient CT scans of the chest that were “combination” (double) scans
Outpatients who got cardiac imaging stress tests before low-risk outpatient surgery
Outpatients with brain CT scans who got a sinus CT scan at the same time
Rating System for Countries
Finally, rating systems or CES can also be applied to countries. An interesting example is the comparative evaluation of democracy level performed by Freedom House (2018), which is an independent organization dedicated to the expansion of freedom and democracy around the world. The information is useful for international investors, policy makers of international institutions, etc. Data for rating systems are collected by means of surveys measuring freedom according to two broad categories: political rights and civil liberties. Political rights enable people to freely participate in the political process, including the right to vote for distinct alternatives in legitimate elections, compete for public office, join political parties and organizations, and elect representatives who have a decisive impact on public policies and are accountable to the electorate. Civil liberties concern freedom of expression and belief, associational and organizational rights, rule of law, and personal autonomy without interference by the state. In particular, the rating process is based on a checklist of 10 questions on political rights and 15 questions on civil liberties. The questions on political rights are grouped into three subcategories: Electoral process (3 questions), political pluralism and participation (4), and functioning of government (3). The questions on civil liberties are grouped into four subcategories: freedom of expression and belief (4 questions), associational and organizational rights (3), rule of law (4), and personal autonomy and individual rights (4). The survey operates under the assumption that freedom for all peoples is best achieved in liberal democratic societies. General characteristics of each political rights and civil liberties rating are as follows (Freedom House 2018).
Rating of 1 – Countries and territories with a rating of 1 enjoy a wide range of political rights, including free and fair elections. Candidates who are elected actually rule, political parties are competitive, the opposition plays an important role and enjoys real power, and minority groups have reasonable self-government or can participate in the government through informal consensus.
Rating of 2 – Countries and territories with a rating of 2 have slightly weaker political rights than those with a rating of 1 because of such factors as some political corruption, limits on the functioning of political parties and opposition groups, and foreign or military influence on politics.
Ratings of 3, 4, 5 – Countries and territories with a rating of 3, 4, or 5 include those that moderately protect almost all political rights to those that more strongly protect some political rights while less strongly protecting others. The same factors that undermine freedom in countries with a rating of 2 may also weaken political rights in those with a rating of 3, 4, or 5, but to an increasingly greater extent at each successive rating.
Rating of 6 – Countries and territories with a rating of 6 have very restricted political rights. They are ruled by one-party or military dictatorships, religious hierarchies, or autocrats. They may allow a few political rights, such as some representation or autonomy for minority groups, and a few are traditional monarchies that tolerate political discussion and accept public petitions.
Rating of 7 – Countries and territories with a rating of 7 have few or no political rights because of severe government oppression, sometimes in combination with civil war. They may also lack an authoritative and functioning central government and suffer from extreme violence or warlord rule that dominates political power.
Rating of 1 – Countries and territories with a rating of 1 enjoy a wide range of civil liberties, including freedom of expression, assembly, association, education, and religion. They have an established and generally fair system of the rule of law (including an independent judiciary), allow free economic activity, and tend to strive for equality of opportunity for everyone, including women and minority groups.
Rating of 2 – Countries and territories with a rating of 2 have slightly weaker civil liberties than those with a rating of 1 because of such factors as some limits on media independence, restrictions on trade union activities, and discrimination against minority groups and women.
Ratings of 3, 4, 5 – Countries and territories with a rating of 3, 4, or 5 include those that moderately protect almost all civil liberties to those that more strongly protect some civil liberties while less strongly protecting others. The same factors that undermine freedom in countries with a rating of 2 may also weaken civil liberties in those with a rating of 3, 4, or 5, but to an increasingly greater extent at each successive rating.
Rating of 6 – Countries and territories with a rating of 6 have very restricted civil liberties. They strongly limit the rights of expression and association and frequently hold political prisoners. They may allow a few civil liberties, such as some religious and social freedoms, some highly restricted private business activity, and some open and free private discussion.
Rating of 7 – Countries and territories with a rating of 7 have few or no civil liberties. They allow virtually no freedom of expression or association, do not protect the rights of detainees and prisoners, and often control or dominate most economic activity.
Comparative evaluation systems (CESs) are an important approach in public administration and other research fields, because they can provide vital information to stakeholders for supporting decision-making processes to achieve specific objectives. Rating systems are the most widespread typology of CES.
Entities of higher rank have better performance (in terms of output, quality and/or good governance) than entities of lower rank. In general, the first numbers or letters in the ranking (e.g., A, A+, etc.) indicate the best performance.
If different entities have the same rating, then they have similar characteristics and/or performance.
Finally, it is possible to deduce the inventory of characteristics that are absent in the lowest rank entities by comparing them with the highest rank entities.
Overall, CES is a systematic process for data collection, measurement, and analysis of the characteristics of different entities to generate a final rating and support the decision-making processes of stakeholders to achieve specific goals. However, CES is a complex task because of manifold factors affecting entities that can change over time and space. To conclude, a comprehensive CES, affected by economic, social, psychological, anthropological, and perhaps biological factors of the entities under study, is a nontrivial exercise and can generate tentative results that need to be continuously verified over time.
- Coccia M (2005) A Scientometric model for the assessment of scientific research performance within public institutes. Scientometrics 65(3):307–321. https://doi.org/10.1007/s11192-005-0276-1
- Coccia M, Rolfo S (2010) New entrepreneurial behaviour of public research organizations: opportunities and threats of technological services supply. Int J Serv Technol Manage 13(1/2):134–151. https://doi.org/10.1504/IJSTM.2010.029674
- Coccia M, Rolfo S (2013) Human resource management and organizational behavior of public research institutions. Int J Public Admin 36(4):256–268. https://doi.org/10.1080/01900692.2012.756889
- Coccia M, Benati I (2018a) Comparative studies Chapter 1197-1. In: Farazmand A (ed) Global encyclopedia of public administration, public policy, and governance –section bureaucracy. Springer International Publishing AG, Cham, pp 1–7. https://doi.org/10.1007/978-3-319-31816-5_1197-1CrossRefGoogle Scholar
- Coccia M, Benati I (2018b) Comparative models of inquiry (Chapter 1199-1). In: Farazmand A (ed) Global encyclopedia of public administration, public policy, and governance. Springer International Publishing AG, Cham, pp 1–5. ISBN: 978-3-319-31816-5Google Scholar
- Freedom House (2018) Methodology. https://freedomhouse.org/report/freedom-world-2012/methodology. Accessed 10 June 2018
- Hospital Compare (2018a) The Official U.S. government site for medicare. https://www.medicare.gov/hospitalcompare/About/Hospital-overall-ratings.html. Accessed 18 June 2018
- Hospital Compare (2018b) The Official U.S. government site for medicare. https://www.medicare.gov/hospitalcompare/Data/Hospital-overall-ratings-calculation.html. Accessed 18 June 2018
- Hospital Quality Star Rating (2018) Hospital Quality Star Rating on Hospital Compare Public Input Period. Enhancements of the overall hospital quality star rating, Prepared by: Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE). https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/Downloads/Hospital-Compare-PC-2017.pdf. Accessed 18 June 2018
- Llorens JJ, Klingner DE, Nalbandian J (2018) Public personnel management. Contexts and strategies. Routledge, New YorkGoogle Scholar
- NRF (2018a) The National Research Foundation (NRF). http://www.nrf.ac.za/. Accessed 25 June 2018
- NRF (2018b) Reviews and evaluation (RE). http://www.nrf.ac.za/division/re/about. Accessed 25 June 2018