Triangulation of Theoretical and Empirical Conceptualizations Related to the Rule of Law

Abstract

Triangulation of the various methodological approaches towards the rule of law is highly desirable since it allows the combination of benefits and elimination of problematic aspects in each. The present article triangulates the conceptualizations of three approaches relating to the rule of law, namely Bedner’s review of the rule of law theories, the World Justice Project’s Rule of Law Index, and the review by Horák et al. of empirical measurement of legal consciousness, and identifies the most significant problem as a lack of communication between them. More precisely, the theoretical conceptualizations are not fully prepared for empirical measurement, and the empirical tools do not reflect the theoretical debate and its outcomes. Therefore, a new conceptualization of the rule of law is proposed to overcome these issues and consequently make the empirical measurement more valid.

1 Introduction

Numerous ways of conceptualizing the rule of law (hereinafter “RoL”) can be found in the literature, ranging from purely theoretical approaches¹ to quantitative empirical research.² It is also well known and described how significantly the particular conceptualizations of the RoL differ from each other. The article follows up the idea and perhaps also accepts the challenge by Taekema, who suggested that the various approaches to the RoL can and ought to be triangulated.³

The idea of methodological triangulation in terms of legal science has already been well explained and advocated as a means to combine the advantages and simultaneously minimize the disadvantages of various research methods and methodological approaches towards the researched object (in this case the RoL), based on “using different methods to answer the same question in order to corroborate or complement results”.⁴

The present article therefore focuses on triangulation of RoL conceptualizations resulting in a novel conceptualization and operationalization of the RoL construct which is rooted in normative/conceptual theories and also allows for its empirical measurement in the general population. To do so, we inspect the possibilities and obstacles of triangulating the three conceptualizations related to the construct of the RoL, namely the normative/conceptual theories of the RoL,⁵ the quantitative assessment tools of the RoL (represented most frequently by various indexes of the RoL)⁶ and (perhaps quite surprisingly) the empirically measurable construct of legal consciousness (hereinafter “LC”).⁷

In looking for the aspects of normative/conceptual theories on the RoL which can be triangulated with empirical measurement, the “thin” theories⁸ that provide clear lists of principles (or demands), such as those introduced by Fuller,⁹ Finnis¹⁰ or Raz,¹¹ represent a good starting point since they are commensurable with the other two conceptualizations. It is worth mentioning that even though these lists primarily focus on qualities which the legal system should possess, they are, as Taekema showed,¹² based on presuppositions concerning the relationship between the legal system and its addressees. For example, Fuller’s story of Rex attempting to regulate the people¹³ is entirely based on the ability of individuals to obey their legal rules. In a similar manner, both Hayek and Raz endorsed that law foremost needs to be “possible to obey by its addressees”¹⁴ and “capable of guiding its subjects”.¹⁵ Waldron also based his procedural theory of the RoL on his own assumptions of how ordinary people perceive it.¹⁶

The substantive theories of the RoL which address the content of legal rules and the social, cultural and political context of the legal order¹⁷ and consequently introduce a broader spectrum of demands and criteria aimed at distinguishing between “bad” and “good” laws¹⁸ are naturally even more profoundly based on these sociological and psychological presuppositions.

The presuppositions concerning the relationship between the legal system and its addressees appear to be necessary for any articulation of the theory of the RoL. However, they remain mere assumptions made by legal philosophers until they are verified with empirical measurement. In other words, these presuppositions are (a) the reason why triangulation in RoL research is highly desirable and mutually beneficial¹⁹ and also (b) the research object shared by all three conceptualizations described in detail below which makes their triangulation possible.¹⁹

In this context, it is evident why empirical measurement tools should naturally focus on the general population (i.e., the addressees) when measuring the level of the RoL.²⁰ However, as shown in the review work on the quantitative assessment tools of the RoL, almost all of them instead focus on legal experts.²¹ Nevertheless, such an approach is similar to the conceptual/normative approach, as the relationship between the legal system and individuals is again assumed by legal experts rather than measured in the population. The one important exception here is the Rule of Law Index, measured by the World Justice Project (hereinafter “WJP Index” or “Index”),²² which we discuss later.

Indeed, the RoL has for a long time been perceived as the domain of legal theory rather than empirical social sciences, thus restricting the methodology used for its measurement. Hence, we believe that an interdisciplinary construct rooted in empirical methodology should be introduced to facilitate the triangulation procedure. In asking which construct is both empirically measurable and focused on the “ways ordinary people think of, talk about, and understand law in their everyday lives”²³ (i.e., the aforementioned presuppositions), the ideal choice would be LC.

For a deeper explanation of this argument, let us compare the research goals of the three conceptualizations in the following triangulation. First, the normative/conceptual theories of the RoL apply the aforementioned presuppositions to generate normative statements, for example, law should be promulgated, prospective, consistent, relatively stable, etc. to be intelligible and understandable by its addressees or law should be guided by certain values and principles (right to a fair trial, freedom and equality before the law, respect for dignity, etc.) to gain the trust and support of the community. Second, the quantitative assessment tools of the RoL focus on estimation of the degree to which the particular legal system satisfies these normative statements (demands) and consequently attempt to answer questions such as: Do the respondents (mostly legal professionals) perceive law as promulgated, prospective, consistent, stable, etc. to a sufficient degree?; To what degree is the legal system based on certain (supposedly desirable) values and principles? Finally, empirical measurement of LC analyzes the degree to which the presuppositions used to generate normative statements resemble reality, for example, asking questions such as: To what degree does the general population truly know and understand the legal rules governing its behaviour, and what variables affect this degree of knowledge and understanding most significantly?; What are the values and principles which the general population identifies with, and to what degree does the legal system correspond with them?

Consequently, we argue that the three conceptualizations use different perspectives to examine the same studied object, which at its core is the set of conditions under which the relationship between the legal system and its addressees is effective and mutually beneficial. These three conceptualizations are therefore commensurable and can be successfully triangulated.

2 Triangulated Conceptualizations

2.1 Rule of Law Theories

The RoL is a rather broad and multi-layered phenomenon.²⁴ The normative demands and consequently also theories of the RoL can be classified according to two criteria: the thickness (thin v. thick)²⁵ of the demands and the character of the demands (formal v. substantive).²⁶ We applied the conceptualization introduced by Bedner since it not only features both of these criteria but also divides the latter into three dimensions (procedural v. substantive v. institutional)²⁷ to provide a comprehensive analytical framework.²⁸

We have slightly adjusted this conceptualization to better suit triangulation. For procedural/formal (hereinafter “formal”) demands, the thinnest is that the state exercises its power over citizens using law and that the state itself and its institutions are subject to law (rule by law and rule of law; RoL1). This elementary demand tends to protect citizens by preventing the arbitrariness of state power.²⁹ The second set of formal demands focuses on the form and structure of legal rules which must be satisfied to permit the bare existence of law (clarity, prospectivity, promulgation, generality, etc.).³⁰ These demands are generally called formal legality (RoL2). The third demand, which has a hybrid character and therefore bridges the gap between formal and substantive dimensions, is that the law is created in a manner which allows the addressees to consider themselves its authors and guarantees their consent to the content of the law³¹ (procedural democracy; RoL3).

For substantive demands, a thinner one is that the law should be in accordance with a system of fundamental values and principles (freedom, dignity, equality, fair trial, etc.). Even if it is often claimed that these principles are universal or even implicit in the legal system,³² the list of recognized values and principles (including their interpretation)³³ varies over time and is highly context-sensitive,³⁴ which is unavoidable: as Barber mentions, “the rule of law needs the support of the community if it is to flourish”.³⁵ This demand is called the fundamental principles of justice (RoL4). A thicker substantive demand then complements the aforementioned values and principles with the recognition and protection of human rights and liberties (human rights; RoL5). Depending on the thickness of the approach, either exclusively classic liberal individual rights and liberties may be incorporated³⁶ or the younger generations of human rights (e.g., economic, social, cultural or group rights) may also be involved.³⁷

Originally understood as a formal demand, one more remains to be added to the list. This demand states that the law should be interpreted, applied and enforced by independent, impartial and sufficiently accessible institutions which respect the due process rights³⁸ (most often judiciary, but also other specialized institutions such as anti-discrimination tribunals, ombudsmen, etc., may be included). This demand is called enforcement mechanisms; RoL6). The entire conceptualization of the theoretical RoL demands is given in Table 1.

Table 1 Conceptualization of the theories of the rule of law

2.2 Quantitative RoL Assessment Tools

To conceptualize a quantitative approach to the RoL, we selected the WJP Index for the following reason: it is the only index which focuses on information which is gathered from both legal experts and general population polls.³⁹ Consequently, in contrast to numerous other methods, which focus on estimating the degrees to which the particular legal systems satisfy these normative statements (demands), the Index also examines, at least to some extent, how the RoL is experienced and perceived in various populations.

The Index currently also represents the most comprehensive attempt to measure the RoL. In the latest version of the Index, the authors identified eight factors (hereinafter “WJP factors”) and 44 subfactors (hereinafter “WJP subfactors”) in 128 countries. These factors and subfactors were calculated according to 127 perception-based and 213 experience-based items. The final sum indexes of the RoL are aggregated (with different weights) from the results obtained from the general population, expert populations and from some related country statistics (e.g., UNODC Homicide Statistics, rate per 100,000 population). Its findings are also generated repeatedly according to appropriate advanced statistical procedures, and its results are therefore undoubtedly some of the most reliable and valid in this research field.⁴⁰ The conceptualization used by the WJP Index is given in Table 2.

Table 2 Conceptualization according to the World Justice Project’s rule of law Index

It should be noted, however, that even though the WJP Index seems to be statistically and methodologically rigorous, the authors do not report any comprehensive data analysis of psychometric properties with each wave of data collection. In fact, knowledge of the statistical procedure of the Index is limited since the main results were published in an article⁴¹ which is ten years old and in two statistical audits from 2012–2013⁴² and 2014.⁴³ Evidence of the applied statistical procedures which estimate properties such as the validity and reliability of the measures applied in the current results of the Index are therefore rather unknown. Nevertheless, based on these sources, we can say that the Index simply calculates the scores of subfactors by aggregating the scores of items into arithmetic means and subsequently weighting them according to rather arbitrary coefficients.⁴⁴ This step is not necessarily wrong, but it reveals some problematic features concerning the WJP conceptualization, discussed below.

2.3 Legal Consciousness

Of the various existing LC conceptualizations,⁴⁵ the most comprehensive and yet empirically measurable was introduced by Horák et al., who performed a systematic review and found six main components (dimensions) in the literature. According to them:

“legal consciousness is a complex of law-related knowledge, skills, attitudes, beliefs, and values of an individual, whereby the mutual relationship between the individual and law is being created, deepened, and developed within the context of a specific society and legal system, providing such system with the necessary authority and legitimacy for the regulation of human behavior.”⁴⁶

The first featured component (LC1) is called general legal knowledge and represents the basic knowledge of law, legal institutions and procedures.⁴⁷ Besides general legal knowledge, legally conscious⁴⁸ citizens need to possess the necessary legal skills. Even mastery of legal information would likely render the citizen helpless in many legal situations without an adequate level of legal capability. The second component (LC2) is called legal awareness. Legally conscious individuals should also possess more detailed knowledge regarding specific legal situations (e.g., entering into a labour contract or filing a warranty claim for defective footwear).⁴⁹ This component (LC3) is called ad hoc legal knowledge. Since three components reflect the intelligibility and understandability of law in the population (i.e., the features addressed by formal RoL theories), we call them formal LC components.

Legally conscious citizens not only acquire knowledge of the legal system and how to use its tools but also learn to critically evaluate the existing system and its concrete outputs (e.g., court decisions, administrative acts or passed legislation) and sometimes even propose respective changes. LC researchers further distinguish between rationally based (i.e., opinion about law; LC4)⁵⁰ and emotionally based (i.e., trust in law; LC5)⁵¹ evaluation of the legal system. The LC construct also contains one more component which focuses on the most abstract relationship between the law and its addressees: legal identity; LC6.⁵² It examines whether citizens share common core values and principles (liberal-democratic, Marxist-Leninist, orthodox Islamic, etc.) with the legal system,⁵³ meaning that LC transcends the axiological core of the current legal system and examines the degree to which it resembles the moral framework of citizens and their community. Contrary to formal components, LC4–LC6 components focus on community support for the legal system and its fundamental values and principles (i.e., the features covered primarily by substantive RoL theories). These three components may therefore be called substantive LC components.

Shown in Table 3, the aforementioned components together create a unique and empirically measurable system of variables suitable for assessing the levels of LC in the population.⁵⁴ It covers all the cognitive and attitudinal aspects of this relationship, enabling the empirical examination of how and why individuals tend to obey or disobey legal rules in their everyday lives.

Table 3 Conceptualization of Legal Consciousness

Having studied the available empirical research, we are aware that the methodological quality of empirical measurement of LC varies significantly. For example, quantitative measurement of LC4–LC6 has been rather scarce (estimated mostly via qualitative methodological approaches). Moreover, all the empirical research covered in this article focused separately on single LC aspects (often being unaware of the existence of others) and could not therefore even aim to explore the relationship between the individual and law in a comprehensive manner. The main reason is that the complex analytical framework and background theory were missing. We believe that this article is a significant step towards bridging these gaps in the current research

3 Methodology

As already mentioned, to fulfil the aim of the article, we applied methodological triangulation of the RoL, WJP Index and LC conceptualizations.

In the first step, we analysed the individual components of the three triangulated conceptualizations (i.e., RoL demands, LC components and WJP factors) described in the previous sections with respect to all their possible linguistic meanings and connotations. In the case of the WJP Index, our analysis considered, besides the main factors, the subfactors to describe all the potential meanings of the WJP factors and their indicators. Additionally, all relevant items measured by the WJP Index were analysed to ensure that the item wording corresponded to the general description of the underlying factors and subfactors.

In the second step, we added these components to a table where cellsrepresent dichotomized variables of mutual resemblance (i.e., components either resemble each other = */, or they do not resemble each other = N/A). Subsequently, the table was modified and divided into three tables, each including pairwise comparisons for one of the analysed conceptualization pairs (i.e., WJP subfactors with RoL demands, WJP subfactors with LC components and RoL demands with LC components). Using these tables allowed us to not only clearly demonstrate which components are shared across two or even all three conceptualizations but also identify all potential overlaps.

In the third step, all possible pairwise comparisons were inspected by two independent researchers (raters) with scientific backgrounds in law (1st author) and empirical measurement (2nd author). This allowed us to bridge the gap between social sciences and law and to mitigate the risks stemming from the multidisciplinary nature of studied constructs (i.e., to identify and properly solve both the theoretically and methodologically based issues in the analysed conceptualizations). Both raters needed to agree on the resemblance of components. In cases where the raters were not able to agree, discussion and eventually also deeper analysis of the primary literature on both the RoL and LC constructs was pursued to reach consensus.⁵⁵

Finally, we adopted some additional restrictions so that triangulation was meaningful. The most stringent restriction regarding comparison of the WJP Index and LC was the omission of all WJP subfactors which did not measure the RoL in the general population at the item-level (they are indicated with “–” in the table), because the core of the LC construct is measurement in the population, and consequently those WJP subfactors which were not measured in the population were not able to resemble any of the LC components. Of the WJP subfactors which measured the RoL in the general population, some contained only items which focused on the experience of respondents with a legal system. These subfactors were therefore also omitted, since they do not directly relate to an LC construct in any way.

The third restriction concerns the comparison of the RoL and WJP Index with LC, in which we reported only direct mutual resemblance, not indirect associations. Even though it is possible that any change in WJP subfactors or RoL demands would manifest in LC components, these indirect manifestations were not indicated in the table, as these types of manifestation could be assumed in almost all pairwise comparisons.

4 General Results

The complete results are given in three tables, each representing one of the conducted pairwise comparisons (see Appendix: Table 1–3). This section therefore describes only a brief interpretation of agreement between the conceptualizations.

The most important general finding clearly shows that all RoL demands were somehow measured by the WJP Index or LC. Both empirical approaches also complement each other effectively; for example, LC measured the RoL2 comprehensively (with three components) in contrast to the WJP Index (which uses only one subfactor), and conversely, the WJP Index measured the RoL1 with nineteen subfactors whereas LC completely omitted it. The combination of LC and the WJP Index therefore provides a solid empirical measurement of the RoL. This finding also means that all RoL demands are already empirically measured, allowing scholars to verify and possibly even compare the current level of the RoL in various populations.

Surprisingly, even though we identified not negligible agreement between the theoretically proposed RoL demands and empirical measurement, agreement between both representatives of empirical measurement was rather scarce. This finding, however, again emphasizes the necessity of combining already established empirical approaches for the assessment of the level of the RoL in all its complexity.

5 Discussion

In this chapter, we discuss the three most important feedbacks which may be drawn from our results for both the theory of the RoL and its measurement. First, we focus on the most general and abstract issue, which is the nature of the RoL construct itself. More specifically, we discuss the possible ways of approaching the RoL from the perspectives of relativism and universalism. Second, we stress the issue of insufficient resemblance between the theoretical and empirical approaches towards the RoL. We analyse the reasons why the current theoretical conceptualization is not prepared for empirical measurement and why empirical measurement does not reflect certain important theoretical perspectives; we also provide recommendations and suggest solutions for these issues. Finally, we closely examine the most problematic element of both the theory and empirical measurement of the RoL, which is effective law enforcement.

We would like to note that this chapter does not contain any specific observations concerning LC conceptualization since the main purpose behind the article and triangulation is the improvement of empirical measurement and the theory of the RoL rather than LC, which we only applied as an instrument to enhance the process of triangulation.

5.1 Nature of the Construct of the Rule of Law

The first and perhaps most important feedback relates to the highly normative nature of the RoL itself, having critical consequences for both the theory and empirical measurement of the construct. As explained above, the rule of law is in its essence a set of demands which must be satisfied by the legal system for it to function well within society. In other words, the construct of the RoL inherently prescribes what ought to be rather than describes what actually is. Depending on the thickness and substantiveness of included demands, the RoL sets a more or less specific normative ideal to be followed.⁵⁶

At the same time, however, there is a desire to use the RoL as an objective analytical tool which would be applicable to all countries regardless of their legal culture or political ideology, as demonstrated by the geographical scope of the WJP Index. Nevertheless, this desire unfortunately conflicts with the normative nature of the construct.⁵⁷ The thicker and more substantive the conceptualization of the RoL, the more relativistic (culturally and ideologically determined) a normative ideal it provides, and conversely, the more objective and universalistic analytical tool the RoL wants to become, the thinner and rather formal conceptualization it requires.⁵⁸

We suggest that these two conflicting aspects can be balanced in several ways, each having advantages and disadvantages. Each of these approaches varies in its level of cultural universalism and cultural relativism.⁵⁹ Although this distinction was understood as dichotomy in the past, it rather represents a continuum composed of four anchors: extreme relativism, moderate relativism, moderate universalism and extreme universalism (Fig. 1). We argue that the RoL theory must deal with this specific continuum and incorporate it into its core definition so that it may be empirically measured correctly.

Fig. 1

Possible implementations of universalism and relativism in measurement of the RoL

The first option is to apply an extreme universalistic point of view. The RoL in this approach represents not only a culturally invariant construct which is shared by the whole of humankind but also a construct which is not influenced in any way by culture. We argue that the only feasible implementation of the current RoL demands into extreme universalism (and achieving its very strong assumptions) lies in using the narrow conceptualization of the RoL (i.e., to include only the thin formal RoL demands). But this type of approach is completely insensitive to the differences between legal systems once the basic level of the RoL has been achieved and can be problematic because, as Barber notes, “there are plenty of real world legal orders that succeed in meeting the minimum demands of the principle but which still fall far short of the ideal”.⁶⁰

The other potential (and from the perspective of legal pluralism rather problematic western-biased)⁶¹ implementation of extreme universalism might be observed in the use of the RoL conceptualization as a normative ideal rather than an analytical tool. In this case, the thick version of the RoL could be applied universally, although only the compliance of all states with western (and therefore most often liberal democratic) normative ideals is measured. Another problem with this approach is that it significantly overlaps the already existing tools which measure liberal democracy⁶² and therefore is somewhat redundant. Unfortunately, this perspective can be implicitly seen in the WJP Index, whose aim is to be a globally applicable universal analytical tool and yet it measures the various RoL demands regardless of their thickness or substantiveness.

A desirable solution here would be to explicitly classify individual subfactors or even individual items of the Index with regard to their thickness and substantiveness. This classification would allow the determination of whether the analysed countries fail to satisfy the thin and formal demands which are preconditions for the rule of law to even exist, or whether they achieve lower scores in only the thicker and more substantive demands which measure the rule of law from a rather culturally and ideologically based liberal-democratic perspective. This is especially important, since the problems caused by the low level of thin (failure of law) and thick (failure of democracy) demands have different natures and therefore can be overcome by different measures.⁶³

The second option is the moderate universalism framework wherein the broader yet “objectivised” conceptualization of the RoL is applied (i.e., to include both thin formal and objectivised thin substantive RoL demands) as a universal analytical tool, applicable again to any legal system. By objectivization, we mean abandoning those substantive demands which are associated with the normative ideal of liberal democracy and replacing them with neutral demands derived from substantive LC components (i.e., opinion about law, trust in law and legal identity). Abandoning the normative ideal, however, suggests that non-democratic and illiberal systems may potentially score higher than liberal democracies when supported sufficiently by the community. As Raz states:

“A non-democratic legal system, based on the denial of human rights, on extensive poverty, on racial segregation, sexual inequalities, and religious persecution may, in principle, conform to the requirements of the rule of law better than any of the legal systems of the more enlightened Western democracies. This does not mean that it will be better than those Western democracies. It will be an immeasurably worse legal system, but it will excel in one respect: in its conformity to the rule of law.”⁶⁴

The third option is to use the framework of moderate relativism and therefore a broader and not objectivised conceptualization of the RoL (i.e., to include not only thin but also thick formal and non-objectivised thin substantive demands) as an analytical tool applicable to a group of countries whose legal systems share their fundamental values (e.g., western liberal democracies, Muslim theocracies, military dictatorships, etc.). This approach therefore might yield several groups of countries according to the demands shared by their citizens. Even though this approach presupposes a universalistic aim in the RoL, it is built on the assumption that the RoL depends on a particular context and legal culture,⁶⁵ and hence, it cannot be measured (or even compared) beyond the countries of one specific group.

The fourth and last option represents extreme relativism, which postulates that facts can be derived only from a specific legal system due to the major influence of the culture. Within this framework, scholars have to use the broadest customised conceptualizations of the RoL (i.e., to include thin and thick formal, non-objectivised thin substantive and customised thick substantive demands) as an ad hoc analytical tool applicable to each legal system individually. By customisation, we mean reformulating all the thick substantive RoL demands in a manner which allows the cultural, historical and other specifics of each individual country, society or even community to be reflected.⁶⁶ However, the results would not be comparable with others since this approach naturally lacks any cross-cultural comparability.⁶⁷

Even though all the positions might be legitimate depending on the research aim, we argue that compared to the mid positions, the extreme positions are extremely reductionistic (i.e., extreme universalism excessively reduces the broadness of the RoL construct, and extreme relativism lacks any comparative aims). Hence, the mid positions should be prefered over the extreme ones.

5.2 Resemblance between the Theory and Empirical Measurement

The insufficient resemblance between the theory and empirical measurement can be clearly demonstrated through the enormous number of overlaps between the RoL demands and the LC components and WJP subfactors or even factors, i.e. the situations where one LC component or WJP subfactor/factor resembles more than one RoL demand (bottom-up overlaps), and conversely, where one RoL demand corresponds to more than one LC component or WJP subfactor/factor (top-down overlaps).

As a feedback for the theory, we can say that both types of overlap suggest that the RoL demands are considerably vague, but they differ in the causes of this vagueness. In summary, the bottom-up overlaps indicate that the RoL demands overlap each other (i.e., the same element measurable by a single empirical dimension or factor is contained in more than one of them). Solving this problem requires precisely distinguishing the individual RoL demands and establishing strict and clear boundaries between them. The top-down overlaps suggest that the RoL demands are defined too broadly and therefore cannot be measured by a single empirical dimension or factor. This problem can be generally overcome by more precise and specific definitions of the individual RoL demands.

Regarding the most important bottom-up overlaps, the LC2 component matched the RoL2 and RoL6 demands, and LC6 resembled the RoL3, RoL4 and RoL5 demands (two out of six, 33.3%; and three out of six, 50%, respectively). We would also like to highlight the WJP1F subfactor, which corresponds to the RoL1 and RoL3 demands, the WJP4C, WJP4D and WJP4E subfactors, which all resemble the RoL3 and RoL4 demands, and finally also the numerous WJP subfactors (WJP1B, WJP2B, WJP7C, WJP7D, WJP8E, WJP8F), which all matched the RoL1 and RoL6 demands (two out of six, 33.3% in all three cases).

Concerning the most important cases of top–down overlaps, the RoL1, RoL5 and RoL6 demands resembled a large number of various WJP subfactors (nineteen out of 44, 43.2%; ten out of 44, 22.7%; and thirteen out of 44, 29.5%, respectively) and even factors (five out of eight, 62.5%; and four out of eight, 50% in the latter two cases, respectively), and similarly, the RoL2, RoL3 and RoL6 demands corresponded to a high proportion of LC components (three out of six, 50%; and two out of six, 33.3% for the latter two cases, respectively).

The overlaps show that the individual demands of the RoL conceptualization are vaguely defined, which makes their precise operationalization and subsequent empirical measurement rather difficult. As a consequence, we can say that the theoretical conceptualization of RoL, without proper adjustments, is not prepared for empirical measurement. The cause of this issue can be identified in the analysed conceptualization of the RoL in fact being a review which attempts⁶⁸ to conceptualize the elements of different definitions of the RoL that do not fully correspond to each other because legal scholars are divided on the issue of how thin or thick the proper RoL conceptualization should be.⁶⁹

These problems can be found not only at the level of the entire conceptualization (i.e., which RoL demands are included in the mix and which are not) but also at the level of individual demands. More specifically, three problematic aspects of the conceptualization of individual RoL demands can be identified.

First, several elements can be found which are featured by more than one RoL demand simultaneously. As a consequence, overlapping demands which typically cause bottom-up overlaps can be identified. Two important examples can be described in this case. Both RoL1 and RoL6 demands feature independence of the courts and the due process of law before them, the former as a mechanism to ensure lawfulness and to prevent the arbitrary exercise of public power and the latter as a core aspect of the institutional RoL demand, with the same purpose. Hence, this element is duplicated in the conceptualization. Similarly, RoL3 and RoL4 are challenging to distinguish from each other, as some of the human rights (e.g., the right to dignity and non-discrimination, rights of the accused, including presumption of innocence, and elementary freedom, including freedom of opinion and belief) can also be understood as fundamental values upon which the legal system is based. The line between fundamental values and principles on the one hand and human rights on the other is so thin, especially in liberal democracies, that distinguishing them is conceptually more damaging than beneficial.

A solution to this problem would be merging the overlapping RoL demands, or alternatively (if the overlap did not include the entirety of at least one of the merged demands), excluding the overlapping elements from one of the demands and including it in the other. In our case, the simplest solution would be to merge RoL6 into RoL1 and RoL5 into RoL4.

Second, some of the RoL demands are hybrid in nature, blending formal, substantive and institutional aspects and consequently causing top-down overlaps. An example here is the RoL3 demand, which contains both the formal (the democratic process ensures that addressees consider themselves co-authors of the legal rules) and substantive (the consent of citizens with the content of legal rules) aspects.

This problem can be solved by dividing hybrid demands into parts and incorporating each part into the appropriate RoL demand (in this case, the formal aspects of RoL3 into RoL1 and the substantive ones into RoL4). If such inclusion is not possible, an alternative solution is to keep the divided parts as new, self-standing RoL demands.

Third, some of the RoL demands are defined too vaguely and therefore can be understood and interpreted in a thinner or thicker manner, again leading to overlapping or hybridisation of individual demands and hence causing both types of overlap. RoL1, for example, can either simply demand that the exercise of power is carried out via positive law, or several thicker demands⁷⁰ can be added to the mix. Similarly, RoL5 may demand the protection of solely negative individual fundamental rights and liberties, or also include labour, economic, social, cultural or group rights. Even though RoL2 seems to demand clearly defined principles which ensure that the legal system is intelligible and understandable (general, public, prospective, certain, etc. laws), the LC1–LC3 components suggest that thicker demands which ensure the law is indeed known and understood by the vast majority of its addressees (plain language, sufficient legal education of citizens, using the Internet and social networks to promote general knowledge and understanding of the law, affordable legal advice, etc.) can also be included here, because intelligible and understandable do not automatically mean known and understood.

In this case, overcoming the problem seems to be the most laborious since it is composed of three steps: (a) clear definition of the core ideas (purposes) of individual demands, (b) creation of lists of sub-demands bound to these core ideas for each demand, and (c) classification of these sub-demands with regard to their thickness.

The problem is not, however, solely on the side of theory. Consequently, as a feedback for the empirical measurement, we should stress that the WJP Index does not reflect the theoretical debate on the RoL and the subsequent conceptualizations which emerged in this research field. This has important consequences.

First, the WJP Index works with its own created clusters of items and subfactors which do not correspond to individual RoL demands and thus do not respect the borders delineated from the theoretical distinction between formal and substantive RoL demands. Creating various clusters based on the obtained data is a statistically legitimate way of finding results which correspond to the RoL in the real population and in most cases is also more accurate than the clusters defined by the theory. However, this approach is jeopardized by the lack of content validity since such unbalanced measurement might not correctly reflect the proposed theoretical construct.⁷¹

Hence, the aforementioned overlaps are not caused solely by the vague theoretical definitions of the RoL demands but also by the inability of the WJP Index to reflect the theory. A good example is the WJP3 factor called “open government”, which measures some properties of the formal RoL2 demand (e.g., promulgation and accessibility of laws), the right to information, which belongs to the substantive RoL5 demand, and also participation and complaint mechanisms, which resemble the hybrid RoL3 demand. Consequently, we suggest not only more precision in conceptualizing the theoretical RoL construct but also reformulation of WJP factors for better correspondence with the generally accepted theoretical distinctions between formal and substantive demands.

Second, some crucial theoretical RoL demands are measured insufficiently by the WJP whereas others might be overrepresented. The RoL2 demand is a good demonstration of an underrepresented measurement. The WJP Index features this complex demand, which consists of numerous principles of formal legality, such as the clarity, prospectivity, promulgation, accessibility, generality, coherence, relative stability, etc. of legal rules⁷² in only one subfactor (WJP3A), which of all these requirements contains only promulgation, clarity and accessibility. Other requirements are completely omitted by the Index and should definitely be added to the mix.

An overrepresentation can be observed on the RoL1 (measured by nineteen subfactors) and RoL6 (measured by thirteen subfactors) demands. Applying this number of subfactors to measure a single RoL demand is not necessarily problematic, but in the case of the WJP Index, it leads to incorporating thicker or more substantive elements of the relevant demands, for example, an effectively functioning system of checks and balances or even free opposition of political parties or factions within political parties. At the same time, the Index does not reflect the theoretical distinction between thin and thick demands, which is unfortunate.

With the rising thickness of RoL demands included in the Index, a lower consensus on the entire WJP conceptualization can be expected since almost all RoL scholars agree on thin demands but are divided on thicker ones.⁷³ Moreover, with greater thickness, the problem of normativity and relativism discussed above becomes more urgent.

Finally, thick conceptualization causes the Index to be based on several assumptions or even axioms (e.g., that all respondents identify themselves with fundamental liberal-democratic values and consequently agree with and trust those regal rules which correspond to these values, and also that all respondents, the general population included, dispose of sufficient knowledge and understanding of legal system in their country) which, as LC research suggests, should be empirically verified rather than only assumed.⁷⁴

Let us use an individual GPP item included in the WJP4E subfactor as an example. The item asks the respondents whether religious minorities in their country can freely and publicly observe their holy days and events. If the minorities are permitted to do that, the overall WJP rating is elevated, and if they are not, the rating decreases. The problem of this type of item is that we do not know whether the respondents know and understand the legal regulation of religious freedom or whether they identify with the statement that the free and public expression of religious belief is desirable, or what their opinions are concerning the current legal regulation of religious rights and their limits. Consequently, we only assume that the protection of rights associated with religion implies a higher trust in the law and thus increases the level of the RoL in the examined countries, which is not necessarily true, especially in religiously based (e.g. Muslim) countries. This problem can be solved by including more items whose aim is to verify the aforementioned assumptions using the LC objective (i.e., not self-reported) measurement tools (e.g., everyone should have the right to express their religion publicly (LC6), the constitutional law of the country protects religious freedoms nor does it allow them to be limited under any circumstances (LC1), or the current legal limits of public expression of religious belief should be adjusted (LC4)).

Solving all the aforementioned problems should eliminate overlaps and consequently prepare conceptualization of the RoL for operationalization and subsequent empirical measurement, and conversely, adapt the current WJP Index according to the needs and requirements of the RoL theory.

5.3 Effective Law Enforcement

A rather specific yet still crucial feedback for both the RoL theory and WJP Index stems from several WJP subfactors (WJP5A, WJP5B, WJP5C, WJP8A and WJP8C) which do not correspond to any RoL demand. The common feature of these subfactors is that they focus on effective law enforcement. Even though Bedner addresses this issue in his conceptualization, he decides without further explanation to omit the demands aimed at promoting security and order in the legal system.⁷⁵ Similarly, Møller and Skaaning, who provided an alternative conceptualization, also decided not to include a “result-oriented” dimension for the reason that it “does not have anything to do with the rules as such”.⁷⁶ Such an explanation is unsatisfactory since there is no reason why enforcement of the rules should have less to do with the rules than, for example, consent with the rules, which the authors included.

A good argument against including effective law enforcement into the RoL conceptualization is that it could collide with the demand of lawful exercise of public power.⁷⁷ The more effective an administration, police or army we want in ensuring order and security, the less stringent legal limitations we can use to regulate them in fulfilling their duties, which could violate the demand that “the discretion of crime-preventing agencies is not allowed to pervert the law”.⁷⁸ However, can we really argue that a state where legal rules are not effectively (or even at all) enforced by public institutions is ruled by law? Hence, we argue that the collision of lawfulness and effectivity should be overcome through a careful balance between them rather than through the omission of the latter from the conceptualization.

Even though the WJP Index includes several subfactors which measure these demands, it does not reflect the aforementioned debate which underlines their problematic nature, which leads to misunderstanding the security and order factor (WJP5). Although the authors claimed that the weight of this factor is “slightly less influential”⁷⁹ due to the insufficient internal consistency of the scale, they understood this factor as linear (i.e., a higher score means a higher RoL), which does not correspond to the theoretical warning that this factor may clash with other (and the most crucial) RoL demands, such as lawful exercise of power and elimination of arbitrariness.⁸⁰

Hence, we argue that a factor which measures effective law enforcement should emphasize the middle values rather than extremes, suggesting that one of the demands (i.e., effectiveness and lawfulness) is highly prioritized over the other. Its incorporation in the current form in the calculation of the general WJP Index is therefore wrong, and such results are most probably biased. An example of bias can be found in the United Arab Emirates, which scores on the lowest level in most subfactors, but in this factor, the country is one of the highest rated.

6 Conclusion

Combining all the discussed recommendations derived from the triangulation, an alternative conceptualization of the RoL suitable for both the theoretical and empirical perspectives can be introduced (see Table 4). The original elements found in the RoL and WJP conceptualizations have been accordingly restructured and classified into two categories of demands: formal (RoL1–RoL3) and substantive (RoL4–RoL5). Each of these demands was then divided into sub-demands which were classified with respect to their thickness. There is also a list of the original WJP subfactors and LC components matched to each of the sub-demands to suggest how our conceptualization could be operationalized.

Table 4 Suggested rule of law conceptualization

In creating this conceptualization, we implemented a moderate relativism perspective. Hence, our conceptualization of the RoL is designed specifically for liberal democracies. The precise classifications of demands and sub-demands, however, allows easy transformation of any of the perspectives from extreme universalism to extreme relativism. In all cases though, deeper inspection of the items used and their subsequent reformulation or even the creation of new sets of items is desirable.

An extreme universalism perspective would include only thin RoL1–RoL3 sub-demands. A moderate universalism perspective would require omitting all the thick sub-demands and objectivization of the thin RoL4 and RoL5 sub-demands. Using the LC terminology, the objectivised thin RoL4 sub-demand would include mechanisms to ensure a positive opinion about law and sufficient trust in laws in the particular society, and an objectivised thin RoL5 sub-demand would measure the level of identification of addressees with the values and principles of the particular legal system.

Finally, even though they should not be included in the moderate relativism perspective, our conceptualization also contains examples of the thick substantive sub-demands, which we included only as demonstration of how the extreme relativism perspective might look. Application of these sub-demands to any liberal democracy would, however, be problematic. Regarding the thick RoL4 sub-demands, the elements of legislative initiative, referendum, recall etc., for example, should be applied only in those countries where the society supports mechanisms of participatory democracy.⁸¹ Similarly, civil associations should be measured only in a pluralistic society, and trade unions should be measured in a neo-corporatist society.⁸² Concerning the thick RoL5 sub-demands, thickness here should be understood as an ordinal variable since every generation of human rights is “thicker” than the previous,⁸³ which means that a significant portion of citizens in a particular state does not necessarily need to identify itself with especially the latter generations of human rights. These latter generations should be applied rather carefully in certain countries. In summary, the thick RoL4 and RoL5 sub-demands must be customized to each particular country accordingly.

This article presented a triangulation of the three RoL related conceptualizations into a comprehensive form. Its main advantage is that it reflects the current theory of the RoL as well as incorporates operationalization, and it is therefore suitable for empirical measurement. As far as we know, such a complex approach to developing a new conceptualization has not been previously applied in this research field. Through the proposed conceptualization and operationalization, a clear and mutually beneficial dialogue between theory and empirical measurement can be established. These findings overcome all currently known conceptualizations of the RoL in terms of their ability to be empirically measured. Our conceptualization is therefore unique, and its contribution to scientific examination of the real levels of the RoL in various populations is far-reaching.