Abstract
While, in the UK, attitudes to the criminal justice system have been routinely investigated less effort has been made to measure attitudes to the civil justice system. However, globally, there are increasingly numerous examples of studies of attitudes to civil justice. Robust standardised scales are important to establishing change and difference between groups; for example, in the context of United Nations Sustainable Development Goal 16.3. Past focus has been on developing ‘composite indicators’. This paper sees the application of modern psychometric methods to the construction of ‘reflective’ instruments to measure single unidimensional attitude dimensions. Drawing on relevant theoretical frameworks, we administered an item pool of 35 attitude questions to a sample of 1061 adults across England and Wales. Principal component analysis was used to identify attitude domains, followed by Rasch analysis to construct scales with acceptable psychometric properties. An Inaccessibility of Justice (IOJ) scale and a Perceived Inequality of Justice (PIJ) scale resulted. Respondents who regarded themselves as having handled past civil legal problems poorly or having experienced unfair outcomes had higher IOJ and PIJ scores. Negative accounts of lawyers or courts from others were also associated with higher scores. Our substantive findings help explain how negative experiences of problem resolution can result in ‘frustrated resignation’ for later problems, and show the importance of positive experiences and accounts of the justice system in maintaining public support. The scales have broad justice policy utility and the methods provide a template for future related scale development.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
1 Introduction
1.1 Background
A 2012 UK review of literature on public attitudes to the justice system found that, while attitudes to the police and criminal justice system have been routinely investigated in a reasonably nuanced manner, research into attitudes to the civil justice system “is close to non-existent” (Wilson 2012: 6). As Moorhead et al. (2008: 2) earlier observed, “the evidence base is dominated by studies either wholly in the criminal sphere or which fail to disaggregate civil and criminal contexts”.
The conflation of civil and criminal justice is a matter of some significance. The philosophical underpinnings, procedural norms, evidential requirements and extent of civil and criminal justice are largely distinct,Footnote 1 and “every society sufficiently developed to have a formal legal system uses the criminal-civil distinction as an organising principle” (Robinson 1996: 202). Thus, conflation reduces clarity of the subject and purpose of questioning, along with the meaning and utility of responses. Moreover, given that, in some jurisdictions at least, people’s conception of “the criminal and civil justice systems are one and the same” (Greene 2016: 1263), and “if left to define ‘the justice system’ for themselves the public do so in terms of criminal justice” (Stratton and Lowe 2005, p. 5),Footnote 2 conflation may well result in responses skewed to perceptions of the criminal justice system.
Reflecting Moorhead’s observation, the British Social Attitudes Survey has asked numerous questions concerning attitudes towards different aspects of policing and criminal justice, a few about confidence and trust in ‘the courts’ or ‘legal system’ in general, but almost none specifically about attitudes to aspects of the civil justice system. This despite the fact that we live in a “law thick” world (Hadfield 2010:133) in which civil legal problemsFootnote 3 have repeatedly been found to be “ubiquitous” (Genn 1999: 249, Pleasence et al. 2013, 2018).
Notwithstanding the above, some UK surveys have included questions seeking to gauge attitudes to particular facets of the civil justice system; including, with some irony, the Crime Survey for England and Wales, which has included questions concerning confidence in family courts in relation to child protection matters (Summerfield and Freeman 2014). Also, 9 of 11 national civil justice focused ‘legal needs’ surveys conducted across the UK over the past 25 years have included a small number of civil justice contextualised attitude questions (Genn 1999; Genn and Paterson 2001; Pleasence et al. 2004a; Dignan 2006; Pleasence 2006; Pleasence et al. 2004b; Pleasence and Balmer 2013; Franklyn et al. 2017).
Conflation of civil and criminal justice has also been common in investigations into public attitudes towards justice systems elsewhere in the world. For example, large scale broad based surveys such as the World Values Survey, the European Social Survey and Eurobarometer have routinely asked about confidence or trust in ‘the legal system’ or ‘courts’, without further specification of the aspects of the legal system or nature of the courts in question.Footnote 4 Even dedicated surveys of justice or court systems, such as the many national and state surveys conducted in the United States, have often made no distinction between civil and criminal justice, or only rarely explicitly specified a civil context.Footnote 5
However, globally, there are increasingly numerous examples of surveys that have incorporated investigations into public attitudes towards civil justice systems; albeit not always entirely separately from the criminal justice system.Footnote 6 Most of the more than 50 national legal needs surveys undertaken over the past 25 years have explored attitudes towards formal civil justice systems, and civil justice more broadly, as have many of the larger number of sub-national surveys undertaken over the same period (Pleasence et al. 2018).Footnote 7 For example, the various iterations of HiiL’s Justice Needs and Satisfaction Surveys have asked about perceptions of the quality of and trust in justice institutions, expectations of fairness in specific civil legal contexts, as well as perceptions of the quality of civil dispute resolution procedures and outcomes personally experienced by respondents (Pleasence et al. 2018).Footnote 8 More broadly, many population survey based ‘access to justice assessments’, undertaken in a development (and legal empowermentFootnote 9) context, have incorporated (sometimes extensive) sets of questions to expressly gauge attitudes to different aspects of civil justice systems, to “capture perspectives of the people on the ground, particularly those who are disadvantaged, in order to help shape policy and programmes” (UNDP 2012:2).Footnote 10 Questions in such surveys have related to issues such as accessibility (e.g. geographical, financial, etc.) consistency, corruption, effectiveness, enforcement, equality, fairness, impartiality, respect, speed and trust (e.g. UNDP 2012).
Within the development context—in which recourse is often made to informal justice mechanisms—public trust in civil justice systems and the effective rule of law have been argued to be “a fundamental prerequisite for sustainable development” (Zoellick 2008: 75). This ‘economic development paradigm’ of justice reform sees effective civil justice systems to be not only a good in themselves, but also instrumental in realising “a range of … development goals: without justice, people cannot easily receive or access public goods and basic services, nor can they effectively access a range of markets” (Desai et al. 2012: 60). Moreover, a parallel ‘state-building paradigm’ of justice reform sees recognised and effective justice systems as “a sine qua non for peace” in fragile and conflict-affected states (Desai et al. 2012: 58). Thus, interest in assessing public attitudes towards justice institutions has also been a feature of rule of law and access to justice research in many Latin American countries with (relatively) recent experience of internal conflict, and where “during the transition from military and authoritarian to democratic regimes since the early nineties, the judiciary has been a central part of development and consolidation of the democratic process” (Cunha et al. 2014: 448).
Most extensively, civil justice features as the seventh of eight factors that comprise the World Justice Project’s Rule of Law Index. The civil justice factor, while mostly compiled from the qualified respondent questionnaires (sent to experts in each of the countries surveyed), also draws on three questions from the General Population Poll, which from 2016 has also incorporated a dispute resolution module (Adams et al. 2017). And, of course, there are also various, contrasting, examples of research into particular attitudes to particular aspects of particular civil justice systems in a range of jurisdictions; such as Quintanilla and Avtgis’s (2017) study of attitudes to pre-dispute binding arbitration clauses in the USA.
Some investigations of attitudes to civil justice have involved the development of standardised scales. For example, the World Justice Project’s Rule of Law Index represents a broad-based scale (and sub-scales) that incorporates such attitudes, as does Cunha et al.’s (2014) Brazilian Justice Confidence Index. There are also examples of much more narrowly focused scale development initiatives, such as Hans and Lofquist’s (1994) Litigation Crisis Scale—used to explore the causes and correlates of perceptions of the United States civil justice system being ‘in crisis’.
The rigorous development and evaluation of scales is important if change and difference between groups is of interest. It is also important to theoretical development in relation to such attitudes and, indirectly, impinges on related areas of research, such as into—to give two examples with significant bearing on contemporary access to justice policy—drivers of legal problem resolving behaviour or public perceptions relating to United Nations Sustainable Development Goals (SDGs) concerning the rule of law and access to justice (SDG 16.3) and the effectiveness, accountability and transparency of government institutions (SDG 16.6).Footnote 11
In the case of the first of these, the psychological mechanism through which people become inactive in the face of legal problems, through processes such as that described by Sandefur (2007, p. 125) as “frustrated resignation”—born of repeated “past experiences of frustration or failure in trying to resolve similar problems, even though the particular other party or parties involved had changed”—could be explored further through measures that could identify difference and change in attitudes to, say, the accessibility or fairness of dispute resolution processes. And, while there is some quantitative evidence that “people have a greater tendency to feel that the law and the justice system are essentially unfair if they had experienced justiciable problems, if they had experienced unfavourable outcomes to those problems and if they have had difficulty obtaining satisfactory assistance” (Currie 2009: 100),Footnote 12 opportunities for detailed and robust investigation to quantify and identify correlates of negative perceptions of different aspects of the civil justice system have been narrowed by the unavailability of appropriate research tools.
In the case of United Nations SDG 16.3, it is notable that, at present, the current global indicators framework includes two criminal justice, but no civil justice, related indicators. In part, this is down to limited availability of data and resources. The broad scope of civil law—touching many aspects of everyday life—along with the fact that only a minority of civil legal problems enter the formal justice system [the mean percentage of civil legal problems that saw court involvement, as reported by the World Justice Project’s 2016 General Population Poll, was only slightly above 10% (Adams et al. 2017)],Footnote 13 means administrative data cannot provide a comprehensive picture of access to civil justice. Moreover, objective survey-based measures of access to civil justice are complex and relatively expensive to implement. Thus, relatively simple and cheap to implement survey-based measures of people’s perceptions of access to justice could be of particular interest in the SDG 16.3 context. Although, clearly many challenges present themselves in this regard. For example, people’s attitudes and wider perceptions are influenced “by a range of factors that may be unrelated to the performance of institutions” (Parsons 2011: 178), such as expectations and other social, cultural and historical factors. As Masterson and Tong (2015: 291) have observed,
“There is clear evidence that justice perceptions generally do not develop in a vacuum, but instead rely on social information and the broader social context. Whether the social information stems from other individuals such as peers or supervisors, or from a larger setting such as one’s work team or organisation, studies have shown that people’s individual justice perceptions are formed and influenced by one’s social setting”.
Moreover, access to justice can extend well beyond formal and state recognised justice systems, and measures in many, particularly developing, countries—along with global measures—must recognise and accommodate this. As the UNDP (2009: 8) have observed, “informal justice can form a key part of individuals’ and communities’ experience of justice and the rule of law, with over 80% of disputes resolved through informal justice mechanisms in some countries”. Furthermore, there are differences in designs and underlying goals of different justice systems around the world, which must also be recognised and accommodated in measurement design when appropriate.Footnote 14
1.2 Attitudes to the Civil Justice System
Attitudes can be defined as “enduring pattern[s] of evaluative responses towards a person, object, or issue” (Colman 2015: 62). Attitudes are of interest not just in themselves, but also because they “provide a predisposition to behave in a particular way” (Coaley 2014: 231). Thus, an understanding of attitudes to the civil justice system is, as suggested above, an important element of understanding civil justice related behaviour.
Attitudes are multidimensional (i.e. they are made up of several domains). At the broadest level, they are commonly regarded as having three components; the affective, behavioural and cognitive (the ‘ABC’ components) (Rosenberg and Hovland 1960).Footnote 15 The affective component refers to emotions or feelings. The behavioural component refers to behavioural intent. The cognitive component refers to beliefs. In the context of the civil justice system, these components can be distinguished in relation to, for instance, feelings, behavioural intent and beliefs concerning lawyers or civil courts, or other aspects of formal civil justice systems, which represent the scope of the present paper.Footnote 16
Further dimensions of attitudes to civil justice systems have also been suggested. For example, Wilson’s (2012: 7) review identified “a range of types of attitude” based on different aspects of evaluation. Those she detailed referenced the “similar, but different” (Moorhead et al. 2008: 4) concepts of satisfaction, confidence, trust, support and legitimacy. Linked to these are also concepts such as interest in (or engagement with) the justice system, that are discussed within constitutional and access to justice debates, and which are also sometimes put forward as dimensions of attitude (e.g. Coaley 2014).
Klaming and Giesen’s (2008) and Verdonschot et al.’s (2008) extensive reviews of conceptualisations of the quality of justice procedures and outcomes, for the Hague Institute for Innovation of Law’s (HiiL) Measuring Access to Justice in a Globalising World project (Barendrecht et al. 2010),Footnote 17 also listed a range of justice dimensions which can be the subject, and therefore further dimensions, of attitudes; namely, as regards process, procedural, interpersonal and informational justice and, as regards outcomes, distributive and restorative justice, along with outcome functionality and transparency.
Procedural justice refers to the various properties that a procedure should possess “in order to be perceived as fair by its user” (Klaming and Giesen 2008: 3), including “voice, neutrality, trustworthiness, consistency, and accuracy” (Gramatikov et al. 2011: 361). Interpersonal justice “reflects the degree to which people are treated with politeness, dignity, and respect by authorities and third parties involved in executing procedures or determining outcomes” (Colquitt et al. 2001:427). Informational justice is concerned with “explanations provided to people that convey information about why procedures were used in a certain way or why outcomes were distributed in a certain fashion” (Colquitt et al. 2001: 427). Distributive justice concerns the fair distribution of benefits and burdens (in this context, system outcomes),Footnote 18 while restorative justice “is the dimension of the outcome which rectifies … damage or loss suffered” (Gramatikov et al. 2011: 363). Functionality of outcome “is the extent to which the outcome solves the problem”, and transparency of outcome concerns explanations for outcomes and the ability to compare the outcomes of similar problems (Gramatikov et al. 2011: 363).
Within these justice dimensions are apparent those that Wilson refers to as ‘neutrality’ (in terms of procedures and decision making), ‘treatment’ (in terms of respect), ‘having your say’ (i.e. the opportunity for parties to state their case) and ‘motive-based trust’ (i.e. the perceived motives of legal actors), associated with Tyler and colleagues’ influential theorising around procedural justice, legitimacy and compliance in the criminal sphere (e.g. Tyler 1990; Tyler and Huo 2002; Tyler et al. 2007).
Then there are dimensions of justice specific to different areas of the justice system (such as areas of law, forms of process, types of court, kinds of legal actor, etc.), which may extend beyond those just described, and which again can also be considered as dimensions of attitudes. So, for example, Moorhead et al. (2008) noted seven dimensions to measures of confidence in the criminal justice system within the British Crime SurveyFootnote 19; namely to: bring people who commit crimes to justice; meet the needs of victims of crime; respect the rights of people accused of committing a crime and treat them fairly; deal with cases promptly and efficiently; treat witnesses well; reduce crime; and, deal with young people accused of crime.
Broader socio-legal theory and research also points to a variety of further potential aspects of attitudes to civil justice.
For example, Ewick and Silbey’s (1998: 28) research in the field of legal consciousness suggests that, in the United States, “people tell three stories” about the law; as being ‘majestic’ (“operating by known and fixed rules in carefully delimited spheres”), ‘a game’ (“a terrain for tactical encounters through which people marshal a variety of social resources to achieve strategic goals”) or ‘arbitrary’ (“rather than objective, legality is understood to be arbitrary and capricious”). Each of these stories provides dimensions for attitudinal measurement.
Galanter’s (1974, 1999) theorising around the structural imbalance between the resources and prospects of ‘haves’ and ‘have-nots’ (and, closely linked to this, ‘one-shotters’ and ‘repeat players’) also highlights potential subjects and dimensions of attitude measurement; both in relation to the broad theme of the prospects of differentially resourced litigants and in relation to more narrowly formulated hypotheses—such as that one-shotter lawyers “tend to make up the ‘lower echelons’ of the legal profession” (Galanter 1974: 116), that the law is skewed towards repeat players, that lawyers improve case outcomes, and that the closer the ongoing relationship between parties to a dispute, the more likely “the relationship will be regulated by some independent ‘private’ system” (Galanter 1974: 130). Likewise, the newer literature focusing on the disadvantage of underprivileged repeat players does the same (e.g. Sandefur 2007; Green 2016).
Theorising around legal pluralism—or “the idea that in any one geographical space defined by the conventional boundaries of a nation state, there is more than one law or legal system … [or] normative orders or customs” (Davies 2010: 805/820)—raises issues around both commitment to competing normative orders (again, pertinent to attitudes to law) and the complex nature of attitudes in the many countries in which “legal pluralism is common” (Barendrecht 2011: 281).
In addition, policy debates concerning access to justice suggest further civil justice attitude dimensions. For example, the Bach Commission on Access to Justice’s “six key problems within the justice system” provide potential subjects for attitude measurement (Bach Commission on Access to Justice 2016: 8). Indeed, attitude to the cost of accessing the justice system—which features prominently within these problems—has been a routine subject of investigation in recent surveys (e.g. Franklyn et al. 2017).
In summary, the literature suggests a broad array of potential domains of access to justice; some theoretically well-developed, some supported by empirical investigation, some more conjectural.
1.3 Standardised Scales of Attitudes to the Civil Justice System
While, as was detailed in Sect. 1.1, examples exist of standardised scales of, or incorporating, attitudes towards aspects of civil justice systems, it is notable that they have tended to be composite indicators (i.e. the compilation of multiple individual indicators “into a single index” and measuring “multidimensional concepts which cannot be captured by a single indicator” (OECD 2008: 13). A significant body of work has examined the construction of composite indicators in legal (and related) contexts (e.g. Maggino 2017; Saisana et al. 2005; Saisana and Saltelli 2014) and detailed guidance on the construction of composite indicators has been published by the OECD (2008). However, less attention has been paid to the psychometric development of standardised measures of single unidimensional attitude dimensions.
A range of terminology has been used to distinguish between these two approaches, though perhaps the simplest is Stenner et al.’s (2009) distinction between index models (formative or composite variable models where the causal action flows from the indicators to the composite variable) and measurement models (reflective or latent variable models where the causal action flows from the latent variable to the indicators). Costa (2015) further formalises the distinction in quality of life research, between composite (or sometimes formative, where the construct is an exact linear combination of the indicators), causal (where the construct is an effect of the indicators) and reflective (manifestations of some underlying construct) approaches. Costa (2015: 2057) noted that “psychometric methods based on inter-item associations are only appropriate for reflective indicators, whereas other statistical and non-statistical validation methods can be used for composite or causal indicators”.Footnote 20
This study represents an early attempt to apply modern psychometric methods to the construction of reflective instruments in a legal context (and, specifically, in the context of attitudes to formal civil justice systems), utilising Rasch analysis, as described in the next section. As well as providing research tools in an area in which, as Stratton and Lowe (2005:3) have argued, many surveys “fail to meet most of the technical assumptions of polling”—often not defining their focus, or clearly explaining it to respondents—it is hoped that the methods set out in this paper will provide a template for the further development of similar and related standardised scales.
Moreover, as the OECD (2008: 18) have noted, modern psychometric methods can make a contribution to composite indicators, with a need further to explore “the relationship between the practice of composite indicators and the traditional measurement theory developed in psychometrics”. A number of authors, such as Gnaldi et al. (2017) have addressed this relationship through multidimensional extensions of Rasch/IRT approaches.
2 This Study
In this paper, we set out details of an application of modern psychometric methods to the construction of reflective instruments to measure attitudes to the civil justice system, principally within a UK context, but potentially of application in similar jurisdictions. The study has drawn on the principles and methods that have evolved in other disciplines (e.g. DeVellis 2012) and the latest developments in psychometric modelling (e.g. Hobart and Cano 2009; Christensen et al. 2013; Boone et al. 2014; Bond and Fox 2015). The study sought to first identify attitude dimensions, or domains, and then construct scales for measuring attitudes within these domains—with acceptable psychometric properties—utilising data from survey questions associated with individual domains. Finally, it sought to investigate the relationship between attitudes to civil justice and people’s experience of civil legal problems, dispute resolution, and social discourse on the same.
2.1 Methods
The development of measures, or scale development typically involves a number of set steps, as set out in DeVellis (2012).Footnote 21 These include determination of what is to be to measured, generation of an ‘item pool’ (in this case, a list of statement-based attitude questions), administration of items to a development sample (in this case, via a general population probability survey conducted exclusively for the purposes of scale development), evaluation of item responses and optimisation of scale contents. The final two steps in this process typically involve either classical test theory (CTT) or modern psychometric methods (in this case, Rasch analysis).
2.1.1 Attitude Item Pool
As indicated above, the literature suggests manifold potential domains of attitude to the civil justice system; but, with limited space available within a survey questionnaire to devote to an attitude item pool, we were not able to explore each of these individually or in detail. So, instead, we sought to construct an item pool containing a broad and coherent range of aspects of attitudes, that might provide some indication of multidimensionality and domains, if there are distinct domains that are reflected in attitudes.
To inform the construction of the item pool, we collated relevant attitude questions from earlier surveys.Footnote 22 There was little commentary on the origins or design of these earlier questions; a reflection of both the general absence of theoretical frameworks to guide formulation and of empirical testing. In many cases, questions appear to have simply reflected the prevailing interests of policy stakeholders; although in some instances questions were more considered, and some had empirical foundations. For example, the four attitude questions developed by Genn (1999: 225) “were largely drawn from comments made by members of the public during focus group discussions at the developmental stage of the research”.
In the end, questions from a variety of surveys were adapted for inclusion in the item poolFootnote 23; along with additional questions drafted to reflect key aspects of the dimensions outlines above. The text of the introduction to the item pool was informed by the framing of the family justice section of the Crime Survey for England and Wales; although it was broader in its scope:
Questionnaire introduction
Thank you for participating in this survey of people’s impressions and experience of justice and the justice system in England and Wales.
The survey is not about crime or criminal justice. Instead it is about the many other aspects of law and justice that touch people’s lives.
For example, people can face problems concerning employment, housing, children, money or injuries that have a legal dimension. When they do, they may ask lawyers (or other experts) to help sort such problems out. And if problems cannot be resolved between parties, then courts or tribunals (a less formal type of court) may also help achieve a resolution.
It is these types of problems and this system of experts and courts/tribunals that this survey concerns.
Some of the questions may not seem relevant to you. Do not worry. This is not a test. Just try to respond honestly and accurately.
All responses you provide are confidential.
Item pool introduction
As a reminder, we are not concerned with the criminal justice system. We are concerned with the justice system that deals with issues such as being unreasonably sacked by your employer, injured as a result of someone else’s negligence, involved in a dispute over money as part of a divorce, or facing eviction from your home
In total, the item pool comprised 35 questions, concerning law, courts/tribunals and lawyers. To support domain identification, they were focused on eight distinct theoretical dimensions of attitudes prominent within the literature and past surveys: outcome fairness; neutrality; manipulability; respect; voice; motivation; access/efficiency; sovereignty. All 35 items are set out in Table 1.
2.1.2 General Population Survey
The survey used to collect item pool data utilised a one-stage sample design, in which a stratified, but unclustered, sample of addresses was drawn from the Residential Postcode Address File; the cornerstone of national probability samples in England and Wales. The survey was an innovative hybrid form of postal and online survey, based on the Community Life Survey web experiment.Footnote 24 The findings of the Community Life Survey web experiment suggest that the quality of data obtained from hybrid postal and online surveys can generally be expected to be similar to that which would have been obtained from face-to-face surveys.Footnote 25 Such hybrid surveys are relatively cheap to administer, but can maintain sampling rigour; unlike many pure online surveys.
Advance letters (and reminder letters and postcards) invited “the person aged 16 or over who has the next birthday” in a household to either take part in the survey online (using a provided web-link) or return a postal version of the questionnaire. The person completing the survey was offered a £10 shopping voucher as an incentive. In total, there were 1061 respondents, of whom 872 completed all the sections of the questionnaire.Footnote 26
In addition to attitude items, the survey also included a range of questions to explore legal confidence and capability, identify past interactions with and discussions about law, legal processes and legal services, and collect socio-demographic data.Footnote 27
2.2 Analysis
Scale development data analysis was conducted in two stages. First, principal components analysis (PCA) was used to explore the dimensionality of item responses and identify domains. Second, for those unidimensional domains that were identified, and associated with a suitable number of items, Rasch analysis was used to attempt to develop and refine scales with acceptable psychometric properties. Our use of PCA and Rasch analysis corresponds to Christensen’s guidance (Christensen et al. 2012). More generally, our approach to scale development broadly followed the guidance set out by DeVellis (2012).
For the two scales that this study yielded, a third stage of analysis was also undertaken. Three normal linear models were fitted to model scale score on the basis of recalled accounts of lawyers, courts and tribunals,Footnote 28 past experience (over the past 5 years) of civil legal problems, lawyersFootnote 29 or courts or tribunals.Footnote 30
2.2.1 Principal Components Analysis (PCA)
The aim of PCA was to establish whether responses to the attitude questions were multidimensional (i.e. were made up of more than one domains) and, if so, examine domains identified from the data. PCA allows identification of relatively independent subsets of items (Tabachnick and Fidell 2013).Footnote 31 Subsets in which items are correlated with one another, but largely independent of those in other subsets, are combined into components, with the components thought to reflect underlying processes that created the correlations; in our case, attitude domains (Tabachnick and Fidell 2013). New components are derived in decreasing order of their importance. The first component accounts for as much variation in the original data as possible. The second component accounts for as much as possible of remaining variation; and so on (Everitt and Dunn 2013).
PCA was conducted using SPSS. Prior to analysis, data was confirmed to be suitable for PCA.Footnote 32
Analysis utilised principal components extraction, with oblique (direct oblimin) rotation.Footnote 33 The number of components retained was determined using examination of the scree plot (Cattell 1966), as well as parallel analysis (Horn 1965) using software developed by Watkins (2000).Footnote 34 These methods were used in favour of the commonly used Kaiser’s criterion (retaining components with eigenvalues above one) due to its inaccuracy (Velicer and Jackson 1990) and tendency to retain too many components (Costello and Osborne 2005).
2.2.2 Rasch Analysis Footnote 35
When solid components were identified using PCA, Rasch analysis was used to attempt to develop and refine scales; along with scoring protocols. Rasch analysis is a modern psychometric method widely used for scale development; and perhaps the only method specifically developed for this purpose (Smith et al. 2016).Footnote 36
The basic assumptions of the Rasch model are that: (a) each person is characterised by an ability, and (b) each item by a difficulty that (c) can be expressed by numbers along one line. Finally, (d) from the difference between the numbers (and nothing else), the probability of observing any particular scored response can be computed (Bond and Fox 2015). Equation (1) shows the basic Rasch model.Footnote 37
where pni=the probability of affirming (i.e. giving a positive response), for item i and person n, Di=the difficulty of item i, Bn=the ability of person n
Rasch analysis (using a partial credit modelFootnote 38) was implemented using RUMM2030,Footnote 39 which allows assessment of a scale and its items against the Rasch model, as well as a detailed examination of scale functioning.
A number of measures of model fit were considered. Overall fit was assessed using an item-trait interaction statistic. This is reported in RUMM2030 as a Chi squared statistic and should be non-significant (following a Bonferonni correction for the number of items in the scale).Footnote 40 Two item-person interaction statistics were also considered (for items and persons). In each case, fit is represented by a z-score, where perfect fit would have a mean of zero and standard deviation of one. In practice, a fit residual value of 1.5 or less (for items and persons) was considered to indicate acceptable fit. Particularly in cases with misfitting items or persons, individual item and person fit was also explored using residuals and a Chi squared statistic (and Bonferonni adjusted p value). Items of concern were indicated by fit residuals below − 2.5 or above 2.5. Values below − 2.5 typically indicate overfit or redundancy, with the item a possible candidate for removal on this basis (i.e. it is already captured by other items). Values above 2.5 indicate a misfitting item. There could be a number of reasons for this, and a range of additional diagnostics allow further assessment, and possible correction of a misfitting item (see below). If an individual item fit cannot be improved, the item may need to be removed. As with items, misfitting persons were identified by fit residual values above 2.5.Footnote 41
Internal consistency of a scale was assessed using the Person Separation Index (PSI), which gives a measure of a scale’s ability to discriminate between individuals with varying levels of the trait. Although acceptable levels may vary depending on the scale and its use, values exceeding 0.7 indicate acceptable internal consistency. Low values may indicate the need for additional items. Discrimination was also assessed graphically by examining the Item Characteristic Curve (ICC) for each item, with poor discrimination a common contributor to item misfit. The ICC allows examination of the extent to which an item deviates from the model; for example, by underestimating scores for low confidence groups or overestimating scores for high confidence groups.Footnote 42 Remedial action may involve removing items with poor discrimination.
Item responses took the form of a four-point Likert scale. Suitability of the response format was checked by examination of the threshold map and category probability curves for individual items, which illustrates category structure. When individuals respond in a manner consistent with their level of the trait, thresholds should be ordered. Disordered thresholds can occur, for example, where there are too many response options or respondents struggle to differentiate between options. Remedial action may include rescoring response categories (i.e. to fewer categories).
Response dependence (or local dependence) was also tested as part of the Rasch analysis. Local independence is a requirement of the Rasch model and means that having extracted the Rasch factor (i.e. the attitude scale) there should be no leftover patterns in the residuals. Local dependence is an issue where response to one item depends on response to another item. A commonly cited example is that of numerous walking items in the same scale, where, if a person can walk a mile without difficulty, they must necessarily be able to walk a lesser distance without difficulty (Tennant and Conaghan 2007). Local dependence can inflate seeming reliability (indicated by an artificially high PSI) and affect parameter estimates in Rasch analysis (Tennant and Conaghan 2007). In practice, values over 0.2 in the residual correlation matrix indicate potential local dependence and redundancy (Marais & Andrich 2008). In such an instance, the first step is to carefully check the wording of the items. Sub-test analysis and re-running Rasch analysis having combined locally dependent items are means to assess the extent to which reliability has been inflated. Remedial action may involve the rewording or removal of items.
Rasch analysis requires that items form a unidimensional scale. While dimensionality was initially tested using PCA, it was also further explored as part of the Rasch analysis, using the procedure set out by Smith (2002). Following PCA of the residuals, correlations between items and the first residual components were used to define two subsets of items. An independent t-test was then used to test the difference in person estimates between the two subsets, with a non-significant result indicating no evidence of multidimensionality (i.e. indicating that the items measure a single attitude domain or trait).
Differential item functioning (DIF), which can have an impact on model fit, occurs when particular groups (e.g. men and women, younger and older respondents) perform differently on an item despite having comparable levels of the trait being measured. Graphically, DIF can be explored by superimposing groups of interest (e.g. men and women) on item characteristic curves to assess whether or not they perform differently. DIF is also assessed statistically using Analysis of Variance, to compare scores for each level of the person factor (men, women) and across levels of the trait (class intervals). There are two distinct forms of DIF (Teresi et al. 2000); Uniform DIF, where consistent systematic difference is observed in the groups’ responses to an item, would be indicated by a significant person factor. Remedial action here might involve separate calibration of the item for each group, though this has the disadvantage of adding complexity to the scoring of a scale. Non-uniform DIF, where differences vary across levels of the trait, would be indicated by class interval interaction. In this case, it is likely that the item would need to be removed.
Scale targeting was assessed graphically through examination of the person-item distribution, which illustrates individuals’ scores and item placement on the underlying trait. In a well targeted scale, items would span the full range of individual scores. This indicates that a scale is not too easy (e.g. attitude items that nearly all respondents would be positive about) and not too hard (e.g. attitude items that nearly all respondents would be negative about). Given that the scale is centred on zero logits, targeting can also be examined by how close the mean location value for persons is to zero. A poorly targeted scale is likely to indicate a need for additional easier or harder items to fully span individuals’ scores or replacement of items with similar difficulty to give a broader spread of difficulty.
Once a scale has been developed, satisfying the various requirements/diagnostics set out above, it can be used to produce a score. This requires providing guidance on how to calculate raw scores from responses, and a conversion table to change raw scores into Rasch converted scores. As Rasch location values have been converted from an ordinal to an interval scale (Wright and Linacre 1989), they are appropriate for a wider range of common statistical analyses. For ease of use, Rasch location values are also commonly converted to alternative ranges (e.g. 0–100).
Alternatively, scale scores can be converted into strata (Fisher 1992; Linacre 2013; Wright and Masters 2002), with the number of strata possible dependent upon internal consistency.Footnote 43
2.2.3 Normal Linear Models
Three normal linear models were fitted for each of the two scales that this study yielded. The first modelled scale score on the basis of recalled accounts of lawyers, courts and tribunals, whether civil legal problems had been experienced, lawyer use (in the past 5 years) and contact with courts or tribunals (in the past 5 years). The second replaced legal problem experience with a variable integrating self-reported adequacy of problem handling,Footnote 44 replaced lawyer use with a variable integrating satisfactionFootnote 45 and replaced contact with courts or tribunals with a variable integrating process fairness.Footnote 46 The third replaced self-reported adequacy of handling with perceived fairness of outcome within the legal problem experience hybrid variable.Footnote 47 In exploring relationships between scales and problem experience/interaction with the law, analysis also aims to explore construct validity.Footnote 48
3 Results
3.1 Responses to Items
Table 1 sets out responses to the 35 attitude items.Footnote 49
3.2 Attitude Domains
To identify attitude domains, the 35 attitude items were analysed using PCA. A significant Bartlett’s test indicated items were related (\(\chi_{595}^{2}\) = 3931.23, p < 0.001). The majority of items had diagonal anti-image correlation matrix values above 0.7, further confirming suitability for PCA.Footnote 50 The correlation matrix also showed that all items, with the exception of items 13, 15, 27 and 30–33 had a correlation above 0.3 with at least one other item; again, confirming the factorability of the dataset/correlation matrix and its suitability for PCA.
PCA indicated that ten components had eigenvalues over one (Kaiser’s criterion). However, examination of the scree plot, while not decisive, indicated a slight drop-off after component six, and parallel analysis (Horn 1965) suggested retention of six components. As a result, six components were retained, explaining 47.6% of the variance. Direct oblimin rotation of the six-component solution yielded the pattern matrix shown in Table 2. Rotation showed a comparatively clear pattern, with the majority of items loading reasonably strongly on a single component, though there were ten items with loadings above 0.3 on two components. Correlation among components varied from 0.014 (between component 1 and 5) to 0.36 (between component 1 and 3),Footnote 51 with clear multidimensionality and no case for combining components into a single score.
On the basis of its content, component one was labelled ‘trust’. The second component was labelled ‘access’. The third component was labelled ‘equality’ and linked to the theoretical domains of outcome fairness and neutrality. The fourth component was labelled ‘fairness’. The sixth label was labelled ‘respect’. There were not enough coherent strongly loading items to label the fifth component. The majority of items also had reasonable loadings, with a minority falling below commonly used cut-offs.Footnote 52 The ‘access’ component was the most coherent. Additional items would need to be tested to further explore other domains and sub-domains.
3.3 Development of Attitude Scales
While five coherent components were extracted using PCA, this did not mean that five scales could be developed using Rasch analysis. Initial Rasch analysis of the ‘trust’, ‘respect’ and ‘fairness’ components indicated that there were too few items to yield scales with acceptable psychometric properties. Specifically, in each case the person separation index fell below commonly used values indicating acceptable ability to discriminate between individuals with differing levels of each trait.Footnote 53 Larger item pools would have to be constructed and tested to develop functioning scales for these components. Nonetheless, of the components identified by PCA, two were suitable for scale development using Rasch analysis: ‘access’ and ‘equality’. Additional items would be required to develop ‘trust’, ‘respect’ and ‘fairness’ components into well-functioning rating scales.
3.3.1 An ‘Inaccessibility of Justice’ Scale
Rasch analysis of access items (the ten items with loadings above 0.3/below − 0.3 in component 2, Table 2) was restricted to those answering eight or more of the ten access items (446 of 535, 83.4%). The initial model (of ten items) was close to acceptable fit, with an item fit residual of 1.42 and person fit residual of 1.51, but with a significant item-trait interaction (indicating some deviation from the modelFootnote 54). The initial PSI was 0.77. There was also some indication of disordered thresholds for item 8 and overfit/redundancy for a further item (item 21). Once response categories had been recoded for item 8 and the redundant item removed, all measures of fit were acceptable, with an item fit residual of 1.10, person fit residual of 1.40 and non-significant item-trait interaction.Footnote 55 There was no evidence of local dependence, multidimensionality or DIF on the basis of gender, age, legal problem experience or prior use of lawyers or courts, while the scale appeared to be reasonably well targeted (though a number of thresholds tended towards the lower end of the scale—Fig. 1).
The final nine-items yielded a scale with acceptable psychometric properties that can be used as a measure of the inaccessibility of justice.Footnote 56 The Inaccessibility of Justice (IOJ) scale is as follows (renumbering items from 1 to 9):
Now, some questions about your general impression and experience of the justice system.
We are not concerned with the criminal justice system. We are concerned with the justice system that deals with issues such as being unreasonably sacked by your employer, injured as a result of someone else’s negligence, involved in a dispute over money as part of a divorce, or facing eviction from your home.
Thinking about issues like this, to what extent do you agree or disagree with the following statements. (strongly agree; mainly agree; mainly disagree; strongly disagree)
- Q1:
-
Issues like these are usually resolved promptly and efficiently
- Q2:
-
People with less money generally get a worse outcome
- Q3:
-
For issues like these, law is like a game in which the skilful and resourceful are more likely to get what they want
- Q4:
-
It is easy to take issues like these to court if needed
- Q5:
-
For issues like these, lawyers are too expensive for most people to use
- Q6:
-
The justice system provides good value for money
- Q7:
-
For issues like these, people like me can afford help from a lawyer
- Q8:
-
Rich people’s lawyers are no better than poor people’s lawyers
- Q9:
-
Taking a case to court is generally more trouble than it is worth
After administration, responses are scored to yield, first, a ‘raw’ score, then a Rasch converted ‘IOJ score’. To calculate the raw score, for items 1, 4, 6, 7 and 8, strongly agree should be assigned a score of ‘0’, mainly agree ‘1’, mainly disagree ‘2’ and strongly disagree ‘3’. For items 2, 3 and 9, strongly agree should be assigned a score of ‘3’, mainly agree ‘2’, mainly disagree ‘1’ and strongly disagree ‘0’. For item 5, strongly agree should be assigned a score of ‘2’, mainly agree ‘1’, mainly disagree ‘0’ and strongly disagree also ‘0’. Across the nine items this will yield individual scores of between 0 and 26. These scores can be converted into IOJ scores (ranging from 0 to 100) using Table 3. Higher scores are associated with greater perceived inaccessibility of justice.
The mean IOJ score among 357 survey respondents was 58.1 (standard deviation = 11.6). The median score was 45.2 (25th percentile = 32.8, 75th percentile = 60.3). Converting scores into strata (low, medium and high scores)Footnote 57 gave 26.1% with low scores (scores of 0–50), 54.1% with medium scores (scores of 51–66) and 19.9% with high scores (scores of 67–100). While based on relatively small numbers, these scores and strata provide initial baseline general population estimates for comparison.
The output of the three normal linear models of IOJ score is set out in Table 4. While there was no significant association between IOJ score and the simple experience of civil legal problems, once self-reported adequacy of problem handling and perceptions of outcome fairness were taken into account, the association became significant. Compared to those who had experienced no problems, respondents who handled one or more problems poorly were associated with a 9.5 point increase in IOJ score.Footnote 58 Similarly, compared to those who had experienced no problems, those who perceived one or more problems as having an unfair outcome were associated with a 6.4 point increase in IOJ score.Footnote 59 Accounts respondents had heard about lawyers, courts and tribunals from friends, family or colleagues were also all significantly associated with IOJ score. In all three cases there was a significant difference between those recalling positive accounts and those recalling negative accounts of 6.0 points for lawyers,Footnote 60 5.4 points for courtsFootnote 61 and 4.9 points for tribunals.Footnote 62
3.3.2 A ‘Perceived Inequality of Justice’ Scale
Rasch analysis of equality items (component 3, Table 2) was restricted to those answering seven or more of the eight items (716 of 908 respondents, 78.9%). Inclusion of all eight items, gave an item fit just over acceptable limits (item fit residual of 1.54), an acceptable person fit (person fit residual of 1.39), but resulted in some deviation from the overall model.Footnote 63 The PSI was 0.76. There were no disordered thresholds, no local dependence, no evidence of multidimensionality and no misfitting individual items. However, there was strong evidence of uniform differential item functioning for item 34 (“For issues like these, if a man and a woman make identical claims in court, the man will be more likely to obtain a favourable result”) on the basis of gender (shown in Fig. 2). This suggested that male and female respondents interpreted the item (and its ‘difficulty’) differently.
In the case of uniform DIF, remedial action can involve separate calibration of the item for each group (in this case for men and women); although this can introduce unwanted complexity into scale scoring and implementation. More generally, this case highlighted issues associated with items that refer to a specific demographic characteristic, such as gender, ethnicity or religion (rather than no background characteristics or general/multiple characteristics). Item 12 (“If a white person and a black person make identical claims in court, the white person will be more likely to obtain a favourable result”) might be expected to lead to similar issue were DIF tested on the basis of ethnicity. As a consequence, items 35 and 12 were both removed, leaving six equality items all of which referred to multiple characteristics or made no mention of background characteristics.
The six remaining items gave acceptable item (item fit residual of 1.40), person fit (person fit residual of 1.31) and overall fit.Footnote 64 The PSI of 0.74 was relatively low, but above the common cut-off of 0.7 and acceptable for group use. There was no evidence of DIF on the basis of gender, age, problem experience or prior lawyer/court use, evidence of multidimensionality or local dependence. All thresholds were ordered and the scale was reasonably well targeted (Fig. 3).
The remaining six-items yielded a scale with acceptable psychometric properties that can be used as a measure of perceived inequality of justice. The Perceived Inequality of Justice (PIJ) scale is as follows (renumbering the items from 1 to 6):
Now, some questions about your general impression and experience of the justice system.
We are not concerned with the ‘criminal’ justice system. We are concerned with the justice system that deals with issues such as being unreasonably sacked by your employer, injured as a result of someone else’s negligence, involved in a dispute over money as part of a divorce, or facing eviction from your home.
Thinking about issues like this, to what extent do you agree or disagree with the following statements. (strongly agree; mainly agree; mainly disagree; strongly disagree)
- Q1:
-
People with less money generally get a worse outcome
- Q2:
-
For issues like these, law is like a game in which the skilful and resourceful are more likely to get what they want
- Q3:
-
The law always treat both parties fairly, whatever their background, gender, ethnicity or faith
- Q4:
-
Judges have their own agendas separate from the law
- Q5:
-
The decisions and actions of courts are influenced by pressure from the press and politicians
- Q6:
-
Courts and tribunals always treat both parties fairly, whatever their background, gender, ethnicity or faith
After administration, responses should be scored to yield, first, a ‘raw’ score, then a Rasch converted ‘PIJ score’. To calculate the raw score, for questions 1, 2, 4 and 5 responses of ‘strongly agree’ should be assigned a score of 3, ‘mainly agree’ a score of 2, ‘mainly disagree’ a score of 1 and ‘strongly disagree’ a score of 0. For questions 3 and 6 should be reverse scored with responses of ‘strongly disagree’ should be assigned a score of 3, ‘mainly disagree’ a score of 2, ‘mainly agree’ a score of 1 and ‘strongly agree’ a score of 0. Across the six items this will yield individual scores of between 0 and 18. These scores can be converted into PIJ scores for use in analysis (ranging from 0 to 100) using Table 5. Higher scores are associated with greater perceived inequality of justice.
The mean PIJ score among 377 survey respondents was 53.0 (standard deviation = 15.7). The median score was 51.3 (25th percentile = 42.0, 75th percentile = 64.2). Converting scores into strata (low, medium and high scores) gave 27.3% with low scores (scores of 0–44), 53.8% with medium scores (scores of 45–66) and 18.8% with high scores (scores of 67–100). Again, while based on relatively small numbers, these scores and strata provide initial baseline general population estimates for comparison.
The output of the three normal linear models of PIJ score is set out in Table 6. There was a significant association between PIJ score and the experience of civil legal problems. Compared to those who had experienced no problems, respondents who reported problems were associated with a 3.4 point increase in PIJ scoreFootnote 65; with differences becoming more pronounced once self-reported adequacy of problem handling and perceived outcome fairness were included in analysis. Respondents who reported handling (one or more) problems poorly were associated with a 9.8 point increase in PIJ score,Footnote 66 and those who reported (one or more) unfair outcomes were associated with a 7.9 point increase.Footnote 67
While the number of respondents with experience of courts or tribunals was relatively small, there was a significant 9.0 point score difference between those who regarded processes they experienced as fair and those who regarded them as unfair.Footnote 68
There were also significant associations between PIJ score and accounts respondents had heard of lawyers and courts from friends, family or colleagues. In particular, positive accounts were associated with reductions in PIJ score. For accounts of lawyers, those recalling positive accounts scored 5.7 points lower than those not recalling an accountFootnote 69 and 7.2 points lower than those reporting a negative account.Footnote 70 Similarly, for courts, those recalling positive accounts scored 6.5 points lower than those not recalling an accountFootnote 71 and 9.3 points lower than those reporting a negative account.Footnote 72
4 Discussion
While notable efforts have been made to develop composite indicators of, or incorporating, attitudes towards aspects of civil justice systems, less attention has been paid to the psychometric development of standardised measures of single attitude dimensions. The development of such measures would support theoretical development around attitudes to the civil justice system and enable reliable comparison of unidimensional scale estimates between population groups and over time. This would also benefit related research, such as into drivers of legal problem resolving behaviour or public perceptions relating to United Nations SDG 16.3 and 16.6.
Our study sought to apply modern psychometric methods to the construction of reflective instruments to measure attitudes to the civil justice system; principally within a UK context, but potentially of application in similar jurisdictions also. Our approach was firmly grounded in socio-legal theory, but also in the principles of scale development that have evolved in other disciplines and the latest developments in psychometric modelling.
Principal component analysis (PCA) of responses to civil justice system attitude item pool questions indicated five attitude domains. While these did not exactly mirror the eight theoretical domains that informed the design of the item pool, they nevertheless broadly linked to these theoretical domains and other domains suggested in the literature. The five empirically derived domains were labelled trust, access, equality, fairness and respect.
Rasch analysis allowed the development of two appropriately functioning civil justice attitude scales, utilising the items associated with the access and equality domains; yielding an Inaccessibility of Justice (IOJ) scale and a Perceived Inequality of Justice (PIJ) scale. Larger item pools would need to be constructed and tested to develop functioning ‘trust’, ‘fairness’ and ‘respect’ scales with acceptable psychometric properties.
Of the nine questions making up the IOJ scale, six directly concern resources. These link closely to the broad access/efficiency dimension of attitude prominent within the literature. More particularly, they link to Ewick and Silbey’s (1998) ‘game’ story of the law (especially Q3), Galanter’s (1974) theorising about the structural imbalance between the resources and prospects of ‘haves’ and ‘have-nots’ (especially Q2 and Q8), and contemporary debates around access to justice. The remaining questions directly concern efficiency.
Of the six questions making up the PIJ scale, two are also included in the IOJ scale. These concern equality of outcomes. However, the majority of the six questions that make up the PIJ scale concern procedural fairness and motive-based trust.
The findings from our analysis of the relationship between PIJ and IOJ scale scores and people’s experience of civil legal problems, dispute resolution and related social discourse were unambiguous, and have implications in relation to both the development of the literature and access to justice policy.
Sandefur (2007) has theorised that repeated experiences of failure in trying to resolve civil legal problems—such as among the low- and moderate-income residents of the Midwestern American city that she studied—can lead to “frustrated resignation” that manifests in increased inaction in the face of such problems. Survey findings, such as M/A/R/C research’s (1998) finding that negative experience of courts was associated with similar or lowered perceptions of courts, and Currie’s (2009) later finding that survey respondents were more likely to regard the civil justice system as unfair if they reported having experienced civil legal problems, unfavourable outcomes to those problems or difficulty obtaining satisfactory assistance point to changes in attitude that might underpin frustrated resignation.Footnote 73 More recently, Green’s (2016:1270) study of race, class and access to civil justice, has further indicated “connections between experiences with the criminal justice system and utilization of the civil justice system”. This “much broader connection between past experiences and decisions about whether to pursue resolving legal issues, even when past experiences have little or no relationship to the civil justice issue at hand” (1275) reflects both low levels of legal understanding of law and the justice system in general (e.g. Jamieson and Hennessy 2007; Pleasence et al. 2017) and the wide range of potential influences on attitudes to the civil justice system.
While indicating that experience of civil legal problems does not impact significantly on mean IOJ scale scores, our findings confirm that negative experience of problems—defined as self-reported inadequacy of problem handling or negative perceptions of outcome fairness—negatively impacts on IOJ scale scores. Those respondents who felt they handled problems poorly and those who perceived problems as having an unfair outcome were both associated with increases in IOJ score.
In the case of PIJ scale scores, a more general association between experience of civil legal problems and attitudes was also observed. Compared to those who experienced no problems, those who had experienced problems had higher PIJ scale scores. But, in line with IOJ scale score findings, self-reported inadequacy of problem handling and negative perceptions of outcome fairness both pushed scores higher still.
Going further than personal experience, both IOJ and PIJ scale scores were also associated with accounts respondents had heard of lawyers and courts from friends, family and colleagues. Negative accounts of lawyers and courts were both associated with increases in IOJ and PIJ scale scores.
In general, the impact of negative experience and social accounting was to some extent counterbalanced by a positive impact of positive experience and social accounting.
These substantive findings are important in the context of the developing literature, as well as access to justice policy. They support and add further detail to Sandefur’s (2007) theory of behaviour change, as well as giving insight into the broader range of influences on attitudes to the civil justice system. They also show that the value of supporting individuals through dispute resolution processes is manifold; extending not only to improving access to justice in relation to individual cases, but also to maintaining the integrity of the civil justice system and increasing the propensity of individuals, and those to who they relate their experience, to appropriately address future civil legal problems. Thus, policymakers should make all practicable efforts to promote positive experiences of dispute resolution. Of course, the present study was designed to facilitate scale development and did not contain the level of detail on justiciable problem experience found in legal need surveys. Integration of scales into future legal need surveys would allow far greater insight into the relationship between attitudes to justice and problem experience, response to problems and problem outcome.
With access to justice policy increasingly informed by an improving understanding of the public’s experience of law and dispute resolution (e.g. OECD 2017), legal professionals and justice institutions are well placed to utilise current understanding, as well as customised user feedback, to improve access to their services and the experience of users. However, the world over, the majority of dispute resolution occurs beyond the purview of formal dispute resolution authorities and legal professionals (Pleasence et al. 2018), particularly in those countries in which informal justice is particularly prevalent. So—particularly given the impact of social accounts of dispute resolution on civil justice system attitudes, and likely impact of social discourse and media representations more generally—the challenge of improving general public attitudes to the civil justice system is a substantial and complex one.
Finally, links between experience, attitudes and behaviour give measures of attitudes to the civil justice system broad relevance to policy objectives such as United Nations SDG 16.3 and 16.6; although, evidently, the challenges of designing globally valid attitude scales in the civil justice sphere are substantial. Nevertheless, our findings suggest that public perceptions of the justice system, as well as being potential proxies for accessibility and effective functioning, also constitute drivers of accessibility. Thus, measures of attitudes to the civil justice system—such as those measured by the IOJ and PIJ scales—which allow determination of levels and changes in levels of attitudes across and within populations can, with relative ease, provide both indication of system functioning and flagging of problems and populations that warrant investigation and policy intervention.
5 Conclusion
While a much broader range of item pool questions will need to be tested to develop further unidimensional attitude scales in the context of the civil justice system, our findings provide new insight and two theoretically coherent and robust attitude scales. Our approach—drawing on the latest developments in psychometric modelling—also provides a template for the development of similar scales in the future.
Our substantive findings provide further evidence of a psychological process through which negative experience of civil justice problem dispute resolution transforms attitudes in a manner that can manifest in frustrated resignation in the face of later legal problems. Moreover, our findings suggest that, in addition to personal experience, social reporting of the experience of others can bring about the same attitude changes. These findings have clear policy implications, both in relation to supporting individuals through dispute resolution processes and in relation to public legal education to counteract the impact of negative social commentary on the civil justice system. However, while improving the experience of clients and better supporting individuals engaged in formal dispute resolution processes are substantially within the control of legal professionals and justice institutions—and can be informed by increased understanding of the ‘user’ perspective of such processes—counteracting social discourse on civil justice systems may present a greater challenge, as this may well be bound up with wider representations of justice systems that are poorly understood by populations.
Notes
Although there is evidently some overlap in the principles of civil and criminal and civil law. For discussion, see, for example, Dyson (2014).
See also, for example, Genn (1999).
For ease of reference, we use the phrase ‘civil legal problems’ throughout the text to refer to problems that raise legal issues, whether or not this is recognised by those facing them, and whether or not any action taken to deal with them involves lawyers or the wider legal system. Such problems are also commonly referred to as ‘justiciable’ problems, following Genn’s (1999: 12) use of that term.
The World Values Survey asked about confidence in ‘the legal system’ throughout waves 1–3 and confidence in ‘the courts’ from wave 5 (Inglehart et al. 2014). All 8 waves of the European Social Survey have asked about trust in ‘the legal system’. The fifth wave also included a rotating module on trust in the police and courts, with an entirely criminal focus. The questionnaire can be downloaded at: http://www.europeansocialsurvey.org/docs/round5/fieldwork/source/ESS5_source_main_questionnaire.pdf.
Such surveys are routine in the United States. In addition to national surveys, such as the 1998 Perceptions of the US Justice System survey (MARC Research 1998)—which included questions from the influential Yankelovich, Skelly and White (1978) survey for the National Center for State Courts—and 1999 How the Public Views the State Courts Survey (National Center for State Courts 1999), Rottman et al. (2003) listed 29 state surveys in the 25 years between 1977 and 2002. The annual ‘State of State Courts’ surveys make no distinction (details available at http://www.ncsc.org/Topics/Court-Community/Public-Trust-and-Confidence/Resource-Guide.aspx), and even surveys such as the 1999 How the Public Views the State Courts Survey have rarely specified a civil context.
It is to be noted that in many developing countries “legal pluralism is common” (Barendrecht 2011: 281) and extensive recourse is made to customary and religious justice systems. In such systems, distinctions between civil and criminal law will often not feature.
As discussed further below, in developing countries informal justice systems can be more extensive and utilised than formal justice systems (e.g. UNDP 2009).
A concept first expounded by Golub and McQuay (2001).
For example, the UNDP (2012) have detailed 23 justice assessments in 15 countries in the Asia Pacific region between 2000 and 2010 alone.
United Nations General Assembly Resolution 70/1, 25th September 2015. The United Nations Sustainable Development Goals build on the earlier Millennium Development Goals and are collectively directed towards ending poverty, ensuring economic prosperity and sustaining the environment. In SDG 16.3, member states agreed to “promote the rule of law at the national and international levels, and ensure equal access to justice for all.” In the case of SDG 16.6, which concerns the development of “effective, accountable and transparent” government institutions, attitudes to the civil justice system are directly relevant, as civil courts represent a core form of government institution.
See, also, for example, M/A/R/C Research (1998), in which negative experience of courts was associated with similar or worsened perceptions of courts, but there was no mirror finding in respect of positive experience. However, this can be contrasted with Kritzer and Voelker’s (1998) finding that recent court use in Wisconsin was associated with more favourable opinions about the courts.
In the case of China, Myanmar, the Philippines and Thailand the figure was 1% or less. As Piest et al. (2016: 81) observed in the context of the Ugandan Justice Needs and Satisfaction Survey, “the formal judicial system is marginal to the experience of justice.” A recent global review of legal needs survey findings found this to be the norm, with only a minority of surveys indicating more than 10% of civil law problems had resolved through courts or tribunals (Pleasence et al. 2018).
For example, Damaska (1986:1) drew attention to the “immense and bewildering … diversity of arrangements and institutions through which justice is variously administered in modern states,” in developing a typology of justice systems which contrasted ‘policy-implementing’ with ‘conflict-solving’ objectives and ‘hierarchical authority’ with ‘coordinate authority’.
Both single component and multiple component attitude conceptualisations have been advanced, and much empirical study has been directed towards validating different approaches and exploring relationships between attitude and behaviour (e.g. Bagozzi 1981).
It would also be possible to look beyond formal civil justice systems to informal justice mechanisms and broader access to justice issues, particularly in the context of developing countries, as discussed above.
The project was directed to developing a methodology to measure the cost and quality of access to justice.
Verdonschot et al. (2008: 7–8) summarise the potential criteria of distributive justice as being on the basis of equity (proportionate to contribution), equality (equal shares), need (proportionate to individual needs), accountability (proportionate to volitional contribution) and efficiency (to maximise the welfare of the parties).
See, further, Mirrlees-Black (2001).
Costa (2015) expands this, suggesting that perhaps the key distinction between types of indicator lies in their psychometric properties. Reflective indicators (such as those in the present study) are manifestations of a latent construct (e.g. inaccessibility of justice), with constituent indicators highly correlated with one another. Methods such as item response theory (and Rasch analysis used in this study) assume that indicators are reflective. For a construct represented by causal or composite indicators, indicators may independently contribute to variation in the construct, and need not be correlated with one another (although causal indicators should be conceptually related to each other). In addition, in the reflective context, the construct is defined independently of the items and represented by a (theoretically) infinite pool of (largely interchangeable) items. In the composite context, the construct is defined by the indicators, with adding/removing/changing items altering the definition of the construct (which is therefore arguably not ‘latent’ and may be purely operational). Both Engelhard and Wang (2014) and Fayers and Machin (2016) provide further useful distinction between composite/formative and reflective approaches, including a specific comparison with an IRT/Rasch approach (Engelhard and Wang 2014).
In the context of composite indicators, the OECD has produced specific and comprehensive guidance on steps and statistical methods (OECD 2008).
An extensive range of surveys was examined, including both surveys within the legal domain (such as the Crime Survey for England and Wales, Scottish Crime and Justice Survey and the increasing number of legal needs surveys and access to justice assessment surveys conducted across the globe), and more general surveys (such as the British Household Panel Survey, British Social Attitudes Survey, European Social Survey, General Household Survey, Home Office Citizenship Survey, Northern Ireland Health and Social Wellbeing Survey, Poverty and Social Exclusion Survey and Scottish Household Survey).
The item pool questions utilised a four-point Likert response format: strongly agree; mainly agree; mainly disagree; strongly disagree. Surveys drawn from included the British Social Attitudes Survey, the Crime Survey for England and Wales, the New Zealand National Survey of Unmet Legal Needs and Access to Services, the Paths to Justice surveys and the Scottish Crime and Justice Survey.
The Community Life Survey web experiment was “one of the largest ever tests of web survey methodology in which random sampling has been employed” (TNS-BMRB 2013: 4).
As the authors of the Community Life Survey web experiment noted, although there were notable differences observed between the face-to-face and online-postal samples, online-postal respondents took the same length of time to complete the survey questionnaire and generally yielded similar estimates; although estimates from the different variants differed and it was suggested that sometimes differences might “be large” (TNS-BMRB 2013: 10). On some measures, the profile of the sample deteriorated when postal questionnaires were added, despite an increase in response rate.
The response rate was just over 10%. While this constitutes a low response rate, this had little bearing on the utility of the survey, as the aim of the survey was to explore attitudes to law and construct scales around specific attitude domains. This objective has very different requirements to an exercise focussed on producing population estimates. The most important consideration for scale development is to ensure a broad range of perspectives, demographics and levels of confidence, rather than to minimise total survey error (Weisberg 2005). More generally, numbers of respondents were more than adequate for PCA, scale development and Rasch analysis. There has been some debate regarding appropriate sample size or subject to item ratios for PCA (Osborne and Costello 2004). Nonetheless, the total sample of 1061 would be regarded as ‘excellent’ by Comfrey and Lee (1992), while the person to item ratio of over 30:1 comfortably exceeds commonly cited ratios such as 10:1 (Nunnally 1978). For scale development using Rasch analysis, Linacre (1994) provides guidance on samples required for useful item calibrations and measurement stability. A sample of 1061 comfortably exceeds 99% confidence that no item calibration is more than half a logit away from its stable value.
Fifty-eight percent of respondents were women, and 92% white. Fifty-three percent were in work, 25% retired, 6% looking after the home, 5% in full-time education, 5% unable to work because of a long-term illness or disability, and 1% unemployed and looking for work. Seven percent of respondents were aged between 16 and 24, 32% from 25 to 44, 36% from 45 to 64, 18% from 65 to 74 and 8% were 75 or older. Thirty-seven percent owned their own home outright, 27% owned their home with the help of a mortgage, 2% had shared ownership and 27% were renting their home. Forty-one percent had a degree of equivalent qualification, 44% another form of qualification and 15% no qualifications. Twenty-six percent reported a long-term limiting illness or disability.
Respondents were asked whether they recalled friends, family or colleagues telling you about good or bad experiences with lawyers or courts or tribunals.
If lawyer use was identified, respondents were asked how satisfied respondents had been.
If respondents had contact with courts or tribunals, they were asked how fair processes had been.
There has been some debate as to the relative merits of PCA and factor analysis (e.g. Costello and Osborne 2005), with proponents of each as well as those arguing there is little substantive difference. For our purposes, they serve much the same function.
This was indicated by a Keiser–Meyer–Olkin (KMO) statistic of 0.89 (above the common cut-off of 0.6) (Tabachnick and Fidell 2013).
Oblique rotation was used since it allows correlation among our attitude to law components, which is a reasonable assumption and generally the case with social science data (Costello and Osborne 2005). Orthogonal rotation methods do not allow for correlation between components, which can result in the loss of valuable information.
Oblique rotation results in a pattern and a structure matrix. The pattern matrix contains the coefficients for the linear combination of the variables; or the matrix of unique relationships, uncontaminated by overlap among components (Tabachnick and Fidell 2013). The structure matrix represents the correlations between the variables and the components. As is typically the case, we focused predominantly on the pattern matrix to interpret components. Loadings of 0.32 and above were interpreted, which represents 10% overlapping variance between item and component. The higher the loading, the more the variable is a pure measure of the component. A loading of 0.71 (50%) overlapping could be considered ‘excellent’, 0.63 (50%) overlapping ‘very good’, 0.55 (30% overlapping) ‘good’, 0.45 (20%) overlapping, ‘fair’ and 0.32 (10% overlapping) ‘poor’ (Tabachnick and Fidell 2013).
As a rule of thumb, five items with loadings in excess of 0.5 is likely to indicate a solid component, with three or fewer indicating a weak or unstable component, with further items likely to be needed (Costello and Osborne 2005).
Examination of the scree plot involves visually identifying the natural elbow or break point in a graph of eigenvalues. Parallel analysis involves comparing eigenvalues from analysis to a randomly generated dataset of the same size, with those components with eigenvalues above the random values retained.
Bond and Fox (2015) provide an introduction to Rasch analysis and its use, while Tennant and Conaghan (2007) provide a helpful guide to using, and reporting findings from Rasch analysis See, also, Boone et al. (2014).
Examination of the scree plot involves visually identifying the natural elbow or break point in a graph of eigenvalues. Parallel analysis involves comparing eigenvalues from analysis to a randomly generated dataset of the same size, with those components with eigenvalues above the random values retained.
Until recently, (what are now termed) traditional psychometric methods were used to construct and evaluate rating scales, underpinned by Classical Test Theory (CTT) methods. See, for example, DeVellis (2006, 2012). In CTT, each person is assumed to have an observed score (O) that represents their true score for a trait (T) plus an error term (ε), with O = T + ε. The standard deviation of the errors (the standard error of measurement) is directly related to reliability, with reducing errors moving the observed scores closer to the true scores (Smith et al. 2016). However, the deficiencies of traditional methods and additional advantages of modern psychometric methods have led to their falling into disuse. CTTs psychometric properties are at an overall test level rather than at an individual item level. Error scores are assumed to be uncorrelated with each other and with the true scores, and observed and true scores are linearly related. However, since true scores and error scores cannot be determined, the appropriateness of the assumptions cannot be verified (Allen and Yen 2002). In addition, many rating scales (including our items) employ Likert scale type response formats (with sequentially ordered response options assigned sequentially ordered integers). Traditional methods assume that ordinal level total scores approximate to interval level measures (Allen and Yen, above), which is not the case and should not be treated as such (Smith et al. 2016). Other problematic assumptions include a homogeneous contribution of items to the final score and equivalence of response options among different items (Martinez-Martin and Forjaz 2012). Moreover, with traditional methods, evaluations of scales are sample dependent and the measurement of people is scale dependent, undermining the use of such total scores as measurements (Allen and Yen 2002). Like CTT, modern methods—such as Rasch analysis—set out theories of how rating scale scores relate to measurements of the variables they seek to estimate. However, unlike CTT, they are underpinned by mathematical models of these theories, enabling verification through formal and rigorous testing (Allen and Yen 2002). The basic rationale is that an individual’s response to a specific item is based on (a log function of) their characteristics and characteristics of the item (i.e. its difficulty, or level of confidence required to endorse it). So, for Rasch analysis, psychometric properties are at the item, rather than the test level. modern methods can also explore whether items are equivalent in meaning to different respondents (differential item functioning), allow inclusion of items with different response formats on the same scale, assess person fit (e.g. assessment of the extent to which individual respondents provide useful data or are taking the exercise seriously) and allow computer adaptive testing (Hays et al. 2000). Missing data is also less of an issue for Rasch analysis (Boone et al. 2014). As psychometric properties are at an item level (unlike CTT), individual items can be comprehensively evaluated. Crucially, scales developed also allow ordered observations (such as Likert scales) to be transformed into an interval scaled measure of the latent trait (Salzberger 2010; Wright and Linacre 1989), allowing for a broader range of statistical analyses.
Rasch is a logistic model (i.e. the expression e/(1 + e) is central). Put simply, in our context, the probability of affirming an item is a logistic function of the difference between an individual’s attitude in a specific domain and the level of that domain an item expresses. So, where an individual generally has a positive attitude towards an aspect of the justice system and level of positivity expressed by affirming an item is low (i.e. an ‘easy’ item that most people would affirm), probability tends towards one. Conversely, where an individual’s attitude is generally less positive and level of positivity expressed by affirming an item is high (i.e. a more ‘difficult’ item that most people would not affirm), probability tends towards zero.
The partial credit model allows thresholds to vary by item, rather than the rating scale model, where items share rating scale structure. A simple introduction to approaches can be found in Bond and Fox (2015), with a short description of advantages and disadvantages in Linacre (2000). The partial credit model was chosen to allow the flexibility of different response formats per item as well as to allow differences between response categories to vary.
Andrich et al. (2016). Specialist software is required to conduct Rasch analysis, and this, as well as relative technical difficulty, have been cited as a barrier to use (e.g. Hays et al. 2000; Hobart and Cano 2009). Common software used to implement Rasch analysis include RUMM2030 (Andrich et al. 2016), used in the current analysis) and WinSteps (Linacre 2016).
A significant value would indicate that hierarchical ordering of items varies across the trait (e.g. confidence) which would compromise the required property of invariance (Tennant and Conaghan 2007). Invariance is the principal that the ability of respondents is independent of the test and conversely, that the difficulty of items is independent of respondents.
Such cases can seriously affect the fit at an item level (Tennant and Conaghan 2007).
Good discrimination is indicated by dots (mean scores for each class interval) lying on or close to the curve.
Separation reports how many statistically distinguishable measurement levels exist in a sample (when high and low levels are treated as accidental). Separation = square-root (true variance − error variance). Strata are considered a refinement of separation (Linacre 2013) where high and low measures are considered valid levels of performance, which seems a reasonable assumption in our context. Strata = (4 * Separation + 1)/3, so for example, a person separation index of 0.8 would allow two levels of separation and three strata (e.g. low, medium and high). Linacre (2013) also sets out the approximate percentage of samples in each separation or strata (for approximately normally distributed samples—for two strata, 50% at each level; for three, 23, 54 and 23%; for four, 14, 36, 36 and 14%. For examples of the use of strata with Rasch analysis see Prietro, Alonso and Lamarca (2003) and Duncan et al. (2003).
No problems, all problems well-handled, at least one problem not well-handled.
No lawyer use, satisfied with lawyer use, dissatisfied with lawyer use.
No court/tribunal use, fair court/tribunal use, unfair court/tribunal use.
No legal problems, all quite/very fair outcomes, at least one unfair outcome.
Construct validity (or external validity) concerns “the degree to which measures are related to external measures of the same construct, similar constructs, and other constructs” (Wolfe and Smith 2007: 268). In our study, no external measures of the same attitude constructs exist. However, we did include a number of questions designed to measure similar, related constructs, allowing examination of construct validity by assessing the relationship between attitudes and responses to these variables.
For the purposes of PCA and Rasch analysis, the following items were reverse coded: 2, 3, 4, 8, 10, 12, 17, 18, 21, 24, 29, 30, 32, 33, 34.
The exceptions were Items 15, 20, 22 and 23.
Absolute scores, ignoring negative values.
The initial Rasch model of the five items included in the trust component indicated some person misfit. The PSI (0.69) was also (just) short of acceptable, and targeting was poor. The initial Rasch model of the five items included in the fairness component indicated poor overall model and item fit. The PSI (0.52) was also well below acceptable levels. Removing one item improved the overall model and item fit (to acceptable levels), but the PSI (0.57) remained low. The initial Rasch model of the six items included in the respect component also indicated poor overall model and item fit, and the PSI (0.55) was similarly low. Removal of items did not yield an acceptable overall model.
\(\chi_{60}^{2}\) = 115.50, p = 0.001. Which is less than the Bonferroni corrected p value of 0.005 (0.05/10 on account of the ten items in the scale).
\(\chi_{45}^{2}\) = 55.93, p = 0.13.
The PSI of 0.74 was relatively low, but above acceptable limits making it suitable for group use (Tennant and Conaghan 2007). The relatively low PSI was in part a result of a general negativity towards accessibility of justice, with the four-category Likert response format not being as effective in capturing a range of views as in other contexts.
Linacre (2013) provides further guidance on use of separation and strata levels with Rasch analysis.
Model 2—\(\chi_{1}^{2}\) = 13.56, p < 0.001. In addition, the difference between those who had handled all problems well and those who had handled one or more problems poorly was also statistically significant (9.3 points—\(\chi_{1}^{2}\) = 11.87, p = 0.001).
Model 3—\(\chi_{1}^{2}\) = 11.96, p = 0.001. The 7.3 point difference between those who had experienced only fair outcomes and those who had experienced one or more unfair outcomes was also statistically significant (\(\chi_{1}^{2}\) = 11.66, p = 0.001).
\(\chi_{1}^{2}\) = 13.36, p < 0.001.
\(\chi_{1}^{2}\) = 8.45, p = 0.004.
\(\chi_{1}^{2}\) = 6.40, p = 0.011.
\(\chi_{72}^{2}\) = 123.32, p = < 0.001.
\(\chi_{48}^{2}\) = 53.80, p = 0.26.
\(\chi_{1}^{2}\) = 4.19, p = 0.041.
\(\chi_{1}^{2}\) = 7.63, p = 0.006.
\(\chi_{1}^{2}\) = 11.96, p < 0.001.
\(\chi_{1}^{2}\) = 3.92, p = 0.048.
\(\chi_{1}^{2}\) = 8.79, p = 0.003.
\(\chi_{1}^{2}\) = 11.18, p = 0.001.
\(\chi_{1}^{2}\) = 10.02, p = 0.002.
\(\chi_{1}^{2}\) = 14.75, p < 0.001.
Similarly, but more generally, Jamieson and Hennessy (2007) found that previous court experience increased the likelihood of American’s believing that courts favour the wealthy or connected. However, other findings, such as Kritzer and Voelker’s (1998) finding that recent court use in Wisconsin was associated with more favourable opinions about the courts point to the likely complexity of the relationship between experience and attitudes.
References
Adams K, Botero JC, Coto A, Evangelides A, Gryskiewicz A, Gutiérrez Patiño C, Chamness Long S, McDaniel L, Mujeeb M, Ponce A, Treacy N (2017) Global insights on access to justice. World Justice Project, Washington, D.C.
Allen MJ, Yen WM (2002) Introduction to measurement theory. Waveland Press, Long Grove
Andrich D, Sheridan B, Luo G (2016) RUMM2030 [computer software] Rumm Laboratories, Perth
Bach Commission on Access to Justice (2016) The crisis in the justice system in England and Wales. The Fabian Society, London
Bagozzi RP (1981) Attitudes, intentions and behavior: a test of some key hypotheses. J Pers Soc Psychol 41(4):607–627
Barendrecht M (2011) Rule of law, measuring and accountability: problems to be solved bottom up. Hague J Rule Law 3:170–185
Barendrecht M, Gramatikov M, Giesen I, Laxminarayan M, Kamminga P, Klaming L, Verdonschot JH, van Zeeland C (2010) Measuring access to justice in a globalising world: the Hague model of access to justice. HiiL, Den Haag
Barendrecht M, Gramatikov M, El Khoury R, Motiejunas G, Muller S, Osborne D, Heijstek-Ziemann K, Verdonschot JH (2014) The need of the Malians for justice: towards more fairness. HiiL, Den Haag
Bond T, Fox CM (2015) Applying the Rasch model: fundamental measurement in the human sciences, 3rd edn. Routledge, New York
Boone W, Staverand JR, Yale MS (2014) Rasch analysis in the human sciences. Springer, Dordrecht
Cattell CB (1966) The scree test for the number of factors. Multivar Behav Res 1(2):245–276
Christensen KB, Engelhard G Jr, Salzberger T (2012) Ask the experts: Rasch vs. factor analysis. Rasch Measure Trans 26:1373–1378
Christensen KB, Kreiner S, Mesbah M (2013) Rasch models in health. Wiley, Hoboken
Coaley K (2014) An introduction to psychological assessment and psychometrics, 2nd edn. London, Sage
Colman AM (2015) Oxford dictionary of psychology. Oxford University Press, Oxford
Colquitt JA, Conlon DE, Wesson MJ, Porterand COLH, Ng KY (2001) Justice at the millenium: a meta-analytic review of 25 years of organizational justice research. J Appl Psychol 86(3):425–445
Comfrey AL, Lee HB (1992) A first course in factor analysis. Lawrence Erlbaum Associates, Hillsdale
Costa SJ (2015) Reflective, causal, and composite indicators of quality of life: a conceptual or an empirical distinction? Qual Life Res 24:2057–2065
Costello AB, Osborne J (2005) Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval 10(7):1–9
Cunha LG, Oliviera FL, Glezer RE (2014) Brazilian Justice Confidence index—Measuring public perception on judicial performance in Brazil. International Law, Revista Colombiana de Derecho Internacional 25:445–472
Currie A (2009) ‘A lightning rod for discontent’: justiciable problems and attitudes towards the law and the justice system. In: Buck A, Pleasence P, Balmer NJ (eds) Reaching further: innovation, access and quality in legal services. TSO, Norwich
Damaska MR (1986) The faces of justice and state authority. Yale University Press, New Haven
Davies M (2010) Legal pluralism. In: Cane P, Kritzer HM (eds) The Oxford handbook of empirical legal research. Oxford University Press, Oxford
Desai D, Isser D, Woolcock M (2012) Rethinking justice reform in fragile and conflict-affected states: lessons for enhancing the capacity of development agencies. Hague J Rule Law 4(1):54–75
DeVellis RF (2006) Classical test theory. Med Care 44(11)(3):50–59
DeVellis RF (2012) Scale development: theory and applications, 3rd edn. Thousand Oaks, Sage
Dignan T (2006) Northern Ireland legal needs survey. Northern Ireland Legal Services Commission, Belfast
Duncan PW, Bode RK, Min Lai S, Perera S (2003) Rasch analysis of a new stroke-specific outcome scale: the stroke impact scale. Arch Phys Med Rehabil 84(7):950–963
Dyson M (2014) Unravelling tort and crime. Cambridge University Press, Cambridge
Everitt BS, Dunn G (2013) Applied multivariate data analysis. Wiley, London
Ewick P, Silbey SS (1998) The common place of law: stories from everyday life. University of Chicago Press, Chicago
Fisher W (1992) Reliability, separation, strata statistics. Rasch Meas Trans 6(3):238
Franklyn R, Budd T, Verrill R, Willoughby M (2017) Findings from the legal problem and resolution survey, 2014–15. Ministry of Justice, London
Gable RK, Wolf MB (1993) Instrument development in the affective domain: measuring attitudes and values in corporate and school settings. Kluner Academic, Boston
Galanter M (1974) Why the ‘haves’ come out ahead speculations on the limits of legal change. Law Soc Rev 9(1):95–160
Galanter M (1999) Farther along. Law Soc Rev 33(4):1113–1123
Genn H (1999) Paths to justice. Hart, Oxford
Genn H, Paterson A (2001) Paths to justice Scotland. Hart, Oxford
Gnaldi M, Del Sarto S, Maggino F (2017) The role of extended IRT models for composite indicators construction. In: Maggino F (ed) Complexity in society: From indicators construction to their Synthesis. Springer, Cham, Switzerland
Golub S, McQuay K (2001) Law and policy reform at the Asian Development Bank. Asian Development Bank, Manila
Gramatikov MA, Barendrecht M, Verdonschot JH (2011) Measuring the costs and quality of paths to justice: contours of a methodology. Hague J Rule Law 3(2):349–379
Green SS (2016) Race, class and access to civil justice. Iowa L Rev 101:1263–1321
Hadfield GK (2010) Higher demand, lower supply? A comparative assessment of the legal resource landscape for ordinary Americans. Fordham Urb L J 37:129–156
Hays RD, Morales LS, Reise SP (2000) Item response theory and health outcome measurement in the 21st century. Med Care 38(9):1128–1142
HiiL (2016a) Justice needs in Ukraine: Legal problems in daily life. HiiL, Den Haag
HiiL (2016b) Justice needs in the United Arab Emirates. HiiL, Den Haag
Hobart J, Cano S (2009) Improving the evaluation of therapeutic interventions in multiple schlerosis: the role of new psychometric methods. Health Technol Assess 13(12):1–177
Horn JL (1965) A rationale and test for the number of factors in factor analysis. Psychometrika 30(2):179–185
Inglehart R, Haerpfer C, Moreno A, Welzel C, Kizilova K, Diez-Medrano J, Lagos M, Norris P, Ponarin E, Puranen B (2014) World Values Survey: All rounds—country pooled datafile Version: http://worldvaluessurvey.org/WVSDocumentationWVL.jsp. JD Systems Institute, Madrid
Jamieson KH, Hennessy M (2007) Public understanding of and support for the courts: survey results. Georgetown Law Rev 95:899–902
Kind M, Gramatikov M, Núñez R, El Khoury R, Kernchen N, Astafjevas G, Lapeira R (2017) Justice needs and satisfaction in Lebanon: Legal problems in daily life. HiiL, Den Haag
Klaming L, Giesen I (2008) Access to justice: the quality of the procedure. Tilburg Institute for Interdisciplinary Studies of Civil Law and Conflict Resolution Systems: Tilburg
Kritzer HM, Voelker J (1998) Familiarity breeds respect: how Wisconsin citizens view their courts. Judicature 82(2):59–64
Linacre JM (1994) Sample size and item calibration stability. Rasch Meas Trans 7(4):328
Linacre JM (2000) Comparing ‘partial credit models’ (PCM) and ‘rating scale models’ (RSM). Rasch Meas Trans 14(3):768
Linacre JM (2013) Reliability, separation and strata: percentage of sample in each level. Rasch Meas Trans 26(4):1399
Linacre JM (2016) Winsteps® [computer program]. Winsteps.com, Beaverton
Maggino F (2017) Complexity in society: From indicators construction to their synthesis. Springer, Cham, Switzerland
M/A/R/C Research (1998) Perceptions of the US Justice System. American Bar Association, Chicago
Marais I, Andrich D (2008) Effects of varying magnitude and patterns of local dependence in the unidimensional Rasch model. J Appl Meas 9(2):105–124
Martinez-Martin P, Forjaz MJ (2012) How to evaluate validation data. In: Sampaio C, Goetz CG, Schrag A (eds) Rating scales in Parkinson’s disease: clinical practice and research. Oxford University Press, Oxford
Masterson SS, Tong N (2015) Justice perception formation in social settings. In: Cropanzano R, Ambrose ML (eds) The Oxford handbook of justice in the workplace. Oxford University Press, Oxford
Mirrlees-Black C (2001) Confidence in the criminal justice system: findings from the 2000 British Crime Survey. Home Office, London
Moorhead R, Sefton M, Scanlan L (2008) Just satisfaction? What drives public and participant satisfaction with courts and tribunals. Ministry of Justice, London
National Center for State Courts (1999) How the public views the state courts. National Center for State Courts, Williamsburg
Nunnally JC (1978) Psychometric theory, 2nd edn. McGraw Hill, New York
OECD (2017) OECD policy roundtable on access to justice: Session notes. OECD, Paris
Osborne JW, Costello AB (2004) Sample size and subject to item ratio in principal components analysis. Pract Assess Res Eval 9(11):1–9
Parsons J (2011) Developing clusters of indicators: an alternative approach to measuring the provision of justice. Hague J Rule Law 3:170–185
Piest J, Gramatikov M, Muller S, Heijstek-Ziemann K, Sallali J (2016) Justice needs in Uganda: Legal problems in daily life. Den Haag, HiiL
Pleasence P (2006) Causes of action: Civil law and social justice, 2nd edn. TSO, Norwich
Pleasence P, Balmer NJ (2013) English and Welsh civil and social justice panel survey: wave 2. Legal Services Commission, London
Pleasence P, Buck A, Balmer NJ, O’Grady A, Genn H, Smith M (2004a) Causes of action: Civil law and social justice, 1st edn. TSO, Norwich
Pleasence P, Balmer NJ, Patel A, Devir C (2004b) Civil Justice in England and Wales 2009. Legal Services Commission, London
Pleasence P, Balmer NJ, Sandefur RL (2013) Paths to justice: a past, present and future roadmap. Nuffield Foundation, London
Pleasence P, Balmer NJ, Denvir C (2017) Wrong about rights: public knowledge of key areas of consumer, housing and employment law in England and Wales. Mod Law Rev 80(5):836–859
Pleasence P, Balmer NJ, Chapman P (2018) Legal needs surveys and access to justice: a guidance document. Draft report presented to the 2018 OECD policy roundtable on equal access to justice, Riga, Latvia, 5–6 July 2018
Prietro L, Alonso J, Lamarca R (2003) Classical test theory versus Rasch analysis for quality of life questionnaire reduction. Health Qual Life Outcomes 1:27
Quintanilla VD, Avtgis AB (2017) The public believes predispute binding arbitration clauses are unjust: ethical implications for dispute-system design in the time of vanishing trials. Fordham Law review 85:2119–2149
Robinson PH (1996) The criminal-civil distinction and the utility of desert. Boston University Law Review 76:201–214
Rosenberg MJ, Hovland CI (1960) Cognitive, affective and behavioural components of attitudes. In: Rosenberg MJ, Hovland CI (eds) Attitude organisation and change: an analysis of consistency among attitude components. Yale University Press, New Haven
Rottman DB, Hansen R, Mott N, Grimes L (2003) Perceptions of the courts in your community: The influence of experience, race and ethnicity. National Center for State Courts, Williamsburg
Saisana M, Saltelli A (2014) JRC Statistical Audit of the WJP Rule of Law Index. In: World Justice Project (ed) World Justice Project Rule of Law Index. World Justice Project, Washington, D.C
Saisana M, Saltelli A, Tarantola S (2005) Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. J R Stat Soc Ser A Stat in Soc 168(2):307–323
Salzberger T (2010) Does the Rasch model convert an ordinal scale to an interval scale? Rasch Meas Trans 24:2
Sandefur RL (2007) The importance of doing nothing: everyday problems and responses of inaction. In: Pleasence P, Buck A, Balmer NJ (eds) Transforming lives: law and social process. TSO, Norwich
Smith EV (2002) Detecting and evaluating the impact of multidimensionality using item fit statistics and principal components analysis of residuals. J Appl Meas 3:205–231
Smith W, Patel A, McCrone P, Jin H, Osumili B, Barrett B (2016) Reducing outcome measures in mental health: a systematic review of the methods. J Ment Health 25(5):461–472
Stevens J (1992) Applied multivariate statistics for the social sciences. Lawrence Erlbaum Associates, Hillsdale
Stratton M, Lowe D (2005) Public confidence and the civil justice system: what do we know about the issues?. Canadian Forum on Civil Justice, Edmonton
Summerfield A, Freeman L (2014) Public experiences of and attitudes towards the family justice system. Ministry of Justice, London
Tabachnick BG, Fidell LS (2013) Using multivariate statistics, 6th edn. Harlow, Pearson
Tennant A, Conaghan PG (2007) The Rasch measurement modal in rheumatology: what is it and why use it? when should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum 57:1358–1362
Teresi JA, Kleinman M, Ocepek-Welikson K (2000) Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures. Stat Med 19:1651–1683
Tinsley HEA, Tinsey DJ (1987) Uses of factor analysis in counseling psychology research. J Couns Psychol 34(4):414–424
TNS-BMRB (2013) Community life survey: summary of web experiment findings. TNS-BMRB, London
Tyler TR (1990) Why people obey the law. Yale University Press, New Haven
Tyler TR, Huo YJ (2002) Trust in the law: encouraging public cooperation with the police and courts. Sage, New York
Tyler TR, Braga A, Fagan J, Meares T, Sampson R, Winship C (2007) Legitimacy and criminal justice: international perspectives. In: Tyler TR (ed) Legitimacy and criminal justice. Sage, New York
UNDP (2009) Informal justice systems: Charting a course for human rights-based engagement. UNDP, New York
UNDP (2012) Access to justice assessments in the Asia Pacific: a review of experiences and tools from the region. United Nations Development Programme, Bangkok
Velicer WF, Jackson DN (1990) Component analysis versus common factor analysis—some further observations. Multivar Behav Res 25(1):97–114
Verdonschot JH, Barendrecht M, Klaming L, Kamminga P (2008) Measuring access to justice: quality of outcomes. Tilburg Institute for Interdisciplinary Studies of Civil Law, and Conflict Resolution Systems: Tilburg
Watkins MW (2000) Monte Carlo PCA for Parallel Analysis [computer program]. http://edpsychassociates.com/Watkins3.html. Accessed 25 March 2018
Weisberg HF (2005) The total survey error approach: a guide to the new science of survey research. University of Chicago Press, Chicago
Wilson C (2012) The public and the justice system: attitudes, drivers and behaviour—a literature review. Scottish Government Social Research, Edinburgh
Wolfe EW, Smith EV (2007) Instrument development tools and activities for measure validation using Rasch models: part II—validation activities. J Appl Meas 8(2):204–234
Wright BD, Linacre JM (1989) observations are always ordinal; measurements, however, must be interval. Arch Phys Med Rehabil 70(12):857–860
Wright BD, Masters GN (2002) Number of person or item strata. Rasch Meas Trans 16(3):888
Yankelovich Skelly, White Inc (1978) The public image of courts. National Center for State Courts, Williamsburg
Zoellick RB (2008) Fragile states: securing development. Survival 50(6):67–84
Funding
This project was funded by the Legal Education Foundation, London.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Pleasence, P., Balmer, N. Measuring the Accessibility and Equality of Civil Justice. Hague J Rule Law 10, 255–294 (2018). https://doi.org/10.1007/s40803-018-0079-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40803-018-0079-0