Keywords

When Deanna Amato had her tax deductions withheld by Centrelink in 2019—a service of the Australian government's Income Compliance Program, responsible at the time for, among other things, the recovery of social security overpayments—she had no idea that this was one of the many erroneous outputs of an automated legal decision-making software. The so-called Robodebt case involved hundreds of thousands of Australians being issued automated reports that they owed money to the Australian welfare system. It was later acknowledged that the use of inadequate data associated with a rogue algorithmic practice resulted in the Australian Government sending out more than 400,000 requests for refunds of non-existent social overpayments (Bennett 2019).

This so-called “Robodebt” case demonstrated that the uncontrolled use of artificial intelligence and machine learning tools for decision-making purposes with legal impacts is far from mature, notwithstanding the billions of dollars in revenue already generated annually by these types of tools (Alston 2019).

The Robodebt case: The Robodebt scheme, also known as the Centrelink debt recovery program, ran between 2015 and November 2019. Using comparison algorithms, Centrelink, an Australian social security state agency, matched the average annual tax rates and tax payments of tax deduction recipients with their actual fortnightly tax returns. If a difference in amount was found in favour of the social security administration, the algorithm systematically concluded that there was an error in the declaration and automatically launched a recovery procedure. In effect, welfare recipients were assumed to owe money until they could prove otherwise. Even worse, Centrelink issued a debt notice based on averaged Australian Taxation Office (ATO) information alone and applied a 10% penalty. Centrelink did not use its information-gathering powers to contact the concerned taxpayer’s employers or banks to provide details of actual earnings for the relevant fortnights to determine whether the taxpayer actually owed the automated debt calculation. When exposed through the Amato case, the proceeding resulted in a large class-action suit, exposing a massive failure of public administration as the government unlawfully raised $1.76 billion in debts against 443,000 people. During that time, the Australian government had also pursued about 381,000 people, unlawfully recovering $751 million, including through private debt collectors. In court, the Australian Government conceded that the averaging process using ATO income data to calculate all Robodebt was unlawful.

Thus, the illusion that administrative or legal procedures can be facilitated or accelerated clashes with the reality of field usage, and raises serious questions regarding human rights (Langford 2020), notwithstanding the strong impacts on involved or concerned parties and an even more significant effect on vulnerable populations (Carney 2020).

In this context, this chapter has three objectives:

  • To identify the types of cases concerned by these situations to propose perimeters of vigilance as well as to identify the critical paths leading to potential errors.

  • To identify (through the different cases raised in existing publications) the elements of technological design that can lead to the scandals and biases mentioned above: the “glitches” in the sense of Meunier et al. (2019).

  • To suggest some good practices to help the supporters of this type of solution to prosper without damaging public liberties, the rights of the citizens or the possibilities of unbiased access to law and justice.

14.1 Where and How the Automation of Legal Acts Takes Place

The example of “Robodebt” has made it possible to highlight that the use of algorithms in processes with legal impacts is not limited to the courts of justice. Artificial intelligence (AI) is the set of theories and techniques implemented to create machines capable of simulating human intelligence in its neural architecture for various uses and in various fields. It combines a set of technological tools (deep learning, structured or unstructured predictive analysis, data) at the origin of methods for solving problems with high logical and/or algorithmic complexity. There are therefore many uses open to AI (support, data collection, decision making, learning, prediction), which can be found in what is known today as LegalTech, i.e. the involvement of technology in developing, proposing or providing products or services related to law and justice. The LegalTech market will be worth more than US$25 billion by 2025, with US$6.37 billion in the European market alone (Alsop et al. 2020).

It, therefore, seems useful, as a preliminary step, to propose a mapping of the concerned domains. The mapping proposed in this chapter is matrix-based. First, it is divided into two branches, the branch where the action of algorithms replaces the judge and the branch where the action of algorithms supports the judge, each of them having its own justifications and operationality, and raising questions about legality, the legal system, public liberties and even efficiency. Finally, it is grouped in a transverse questioning that covers what could be called predictive or prophetic law: the data coming from both the results of the decision support and the positive actions of substitutions to the judge (for example, which automated decisions are more likely to be appealed) being themselves concatenated to permanently reformulate the perimeter and the distribution of the elements of the matrix (Fig. 14.1: The distinctive branches of AI use in the field of law and justice).

Fig. 14.1
A tabular image depicts the applications of artificial intelligence in law and justice. The image depicts the uses in two separate tables.

The distinctive branches of “AI uses” in the field of law and justice

Full size image

Today the substitution—at least the one perceived by the simple citizen—often covers only subjects of first instance law (parking fines, speeding tickets, tax or social deductions), but tomorrow we can imagine a tightening of practices limiting the appeal procedures or making them as complex as possible to avoid the occurrence of such disputes. The practical result would be the use of algorithms to distance the citizen from the law or from the exercise of his rights. The appeal of an automated decision is indeed quite complex, especially to access a human explaining clearly the procedures to follow (and even when he is trained to do so). In the Robodebt case, the Centrelink service at the origin of the scandal had such a long waiting time (more than 15 min online) to access a call center of untrained agents, that the only recourse for the citizens was a class action suit which still has a devastating impact both financially and in terms of image for the company, notwithstanding the intervention of many spin doctors to try to redress the situation (Towell 2017). Indeed, the proceedings, some of which are still ongoing, have so far cost the Australian state and the social security administration a total of A$1.8 billion.

The substitution of judges by other legal operators, such as arbitrators or mediators, called alternative disputes resolutions (ADR), is becoming increasingly common, in business or family law for example. There is thus a habit of differentiated practices that is taking root in society (Ferrand 2015).

At the same time, the interference of digital technologies in the legal process is not limited to the use of artificial intelligence (Ulenaers 2020). It is an older and broader issue (Stranieri et al. 1999) than the question underlying the intrusion of this new approach into the legal world or in the service of the legal ecosystem. Therefore, the issues raised are both non-specific and specific to AI. All the subjects to be analyzed below related to access to law, and to the respect of the adversarial process, are non-specific inasmuch as they relate to automation as to AI, whereas AI raises specific issues in the approaches related to network and data architectures, to the quality of the data and especially its neutrality. These new uses (deep learning, etc.) increase the paradox between the expectation of neutrality with regard to the judge's decision and the respect for public liberties.

14.1.1 Why Substitute Judges with Algorithmic/Automated Operations?

The substituting of judges with algorithms has been justified by the need to relieve overloaded courts of justice. Political actors on all sides have been trying to do this since access to the law was considered a fundamental right during the Enlightenment (Gray 2008).

Indeed, access to the judge to settle disputes with the constituted bodies (police, state, etc.) grew very quickly as people developed an expectation for a justice system that was autonomous from political actors (Bell et al. 2004), and that could be considered neutral. However, since the beginning of this century, justice, like the political-administrative ecosystem that is part of the rule of law, is in crisis (Stoett 2019). It suffers from a triple conflict between expectations, means and legitimacy (Fig. 14.2: The three tensions facing the judiciary).

Fig. 14.2
An infographic image depicts the three tensions facing the judiciary. It represents a triplet conflict between expectations, means, and legitimacy.

The three tensions facing the judiciary

Full size image

Expectations: if recourse to depenalization—i.e. the exclusion of certain disputes from the scope of the first instance judge—has had a definite effect on the volume of cases to be tried (we will come back to this in detail below), the contestation of the administration's actions and decisions in courts is at best stable, and at worst rising sharply, including in member countries where the rule of law is undermined.

Means: if the financial means are globally increasing, they remain quite scarce compared to the populations concerned: in this respect, Switzerland (a non-EU member state) is by far the European country that invests the most money per inhabitant in its judicial system. With an expenditure of €220.6 per inhabitant in 2018, well ahead of Luxembourg (€163.5) and Germany (€131.2). The three least well-endowed states within the European Union are Lithuania (€41.5 per inhabitant), Bulgaria (€42.3) and Romania (€42.6). All other EU members fall in between (European Commission 2021).

Legitimacy: the imperative of judicial neutrality (Ellis 2018) has been undermined on the one hand by the instrumentalization of the processes of appointing magistrates by politics, and on the other hand by the politically marked decisions that have quite naturally followed (Delsol 2021).

The impact of this probing of the legitimacy of decisions taken by judges has been reinforced by the relatively continuous questioning by whole segments of the European population—since the beginning of the COVID-19 crisis—of the legitimacy of elected decision-makers. When these actors have not demonstrated what Parviainen et al. (2021) have called “Epistemic Humility”, problems arise. Instead, they present to the public their sometimes biased certainties. “When the law obliterates itself at the top, everything becomes permitted,” Jean Pierre Delsol (2021) rightly tells us.

To a request for neutrality, the public actors answered by the automation of certain decision-making. This was based on the following presupposition: A decision made by an automated device would not be influenced by humans and would be in compliance with the law. This is true, but a decision that is legal at a given time does not imply that it is right or fair (Barocas and Hardt 2017), nor that this legality will remain over time. History is replete with the instrumentalization of the rule of law for political purposes—is it worth recalling the sinister legalism of the Nazis who made the Final Solution legal under the laws of the time and the country, of which Hanna Arendt has painted a vivid picture (Poizat 2017)? Robodebt, or the fines imposed on drivers who respect the law, in this case, the speed limits—due to a change in legislation not carried over in time in the machines—underlines the fragility of the assumption that the machine can't go wrong. Yet that assumption is still supported by many public actors in Europe (Pasquale 2019).

These are the first limits to legal automation: the hiatus between the legality of the sanction and the reality of the infringement. Just because an offence has been found by a machine does not mean that it is genuine or meaningful from a legal point of view. The context is just as important as the offence itself.

The appetite for neutrality appears therefore as one of the elements of the observed tendency to automate judicial proceedings. Other more fundamental intentions are added to this. It will be recalled that the European Union measures the legal performance of each member country according to three criteria (European Commission 2021):

  • Efficiency, which focuses on indicators such as the length of proceedings, the rate of variation of the stock of pending cases and the number of pending cases.

  • Quality, which highlights indicators related to accessibility, such as legal aid and court fees, training, budget, human resources, and digitization.

  • Independence, which focuses on indicators relating to the perception of the independence of justice by the general public and businesses, guarantees concerning judges and guarantees concerning the functioning of national prosecution offices.

14.2 Problems Raised by the Substitution of Automation for the Judge

Admittedly, as indicated in the latest iteration of the Council of Europe’s European Commission for the Efficiency of Justice (CEPEJ 2020a), automation makes it possible to reduce the response time to a legal problem, and thus the congestion of the courts, especially when they deal with only relatively low-value disputes. But this practice poses serious problems as regards the rule of law because:

  1. a.

    this is problematic in terms of respecting the principle of adversarial proceedings, which is a positive right recognized both by Article 47 of the Charter of Fundamental Rights of the European Union and by Article 6 of the European Convention on Human Rights.

  2. b.

    it requires the removal from the control of the judge (for the benefit of administrations often under the tutelage of the Executive) a whole part of the societal activity by decriminalizing certain facts (Croze 2017).

  3. c.

    it condemns more than it exonerates. The automation is systematically incriminating because it is only interested in the potential offender, but it can be wrong (we have seen this with the problems of speeding or parking tickets as well as Robodebt). The burden of proof is then on the potential offender, who must prove that he was not at fault, which is complex, since he may not have legal training (Maurel 2019). The wrongdoing is then converted into a sanction without measure or understanding of the ecosystem. As such, the process breaks the social dimension of justice by reducing it to a mere decision. Justice is fundamentally a process, and trials, as they unfold, contribute to the acceptance of sentences.

  4. d.

    it raises questions about access to the law and corrective procedures in the event of a challenge to the ultimate decision.

It should be recalled that the European Commission itself, in order to deny access to documents and elements of internal procedures, hides behind the notion of “space to think”. The legal contours here are vague, even though the principle is expressly condemned by the Court of Justice of the European Communities (Judgement of the General Court (Seventh Chamber, Extended Composition) of 22 March 2018 in case T-540/15 De Capitani / Parliament) on the grounds that an EU decision cannot be opposed to a citizen on the basis of elements to which he or she has not had access (Hillebrandt and Novak 2016). “It would violate an elementary principle of law to base a judicial decision on facts and documents of which the parties or any of them have not been able to take notice and on which they have therefore not been able to adopt a position” (Judgment of the Court of 22 March 1961.—Société nouvelle des usines de Pontlieue—Aciéries du Temple (S.N.U.P.A.T.) v High Authority of the European Coal and Steel Community.—Joined cases 42 and 49/59).

The principle of adversarial proceedings has been consecrated as a general principle of law by most of the supreme national jurisdictions in European countries (Ferrand 2000) and is included in Article 6 §1 of the European Convention for the Protection of Human Rights (CESDH). Even if this principle seems to apply to only the judge according to the drafting of the aforementioned Article 16, such is not the case since all the legal actors are also debtors of the respect of the contradictory. The European Court of Human Rights (ECHR) stated this in its Mantovanelli v. France judgment (ECHR, March 18, 1997, no. 21497/93). In this case, the applicants challenged the enforceability of the expert report on the grounds that the conclusions of the legal expert were based on elements that had been communicated to them without respecting the adversarial process.

The adversarial principle guarantees that the parties will not be judged without having been heard, or at least called. A person who has not been informed of the proceedings against him or her has certain guarantees, both in terms of the remedies available and the enforcement of the decision. The principle of adversarial proceedings guarantees each party the right to be informed of the arguments of fact, law and evidence upon which it will be judged. The various participants in the trial must therefore be loyal and diligent in communicating their documents and conclusions. Any element produced in court must be able to be debated and must therefore be communicated to the opponent.

The judge himself is required to respect the adversarial principle. For example, when he considers raising a legal argument of his own, he must give the parties the opportunity to explain this point, otherwise, he will not be able to use it in his decision. In the Mantovanelli v. France judgment (ECHR, March 18, 1997, no. 21497/93), the ECHR recalled that with the principle of adversarial proceedings, “the essential thing is that the parties are able to participate adequately in the proceedings before the ‘court’.“ None of this is possible in the case of automated decisions because the latter precede the debate, not follow it.

There is thus potentially a pronouncement of the sanction “in abstentia”, based on exogenous mechanical elements (the speed is x and such and such a driver has exceeded it by y) without the litigant having any means of ex-ante contradiction. The context of non-compliance with a law is not addressed: should a driver be convicted of speeding when he could potentially be transporting a dying COVID-19 patient to the hospital? A judge would admit the (so-called) mitigating circumstance and probably limit the impact of a conviction to the pedagogical understanding that not respecting road safety constitutes a danger. The machine will automatically condemn, and make the person pay a fine before allowing the launching of a possibly contradictory procedure. A deposit is often compulsory to contest the case, which restricts access to the justice system for the poorest people who do not have the means to block the funds or to mobilize the means to take legal action.

An automation process, or an AI that would respect this principle, has yet to be invented. Therefore, the solution currently found is the depenalization of certain situations. “Legal depenalization” covers two hypotheses. Firstly, it can consist of a downgrading of a given behaviour within the penal system: a crime becomes a misdemeanour; a misdemeanour becomes a contravention. The behaviour remains prohibited but punished less severely. Secondly, depenalization can also consist in taking a behaviour out of criminal law and into another legal sphere: civil law, administrative law, etc. (Jaafari 2016). The said conduct is still prohibited, but it will be sanctioned by another authority such as an alternate court circuit, or by an independent administrative authority such as the Competition Authority in France (financial sanction: Coulon 2008). Depenalization is frequently used to transform road traffic offences into simple contraventions (Fallery 2019).

Since the object is no longer the domain of the judge but of an administrative authority, not only does automation become possible but the nature of recourse to the judge is displaced: it is no longer a question of validating the existence or not of a fault but of arbitrating between two interpretations of a fact by two actors who are opposed to each other and whose relations are necessarily unbalanced (Ellis 2018). It is no longer the triptych ‘law—judge—citizen’ that meets but the triptych ‘citizen—administration—law’ arbitrated by the judge. The law is set by the administrative authority and the judge must only validate the adequate interpretation. The question put to the court is no longer, ‘is there fault therefore sanction?’, but, ‘is the sanction valid with regard to the offence?’.

In this sense, automation transforms profoundly and durably the balance of the social interactions between litigants and justice, at the risk perhaps of upsetting such interchanges. Indeed, in this automatic response, there is a break between cause and effect, and the person subject to trial who has not been confronted with justice will probably find it difficult to understand what he or she is accused of, and therefore to accept the sentence (Bathaee 2017). This is another element feeding the crisis of confidence that citizens have towards their institutions (Guggeis 2020).

In this context, artificial intelligence—and deep learning—is one more tool at the service of an already advanced process. It is not so much the process of automation that is being questioned, but how the architecture of AI and associated technologies impact existing automation tools by modifying them, even though the legal questions they raise have hardly been resolved.

This inquiry makes sense when we look at the second branch of our mapping of automation in the field of justice (Fig. 14.1): the support provided to the judge and to the judicial process.

14.3 Automation as a Support to the judge's Decision or as a Procedural Assistant

Another area where automation is nowadays omnipresent is judicial decision support. The issue here is far more critical, as it is not limited to justice alone, but also to investigative procedures; in other words, to law enforcement.

The Council of Europe has since 2018 deployed an ethical charter of good practice in the use of AI in legal proceedings and/or as an investigative tool. This doctrinal approach rather than positive law is to be regretted, as it singularly weakens the scope of this publication, which therefore has no binding value (Hyde 2019).

Legal proceedings, especially in criminal matters, call for much information from various sources and of various kinds. The role of AI in legal matters (Prakken and Sartor 2002) is to classify, link and integrate them without omitting any elements. With this support, the investigator and/or the magistrate can view, analyze and compare more documents in an increasingly voluminous and complex file.

AI is a complex set of means likely to endow computer systems with cognitive capacities in order to analyze data in large quantities, with statistical and probabilistic tools and with algorithms that classify, value and confront these sets.

Obviously, there are access limitations related to confidentiality, security or anonymization. Nevertheless, these data can naturally be coupled with other data, which may or may not be freely accessible, in order to constitute resource bases for end-users capable of linking, classifying, contextualizing and cross-referencing them. This allows finer information, with a better granularity, to be calculated or extrapolated than exists in their sources.

In this complex context, two “applications typologies” exist: expert systems and machine learning. In expert systems, the tool is taught by providing it with examples; in machine learning, the tool is asked to find, among the data it has access to, what is likely to be of interest. Expert systems are built on explanatory models to which parameters must be associated to establish one or several results. Machine learning algorithms are used in situations where a mechanistic approach not enhanced by the performance of algorithms is to be excluded.

Hence, machine learning will aim to develop a realistic correlation model between predictive variables (input data) and target variables (results). To do this, machine learning relies on a set of statistical tools and computer algorithms automating the construction of a prediction function (Singh et al. 2007).

An expert system is a tool capable of replicating the cognitive mechanisms of a human expert. For example, it could be software capable of answering questions, by reasoning from known facts and rules. It has 3 parts: a fact base, a rule base and an inference engine: a “thinking machine” (Ross 1933). The inference engine can use facts and rules to induce further facts until it finds the answer to the given “expert query”.

Machine learning uses an inductive approach to build a mathematical model from data, including many variables that are not known in advance. The parameters are configured as they are learned in a training phase, which uses training data sets to find and classify links. Designers chose different machine learning methods depending on the nature of the task at hand.

Expert systems rely on mathematically constructed logic and use deductive reasoning. As such, they are considered the first level of what constitutes Artificial Intelligence. Despite their performance in many fields, the choices deduced by the algorithms are difficult to explain in simple terms. This difficulty is increased by the paradigm shift introduced by deep learning. Thus, the choice of the learning technique used, the ethics of design, the absence of bias and the absolute necessity of being able to explain the arguments that allowed the tool to make a particular decision are all prohibitive elements in the initial construction of these support tools. This is where the problems arise (Whittlestone et al. 2019).

Several emblematic cases can be mentioned:

14.3.1 Easing Access to Law Enforcement and Filing

The deployment of automation and AI speeds up how files are processed, sorted, and considered, contributing to access to the law. Professionals see their workload reduced by tools like “telerecours” (facilitation of procedures reserved for legal professionals), while other instruments bring citizens closer to law enforcement authorities, notably through automated complaint tools (for example, with cybercrimes: https://www.europol.europa.eu/report-a-crime/report-cybercrime-online).

14.3.2 Solving Criminal Cases: Artificial Intelligence and Judicial Investigation

During an investigation, data are linked together with observations of a crime scene or misdemeanour. In a few years, this has gone from sketches to 3D extrapolations which include the statements of traces and various objects whose utility for a file is not necessarily established at the very beginning of the enquiry. We are in the middle of over-collection of declarations, testimonies, and hearings which often constitute the first step in the search for contradictions or verifications of facts. However, this collection is a prerequisite for the establishment of what is known as the index value, i.e. the interest and justification of the samples. If AI arrives to accomplish classification (and thus of operational enhancement through the construction of logical links), cross-checking and connecting facts and clues in the judicial files, the expectation is that the truth will be exposed in the trial. For this, various methodologies are used ranging from systematic automation (such as for recognition of people or objects), the establishment of models of existing relationships between information (so-called relational links), the detection of inconsistencies, for example between testimonies using semantic tools (Keyvanpour et al. 2011). All this information is made available to the investigator or the judge without any filter and often without him understanding its full power and scope. Of course, the demonstration of the truth implies authorizing a certain form of exploitation of mass data but ignores ethical questions about over-collection, post-processing, extensive cross-referencing, etc. If the matching work is well understood and structured, the question of false positives remains open. The balance between revealing the truth and considerable means of investigation must be raised. For if these practices are undeniably useful, they require an objective and strong framework. Otherwise, they risk becoming social scoring or the involvement of innocent third parties whose only fault will have been to be virtually identified as having a link with a person of interest.

In this sense, the data collected is central, but also, the data architecture, particularly about the links established. It is certainly necessary to control the algorithms, their level of transparency, but probably also the objectives given to them, because if not done, the power of AI could be misused. For example, tools could anticipate jury decisions by analyzing their profiles (especially in the United States, where contradictory opinions are expressed, including during jury selection). This differs from France, where juries are drawn by lot from citizens, and are used less and less in favour of professional judges. Another example could be the anticipation of judges’ decisions by analyzing their pronouncements. Depending on the analytical results, legal strategies may evolve where some litigants may request a change of jury or of location, while others will not have this possibility (Thagard 2004). Similarly, the use of AI attacks the integrity of the evidence by restricting the judge's choice over the validity of the expert opinions provided (Katz 2014). Such examples call for the greatest caution in weighing the contributions of this type of tool to the judicial world.

14.3.3 Predictive or Prophetic Justice

We must also mention areas that carry the most fear and fantasy about automation: predictive or prophetic justice (Queudot and Meurs 2018). This involves predicting the future of a legal action in order to anticipate it (which is sometimes presented as risk management) or to harmonise the scope of sanctions according to given offences/crimes. In addition to the ethical and fairness issues, experiments conducted in France and Europe (Aguzzi 2020) have so far shown reasoning biases leading to aberrant results. The use of algorithms for predictions could flout the principle of sentences in criminal matters and reproduce or reinforce inequalities.

This raises other issues. Firstly, the operation of existing algorithms remains in the hands of their designers due to business secrecy (Singh et al. 2021). Transparency about their design is highly limited, even though justice actors have embryonic expertise to evaluate them. Secondly, because algorithms ignore the possible interpretation of texts and the social context of judgements, the use of AI in justice entails a risk of standardising legal decisions.

14.4 Errors and Risks of the Interference of Artificial Intelligence in Legal Procedures: Integrating the Gap Between the Calculated World and the Sensible World

Meunier et al. (2019) propose an innovative approach extending to algorithms the phenomena of malfunctions affecting electrical systems, machines, software and interfaces (Berti-Equille et al. 2011). The aim is to identify the “bias zones” into which legal algorithms can fall, due to either structural errors or faulty assumptions. The goal is to detect when the algorithm fails either by going beyond the expectations of its designer or deviating from or misinterpreting its underlying assumptions. As AI learns from its own actions, it only takes one faulty element for the whole process to become biased. This could generate false positives which have little impact when it comes to not authorising a payment, but which, when applied to justice, can have otherwise serious effects.

With the progressive “datafication” of all human acts, data is at the heart of the problem. Are the algorithmic biases that often appear during the implementation of this type of process linked to the data collected, to the nature of the data collected, or to the methodologies for using the data? In other words, is it a question of the quality of the data, of the use of the data, or both?

This question was not born with AI. This goes back to the early days of cybernetics—the science of human government—which led people to believe that a society ‘steered and managed by machines’ would help humans avoid repeating the futile and bloody conflicts of history (Bateson 1972). This assumption ignored that behind the machines there were always humans, and they could only structure the machines according to their own conditioning or culture, reproducing or even reinforcing existing societal inequalities. Thus, many of the indicators that underpin medical decision-making and yet are used every day are—despite themselves—racialised (Cerdeña et al. 2020). There is no reason why justice and its automations should escape this problem (O'Neil 2016). AI gives greater resonance to these biases: it sorts and directs the choices of social actors, including judges, helps them make decisions, assigns indicators of recidivism, and supports recruitment processes. However, the points of vigilance that should be present at all levels of data collection, construction and interpretation are absent. Only the method of calculation and its efficiency or non-efficiency are addressed.

However, O'Neil, like other authors (Richard 2018), has faced much criticism, on the grounds that new algorithms would smooth out old biases until a form of quasi-automated ‘algorithmic fairness’ was achieved, with differences that would be measurable using dedicated indicators (Chadli et al. 2021). In other words, to improve technology, more technology is needed.

The question of bias is thus answered by calculated answers, even though the bodies of data that feed these systems remain imbued with pre-existing structural inequalities. Thus, algorithmic systems are trained on data that correspond to categorisations that have been the subject of very human choices. AI has not created labels like “individual at high risk of recidivism” or “having a cardiovascular risk”.

Ensuring a neutral reading of data does not guarantee its quality or accuracy. If the algorithm notices a bias in a body of data, it should refrain from using this source or correct the biased features, i.e. be able to look critically at its data source. Although data research has taken up the subject (Chereni et al. 2020), tools using AI for decision-making purposes are already being deployed on a large scale. Predictive policing, too, works with “manipulated” data because it relies on information about decades of past convictions without considering that these themselves may be the result of past societal biases (Noriega 2020). The same scenario can be found for algorithms used in the justice sector to detect the risk of recidivism (and thus possibly increase sentences). These are known to be largely unfavourable to certain populations. Even when smoothing out the first instance biases, Cyclic Redundant Coding (CRC) practices dedicated to the detection of errors in databases make it possible to recover traits by verifying other typologies of (socio-demographic) indicators that illustrate membership of given groups in the population. Information can be unintentionally modified during its transmission or storage in memory. Codes must therefore be used to detect or even correct errors due to these modifications. These codes cover more information than that strictly necessary to encode the information. To ‘m’ data bits, we add ‘k’ control bits. This means that n = m + k bits will be transmitted or stored in memory. These are known as redundant codes (Ntoutsi et al. 2020). Currently, some methodologies exist to support choices that are free from influences linked to the socio-historical context and economic imperatives. Indeed, even before tackling the technical part, ethical issues, such as the protection of people and their rights, must be integrated into the design of AI processes, as this will influence their architecture and the so-called “learning model” (supervised or unsupervised). In a supervised model, the training data is mastered by the designers who statistically structured the concerned dataset (this is called “data prep”) to avoid discrepancies (adequate population sizes and distributions). In an unsupervised model, the neural network itself detects bias. The network must learn to identify what is or can be a bias. The second important focus must be on the training dataset, the information used to teach the network how to behave. Usually, this is part of the global dataset used. If this dataset is biased (statistically not sound), the training will suffer the same failures. As such, a dataset including certain labels (race consideration for example) should not be used. This operation must be done with very special care at the three levels of data processing (pre -, in- and post), and all data that might introduce bias should be excluded in this very moment. Finally, a neural network is never fully trained. Periodic checks for bias in the results are important. AI tools can lose efficiency and performance over time, which, in the case of automated court decisions, could have serious societal impacts. Finally, and often forgotten, AI infrastructure is just as fundamental as the data itself. Collections made in real-time (e.g. stock market prices) can suffer from delays in their injection time into the processes, computers can break down, and, in the end, each grain of sand can bias the outputs.

The potential fallibility of AI’s predictability is precisely the reason for creating points of vigilance. The choices that prejudge the design of the tools deployed today actually raise questions that are more ethical than technical. These considerations can be reconsidered by abandoning the idea that society would be powerless in the face of AI’s opaque functioning. The tools currently deployed in the field of justice are far from neutral. To ensure the implementation of balanced or neutralised algorithms, three levels of intervention must be considered:

  • The data

  • The data integrity test

  • The calculation, i.e. the algorithm.

These three layers are well known and understood on an individual basis by deep learning domain specialists. We know that ethical AI can be deployed; the only obstacle is the societal will to do so.

14.5 Recommendations by Way of Conclusion

In applying AI to the judicial and law enforcement domains, the question of the promoters of these processes arises. In concrete terms, this amounts to questioning the decision-making power in the field, both in terms of the structure and the reasons inherent to deploying this type of tool.

In law and justice, most people point to their multiple advantages. LegalTech advocates keep repeating that by automating certain tasks, AI would lighten the daily lives of judges and lawyers by relieving them of certain administrative constraints. Judges could also rely on AI for decision support, while lawyers would refine their strategies thanks to the case analyses proposed by AI. As for litigants, the creation of databases from court decisions would give them access to a real legal library. They could see the cost of their proceedings fall due to simplified management, and the time taken to appear in court would decrease.

The problem is that while the costs of implementation are known, the benefits, both operational and financial, for stakeholders other than the promoters of these tools remain unclear (Tung 2019). For example, although the potential financial savings from these deployments are frequently discussed (Rigano 2019), statistics on the reality of these gains are sorely lacking. The cost/benefit trade-off should be reviewed in the light of transparent economic impact studies, as it is not certain that systematisation in this area would be genuinely efficient at optimising the resource use, notably public judicial funds. Regardless, if it seems that AI would improve support for the legal process, it is unclear whether it can be considered a tool for justice in the full sense of the word at present (Wachter et al. 2021).

Certainly, AI could have macroeconomic effects (Mateu and Pluchart 2019). Political and institutional support for LegalTech actors as a well-thought-out tool for international leadership can be agreed upon. This would be a continuation of the multi-secular struggle of legal systems around the world (Ogus 1999). But questions remain as to whether these reasons are of sufficient importance to the general interest to justify and continue supporting technical arbitrations that are known to be biased (Pasquale 2020).

By combining the recommendations of the EU strategy for AI (Act, A. I. EUR-Lex-52021PC0206, 2021), especially point 3.5 on fundamental rights, the institutional studies carried out on this subject (Spajosevic et al. 2020), the analyses of the Council of Europe (CEPEJ 2020b), and the visions of major specialists in the field (Pasquale 2020; O'Neil 2016), certain features of a broad response to the problems mentioned above are emerging. They can be summarised in four points, all of which are based on a logic of ‘guaranteed transparency’ of systems.

A—Knowledge of the production triptych: the promoter/financier, the designer, the developer

An AI is only a tool that will answer a given question. The more societal the question, the more neutral the AI must be in its learning to be accepted. This implies moderation operations that have yet to be deployed, specifically to avoid what is called “false positives” As a reminder, a false positive is the Bayesian result of making a decision in a two-choice situation that is declared positive when it is actually negative. The outcome may be the result of a hypothesis test, an automatic classification algorithm, or even an arbitrary choice. However, if false positives are tolerable in terms of payment by bank card (after all, it can prevent compulsive purchases), it is unacceptable in terms of justice because it attacks the very foundations of equality, and therefore the balance of our societies. Destroying the balance amounts to destroying all confidence in social systems that are already under attack from populists.

Each AI deployed in the legal field, and therefore likely to become a tool for generating evidence, should have its promoters and developers identified so that the results of this type of activity can be integrated into the adversarial process in the same way as other types of evidence. Without falling into any kind of naivety, calculated evidence should be challenged in the same way as observed or recorded evidence.

B—Knowledge of the essential characteristics of the algorithms

The nature of information processing (supervised or unsupervised learning), and of the associated processes of moderation, whether human or calculated (so-called ‘neutralising’ bias), should be made public if only to demonstrate their existence.

C—Knowledge of the sources, nature and architecture of the data used to train the algorithms

Whether data is structured or unstructured, and in particular the so-called data-prep processes, should be made clear. This would make it possible to test the quality of the data and therefore the reliability of the foundations of the AI.

D—Knowledge of uses to avoid unanticipated uses

Any technology can be misused. It is not a question, as some suggest, of issuing authorisations for use, but of knowing who uses this type of tool and why.

All this information could be deposited in a supra-national registry (perhaps carried out by the EU) in return for which applicants would be granted rights or certificates of use as suggested by the CEPEJ in its feasibility study (CEPEJ 2020c). Once this knowledge ecosystem has been established, the possible deployment and further uses of an algorithmic approach to justice can be imagined, ensuring that artificial intelligence is put to the service of all.