1 China’s Foray into legal AI

In 2017, China’s State Council published the Artificial Intelligence Development Plan (AIDP), a national strategy white paper that charts China’s AI development aspirations up to 2030. While the Chinese government had released a number of documents that touch on AI in prior years, the AIDP was the first that specifically focused on outlining a broad AI strategy that highlighted the state’s long-term intent to make China a center for AI innovation by 2030, and to leverage artificial intelligence in new forms of ‘e-governance’ [1].

In particular, the AIDP outlines China’s ambition to incorporate AI for ‘social’ and ‘moral’ governance. This includes use cases, such as reforming the nation’s welfare system, addressing negative externalities for environmental protection, building the oft discussed ‘Social Credit’ system, and modernizing the judicial system. On this last area, the AIDP contains a brief paragraph on the capacities envisioned for future ‘Smart Courts’:

“Construct a set of trial, personnel, data applications, judicial disclosure, and dynamic monitoring into an integrated court data platform. Promote AI applications for applications including evidence collection, case analysis, and legal document reading and analysis. Achieve the intelligentization of courts and trial systems and trial capacity” [1].

It is worth noting that AI and the law is one of the oldest interdisciplinary fields in the study of AI. As early as the 1970s, there was exploration at leading American universities on the possibility of AI use for legal research, argument construction, and expert systems, with robust research in the field continuing today [2]. However, China’s current policy to modernize its judicial institutions and use AI, big data, and algorithmic adjudication into broad application by state institutions is pushing into new territory. As of 2022, the courts and judicial processes of most countries have been largely untouched by AI and other disruptive technologies, with France outright prohibiting any development of predictive litigation AI in 2019 [3]. China’s motives to foray ahead into uncharted space can be traced to factors specific to China’s needs.

China’s judicial reform and interest in leveraging AI is motivated by a chronic shortage of judges, a weak public image, and a pressing need to modernize [4]. In the last four decades, China’s economic growth and the emphasis on the rule of law have led to an exponential increase in the number of court cases. However, the roughly 30-fold increase in caseload since 1978 is matched only by a threefold increase in judges as of 2015 [4]. Complicating the chronic staffing issues is the parallel need to improve the judiciary’s legitimacy and public image through professionalization of court personnel. In 2014, the Supreme People’s Court (SPC) introduced stricter internal evaluations and set a quota that limited the number of court personnel authorized to hear disputes to 39 percent, with existing judges that failed the examinations reassigned to administrative and support roles in the court [5]. As of 2017, this policy has resulted in a 49 percent decrease in the number of judges from over 200 thousand to around 120 thousand [5]. Consequently, while the “embrace of legality” and the “professionalization of the judiciary” are not inherently contradictory goals, China’s present situation is one of increasing caseloads and decreasing judicial capacity [5]. It is in this context that artificial intelligence solutions aimed at vastly improving court efficiency while achieving judicial modernization has become top priority in the SPC’s reform agenda.

These efforts follow a top–down approach, led by the Supreme People’s Court (SPC) and aligned with the general direction of the Party’s policy themes. Adopting AI into the judicial system is seen as a means for the SPC to create greater institutional oversight over “lower level institutions,” to improve legitimacy and efficiency for court staff, to alleviate staffing strain due to limited judges, and to encourage better public access to information [6]. Though guided from the top, these broad directives leave space for local courts to experiment and innovate via small scale pilot programs. If deemed successful, local pilot programs may be expanded to provincial or national scale.

2 Current AI developments

In the five years since the publication of the AIDP in 2017, both China’s courts and the country’s technology sector have made significant inroads towards the use and development of AI. A patchwork of pilot programs in many major cities have resulted in a showcase of different platforms and use cases for AI, with some successful programs expanded to provincial or national adoption. The systems implemented in these early examples of the ‘Smart Courts’ envisioned by the AIDP fall in two main usage categories: AI clerical assistive systems, and AI-based recommendation systems.

On the former, the purpose of assistive systems is to increase court efficiency - both in terms of reducing working hours per case, and in terms of rendering accurate and correct judgements. Developments that fall into this category range from open access online databases supported by a nationwide effort to digitize and publicize court decisions, as well as courtroom systems directly supporting trial procedure.

In 2014, the SPC elevated ‘openness’ and ‘judicial transparency’ as key policy objectives in light of a number of wrongly-decided cases [6]. The government-run platform ‘China Judgements Online’ (wenshu.court.gov.cn) now hosts over 130 million public documents at time of writing. Additional efforts to attach accurate metadata to these cases create a significant source of consumable data. Platforms, such as the ‘Similar Case Intelligent Recommendation System’ and the ‘China Justice Big Data Service Platform,’ allow both judicial staff and members of the public to search through past decisions, leveraging metadata to find similar situations and past decisions that can help inform potential outcomes [5].

Inside the courtroom, advancements in perceptual technologies such as optical character recognition, automatic speech recognition, and natural language processing, have been leveraged to produce automated transcriptions, reducing average trial times by 30 percent and minimizing the manual work performed by court clerks [7]. More advanced systems such as the ‘206 System’ in Shanghai also possess cognitive features, relying on machine learning through past cases and digital renditions of the legal code to make determinations that include detecting inconsistencies in evidence, validating requirements for sentencing, and recommending sentencing severity based on past decisions [7]. As of writing, the automatic speech recognition technology provided by iFlytek has been implemented in over 4,200 courtrooms nationwide [4].

Beyond these assistive systems, more publicity has focused on purported AI judges that directly assist in rendering court decisions. In Hangzhou, the ‘Xiao Zhi’ robot judge has been used to adjudicate a private lending dispute, helping the human judge conclude the case in under 30 minutes [8]. ‘Xiao Zhi’ is able to assist judges in real time with live summarization of arguments, evaluation of evidence, and award recommendation [8]. However, It is important to emphasize that at the time of writing, while there are some AI judge programs in pilot testing, these are under close human judge supervision, and no court decisions are implemented without human approval. The stance of Zhou Qiang, the current Chief Justice and President of the Supreme People’s Court of China, is that “AI will never replace human judges and can only serve judges [as assistants]” [8]. Given the rapid development and adoption of AI into China’s legal system, and the apparent conviction by both the SPC and the broader state in leveraging AI in all parts of government, the remainder of this article aims to paint in broad strokes some of key considerations, implications, and recommendations as China commits to technology-driven modernization.

3 Key considerations

The use of AI in China’s court system stems from its unique circumstance and pressures. This trend has been accelerated due to the COVID-19 pandemic, where the majority of the trials have been moved online [8]. In this environment, AI functions supporting court processes, such as evidence submission and trial record translation, have increased the efficiency of the courts. However, the rapid deployment of AI legal systems and robot judges programs combined with the additional stress testing caused by the COVID-19 pandemic have revealed challenges and potential difficulties with an AI-power judiciary.

The most pressing challenges consist of the potential to overestimate the efficacy and usability of AI systems, coupled with the potential of overreliance and behavioral changes with the introduction of increasingly comprehensive tools.

Currently, some AI functions are simply not developed enough for actual use. For example, current pilot programs testing facial and emotional recognition technology for validating testimony credibility are precarious and technologically unfeasible in the present time [4]. Using AI to detect deceit has also proven to be unreliable [4]. It is critical that while these systems undergo trials that their results do not directly influence case decisions unless validated by existing practices. Another weakness that has been highlighted is the uneven availability of digitized and available case data. A closer examination of China’s judicial disclosure efforts shows a fairly uneven variation across provinces and case types, with higher compliance in wealthier coastal regions, and higher disclosure rates for criminal cases over civil and administrative cases [6]. While most AI assistive programs have also been implemented in these wealthier provinces, the lack of readily available case data in other provinces weakens the legitimacy and usability of AI systems dependent on prior case machine learning. This challenge is already evident in user feedback of existing ‘intelligent recommendation systems.’ Judges in Sichuan and Jiangsu provinces have stated that the software “rarely succeeds at accurately matching the kinds of complicated cases where judges would most welcome guidance” [6]. A number of Chinese legal scholars have also voiced concerns about building algorithms on an incomplete public record [6].

The second challenge is minimizing potential overreliance as well as negative behavioral changes with the inclusion of AI systems in the judiciary. There is concern that certain design elements promote decision conformity and ‘rubber stamp’ courts [7]. One feature present in some of these systems is a means to score how much a present case decision aligns with past decisions. This form of algorithmic monitoring may lead judges to avoid “decisions that stray from the mean” [6]. Furthermore, studies from Israel, where courts have implemented real-time judicial monitoring, have found that judges “resented the shift to a production-line mentality” [6]. Even if the conception of these systems aimed to alleviate workload and reduce job difficulties, a sense of lost autonomy may induce additional dissatisfaction among an already strained judicial workforce. However, a more concerning issue that arises from both overestimation and overreliance is the potential for key elements or unique factors within individual cases to be missed, underweighted, or ignored amid aggregate outcomes provided by recommendation systems [5]. This may cause what Gökçe Günel refers to as ‘technocratic dictatorship’ which may ultimately bring injustice [9]. An adjacent concern that is applicable to all AI design is the pitfall of ‘engineered inequality’ [10]. Current AI systems learn from big data which may have underlying biases that result in systems that replicate and reinforce prejudice. The engineers and firms that develop legal AI systems usually lack significant interdisciplinary knowledge given the relatively nascent status of the field. They also have different values, interests, and stakeholders relative to the state, the judiciary, and general civil society. To mitigate engineered inequality, human judges remain primary decision-makers and are active participants in the development of Chinese AI systems. Just as law professor Wang Zhuhao stated, “The ‘intelligence’ behind the intelligent court project must ultimately be the crystallization of Chinese judges’ intelligence, not that of any IT software or program technician or company” [4].

4 Our recommendations

In light of these challenges, many Chinese legal scholars have urged caution and have “tried to ‘pour cold water’ on AI fervor” [6]. Nevertheless, China appears committed to embed AI into the judiciary, and optimism towards the technology is still high. In light of this trend, our recommendations aim to provide guidance to policy makers and system designers to hold future AI systems accountable to not only the courts, but also to the citizens it serves.

4.1 Effectively auditing algorithms

AI auditing will become a crucial supplement to the fast-developing AI industry. Current AI system development requires a neutral and impartial third party to audit its designs and algorithms to ensure transparency, accountability, and mitigation of human biases that may bring inequality to social justice. Currently, many private sector organizations developed AI systems do not disclose how various factors are weighed in their algorithms, in part due to the ‘black box’ nature of machine learning, but also due to competitive interests [11]. Nevertheless, the present opaqueness increases the difficulty for judges and others to understand and accept AI recommendations [11]. Organizations such as ForHumanity are developing independent audit frameworks for AI systems, and offer opportunities to learn how to build trustworthy AI infrastructure. We believe specialized AI auditors will be a forward-looking role that will be in high demand in coming years. We expect that the role will be analogous to Certified Information System Auditors (CISA) in the cybersecurity field. However, given the reach of AI systems and their direct impact on state-society relations, these AI auditors will have a much larger burden than current cybersecurity professionals. To be impartial and objective in their audits, it is critical that these new auditors and the firms that employ them are trained to reduce individual bias and are institutionally insulated from external incentives and pressures from other players in the ecosystem, including the state, the technology providers, and media narratives. Consequently, the ideal structure for these auditing agencies are NGOs that can maintain some degree of distance from the government agencies. They are responsible for auditing, while reducing market incentives for profit that may arise from relationships with private technology firms.

4.2 Decision-makers need to be accountable

China has outsourced the development of AI legal systems to many private sector firms, who are the main contributors of design and production of AI systems. Although innovation is often more efficient through private sector agility and resources, it is important to maintain the right balance between government and technology firms. The issue of outsized influence by private technology firms is a challenge facing many national governments. A universal challenge in these relationships is the imbalance of technical expertise and intelligence housed within cutting-edge technology firms compared to government officials. Consequently, even in best case scenarios, state agencies often fail to capture the capabilities and features they want at project initiation, either overestimating the abilities of present technologies, or being ignorant of available tools. At the same time, technical experts can often miss the nuances and complexities of civil society, especially when it comes to the reliance of data to explain reality. In worst case scenarios, technology companies can abuse the monopsony relationship they have with the state. An over dependence on private sector services without internally developed alternatives can weaken state institutions and make it difficult to decouple or contain private firms. For legal AI, decision-makers in Chinese courts should take the lead in the development process to ensure that the system would benefit social welfare. Projects and pilot programs should be initiated by government requisitions and need, as opposed to firm led marketing campaigns that aim to secure lucrative government contracts. In this context, the Chinese Supreme Court is developing its own engineering corps to partner with their private sector counterparts and master the core technology used in the court system to ensure its accountability [12]. An early example of this working partnership can be found in ‘Little Judge Bao,’ a privately developed AI system sentencing prediction [11]. According to their website, the development team limited the system’s inputs only to factors actively compiled in official court sentencing guidelines, adhering to the principle that only “courts should determine which factors should be considered in sentencing and which factors should consequently be included in AI algorithms” [11].

4.3 Rethink foundational design

Amid the rush to develop AI systems that demonstrate novel features and perform well in pilot showcases, designers must consider issues of inclusivity and avoid ‘coded inequality’ within these systems. One specific area that needs additional scrutiny is the underlying data that are being used to train and power these judicial platforms. While China’s overall efforts to digitize court data have yielded impressive aggregate figures, the reality of patchwork compliance and disproportionate unavailability of certain case types over others among China’s provincial and local courts creates some levels of uncertainty over the integrity and hygiene of the data. Furthermore, designers will need to consider sustainable solutions that account for how these systems intake data and how they are updated. Between 2012 and 2022, 68 new laws were issued and 234 laws were amended in China [13]. Along with the updates to the Civil Code of the People’s Republic of China, many laws were revoked and replaced, including the Marriage Law and the Law of Succession [13]. How AI learns from cases that were applied to previous laws and if they can adapt to newly issued laws or judicial explanations to make future judgements will be key to the sustainability of its design. Additional complexities in the Chinese judicial system include the use of “hard actions” [11]. These are periods when the judiciary enforces harsher sentences for certain types of crimes in accordance with initiatives from the central government [11]. Whether these special judgments should persist in the Chinese system is outside the scope of this paper, but current cases that have been decided under “hard actions” periods are in the databases that AI systems use and may lead to skewed sentencing recommendations. Decision-makers should consider China’s particular history, policies and court judgements in special times during the development of AI systems. Moreover, how will cases that have been concluded with the assistance of AI systems be used in future machine learning schemes? Furthermore, there will need to be serious discussion on the ‘black box’ nature of AI algorithms and how that may impact judicial transparency. Ensuring that there is integrity in the foundational data used to build legal AI and that these systems remain sustainable without compromising the validity of source data is a baseline requirement for ‘design justice,’ and is the first step of ensuring the accountability of AI system development.

4.4 Engage partners with different social backgrounds

The development of the AI legal system needs to include people from different social and cultural backgrounds. Due to the complexity and technical nature of AI-based technologies, the perspectives of judges, legal scholars, social scientists, anthropologists, and the general public must be actively included to ensure different values and diversified perspectives are heard to make the system inclusive and beneficial to the entire society. Events organized by international organizations including the UN’s Science, Technology and Innovation (STI) forum as well as the World Bank’s Annual Technology Workshop engage participants from different industries, academia, and civil society to share their innovative technology solutions and research for sustainable development and social good. Such events or forums should also be organized for the discussion on AI legal systems at both the international and local level, and established as a requirement in their design process. Engaging stakeholders with various backgrounds and differing interests is a key safeguard against data bias and keeps systems confined within a scope that matches their actual capabilities. In a broader sense, this kind of dialogue will also avoid ‘technocratic dictatorship’ which “allows technocrats to make decisions for global collectives” [9]. Regardless of whether a judicial system predominantly leans towards a civil law or common law framework, many theorists claim that the legitimacy of the judiciary hinges not on strict adherence to some definition of fairness, but on whether the public and the participants of the process perceive the system to be fair [11]. Therefore, engaging more stakeholders will help gain public support on using AI for justice and enhance their confidence in fairness and trust in ‘Smart Courts.’

5 Conclusion

These four points by no means encompass all the areas that need serious deliberation when designing for legal AI. It is clear from the AIDP that China has made a commitment to leverage AI in many areas of government, including its judicial system. This article highlights some of the driving forces behind the decision to invest in non-human assistive systems. At the same time, we hope to show that the potential these technologies may provide can be marred by missteps and oversights if rushed. The concerns that have already been voiced on these issues must be translated into practice, with adequate checks in place. Furthermore, it is critical for Chinese policymakers and practitioners, as well as literature on the subject, to be able to distinguish policy ideals and promotional claims from the realities on the ground. Without being clear-eyed, misplaced trust in underdeveloped technologies can have real and negative consequences for the individuals involved, and for the rule of law as a whole.