1 Introduction

Despite the hype surrounding Artificial Intelligence (AI) in recent years, many AI technologies have matured and are rapidly being adopted by organizations to fulfill their business goals, automate their procedures and improve their operational efficiency.Footnote 1, Footnote 2 Technologies such as machine learning, deep learning, and natural language processing are used by 48% of businesses to date, while 38% of medical providers in healthcare use automated diagnosis assistants.Footnote 3 The global AI market size is expected to grow from $93.53 billion in 2021 to $997.77 billion by 2028, predicting an increase of 40.2%.

Any AI artifact consists of algorithms, and algorithms are now ubiquitous. They have been integrated into our smartphone apps and services we use daily, as well as into the most complex decision-support systems that are actively or passively granting us access to opportunities (e.g., getting a loan approval). They can often be a part of a socio-technical system that has a direct influence on communities of people and individuals. The underlying premise of socio-technical thinking is that system design should be a process that takes into account both social (e.g., structures, roles, and rights) and technical factors that influence the functionality and usage of computer-based systems [2]. Many have characterized the opaque and proprietary nature of many modern algorithmic systems as “black boxes” [3]. In black-box systems, it is difficult or impossible to interpret and explain their inner algorithmic processes to justify their actions or output. Often enough, even developers cannot explain the inner workings of their creations to justify their algorithmic behavior.

Regardless of the benefits that such systems often bring to the table, the research community is concerned about their influence on humans’ cognitive behavior, actions, and overall effects. Unfortunately, a number of incidents have been reported through time, concerning algorithms’ faulty, undesirable, and/or discriminatory behavior with respect to individuals and protected groups of people. For instance, an incident involving the Google Photos application was widely reported in 2015 [4]. Google Photos is an app for organizing photos on users’ devices or the cloud. It provides functionality that automatically labels and categorizes photos using an image recognition service. In 2015, a Black software engineer’s photo depicting himself and a friend was labeled with the tag “gorillas”. Google publicly apologized and promised a solution. However, the solution, announced in 2018, which involved removing the offending tag from the database, was criticized as an “awkward workaround”.Footnote 4 Another incident was reported in 2015, which involved Amazon’s recruiting tool, used to screen candidates for hiring based on their resumes [5]. Reports stated the tool discriminated against women, having a preference for short-listing men.Footnote 5 In 2019, an investigation shed light on mortgage-approval algorithms, finding that the acceptance rates correlated to the respective individuals’ race.Footnote 6 According to the report, lenders were 80% more likely to reject Black applicants, 70% more likely to deny Native Americans, 50% more likely to turn down Asian/Pacific Islanders, and 40% more likely to reject Latino applicants, as compared to white applicants.Footnote 7 Finally, another serious incident was revealed in 2019 by a ProPublicaFootnote 8 study, in which the authors audited COMPAS, a risk assessment tool for scoring defendants in courts as posing a low, medium or high risk of re-offending, according to the respective crime category [6]. Surprisingly, the study found that the system perpetuated racial bias, as it tended to associate the defendants of color with higher risk scores, compared to white defendants, regardless of the severity of the crime. Consequently, Black individuals were often treated disproportionately as being likely re-offenders.

In view of such harmful incidents, the research community has been questioning the behavior of algorithmic processes [7]. As a result, an active area of research concerns the development of ways to evaluate the behavior of algorithmic processes. Many researchers have proposed auditing techniques, while others conducted audits on specific algorithmic components, systems, and services in context-specific domains. Auditing studies have brought the aforementioned concerns on algorithmic behavior to the surface, as they found evidence of their existence and influence on society.

Another important consideration is that the social biases of AI are often propagated to third-party applications, by developers and organizations, through the “democratization of AI”Footnote 9 and Software-as-a-Service (SaaS) models. In a 2018 study, Buoalmwini and Gebru [8] found that commercial algorithmic services that process images of people tend to have higher error rates in recognizing the gender of the depicted person on photos of darker-skinned individuals. In a similar vein, Kyriakou et. al. audited six proprietary image tagging algorithms (ITAs) of major providers. The authors found evidence of algorithmic gender and race bias perpetuated in those services [9]. In another study, Kyriakou et. al. found that proprietary emotion analysis services (EAS) perpetuate emotion-based stereotypes. More specifically, the authors found that some services were more likely to infer anger in photos of Black individuals in different cases, reflecting the psychological tendency to categorize ambiguous faces of Black individuals under emotions of hostility (e.g., anger) [10]. These are only a few examples of how bias can be propagated through third-party applications and end up affecting a concerning number of individuals. Beyond reports in the scientific literature, the public has also criticized many similar incidents in the press.Footnote 10, Footnote 11, Footnote 12

Obviously, we need systematic, technical procedures and regulations for monitoring algorithmic processes. Monitoring can be seen as the first step in ensuring that algorithmic behavior is in line with human values, and is thus positioned to benefit society as a whole. Of course, monitoring must be accompanied by coordinated efforts to prevent, mitigate and remediate any problematic behaviors. Researchers have suggested using human intelligence to interpret opaque machine behavior [11]. Thus, many have called for exercising human oversight over these algorithmic processes, especially the socio-technical ones, which are likely to impact human social relations. In its Ethics Guidelines for Trustworthy AI,Footnote 13 the European Commission (EC) refers to human agency, highlighting the protection of individual [users’] autonomy, which must be central to the system’s functionality. The key to this is a human’s right not to be subjected to a decision based solely on automated processing when this results in significant consequences for the user. Similarly, human oversight helps ensure that an AI system does not undermine human autonomy or cause other adverse effects. Oversight can be achieved using various governance mechanisms; in this work, we focus on oversight that involves human(s)-in-the-loop.

Currently, the literature is fragmented, with no common consensus on actions toward enabling systematic human oversight procedures. To date, there are only domain- or context-specific solutions for a portion of the algorithmic issues already identified. Even the term human oversight is not commonly agreed upon or used in the scientific literature in a uniform manner. Thus, we first need to clarify what we mean by human oversight and then develop a common framework for monitoring these algorithmic processes to make the systems fairer and safer. We envision a human oversight methodology that considers a set of generic algorithmic characteristics, which can be modified through human intervention, according to the domain and application of use. When implementing such approaches, we should consider leaving room for variations for different kinds of users, contexts and application scenarios so that the approach is expandable and effective.

To work towards this, one first needs to identify the requirements for successfully exercising a human oversight methodology over a target system. In this work, we aim to better define the requirements of human oversight, which invokes the Society-in-the-Loop (SITL) [1] as will be discussed in detail. Because the literature around human oversight and its requirements is diverse and multidisciplinary, originating from fields ranging from law and philosophy to computer science and its subareas, we identified five representative publications that reflect the requirements of such an approach, to construct a conceptual requirements model following a multidisciplinary perspective.

2 Foundations of SITL

Rahwan [1] proposed the concept of Society-in-the-Loop (SITL) as a means to regulate AI and algorithmic systems. He highlights the need for tools to program, debug and maintain an algorithmic social contract, a pact between human stakeholders, mediated by machines. Using the human-in-the-loop (HITL) concept as a base, he developed a SITL agenda, and defined it as a combination of the HITL control paradigm paired with mechanisms for negotiating the values of various stakeholders affected by AI systems, and monitoring compliance with the agreement. In short, “SITL = HITL + Social Contract”.

Based on the Social Contract term from political philosophy, Rahwan discussed how an ideal Algorithmic Social Contract would be. As he stated, SITL can be conceived as an attempt to embed the general will into an Algorithmic Social Contract. More specifically, under the SITL approach:

  1. 1.

    Society must resolve tradeoffs between the different values that AI can strive towards (e.g., different notions of security, privacy, or fairness).

  2. 2.

    Society must agree on which stakeholders would reap which benefits and pay which costs (e.g., what is acceptable or even which degree of collateral damage is acceptable).

It is not clear how to get from this conceptual approach described at a high level by Rahwan, to the actual practicalities needed to implement it. Thus, we aim to take some initial steps, establishing a starting ground for future research and relevant implementations.

Our work focuses on identifying the prerequisites needed for a successful application of a human oversight approach in black-box algorithmic processes. More specifically, these prerequisites will act as the facilitators and foundations for SITL. However, some of these will act also as a part of an Algorithmic Social Contract to enable a meaningful interaction between members of society, based on different values or agreements identified in the literature. Thus, our research question (RQ) is “What are the requirements for a successful application of a human oversight process?”. The requirements we refer to can be social, technical, or philosophical in nature (i.e., ethics, human rights, governance mechanisms, etc.). Ideally, we want to capture them all—or at least the majority of them—as this would be beneficial for establishing systematic, aligned human oversight processes on the respective requirements.

3 Methodology

Before explaining our methodology, it should be noted that in this work, we focus on providing a critical synthesis around the matter of human oversight. To do this, we used the first step of a systematic literature review (SLR) process, up to the point that resulted in a refined eligible pool of articles. During the second step in our methodology, we used the pool of articles to select five key publications from multidisciplinary domains to outline a first synthesis of the human oversight requirements. In other words, we do not provide an SLR analysis in this work, but rather, begin a discussion on the necessary requirements of a human oversight process from a multidisciplinary perspective. We view this work as a byproduct of an ongoing SLR study, a critical synthesis that can generate valuable insights and thoughtful considerations on this important and timely topic, to drive future work and discussions around it. Figure 1 presents the overall methodology and processes followed, which will be explained in the following subsections (Phase A-C).

Fig. 1
figure 1

Methodology of the critical synthesis process. The diagram presents the parts of the traditional systematic literature review (SLR) that have been utilized and how the byproduct of that SLR led to the critical synthesis

Full size image

3.1 Phase A: constructing a candidate pool of articles

We followed the PRISMA framework [12] to prepare a pool of articles around the area of human agency and oversight in algorithmic processes for an ongoing SLR on the topic. We used Harzing’s Publish or PerishFootnote 14 publication scraping tool to query Google Scholar for relevant publications on the topic. More specifically, we used a predefined set of queries for each of the research areas of interest, in investigating further such as the (a) Human Agency and Oversight, (b) Crowdsourcing, and (c) Algorithmic Fairness, Accountability, Transparency, and Explainability. We briefly present the queries by category in the list below:


Area: Human Agency and Oversight

  • Query 1: “human agency and oversight algorithms”

  • Query 2: “human agency and oversight AI”

  • Query 3: “algorithmic agency and oversight”

  • Query 4: “algorithmic behavior oversight”

Area: Crowdsourcing

  • Human-in-the-loop (HITL):

    • Query 5: “human in the loop AI”

    • Query 6: “human in the loop algorithms”

    • Query 7: “human in the loop oversight algorithms”

  • Society-in-the-loop (SITL):

    • Query 8: “society in the loop AI”

    • Query 9: “society in the loop algorithms”

    • Query 10: “society in the loop oversight algorithms”

  • Query 11: “crowd oversight in algorithms”

  • Query 12: “crowd oversight in algorithmic systems”

  • Query 13: “crowd oversight in AI”

  • Query 14: “crowdsourcing human agency and oversight”

  • Query 15: “crowdsourcing algorithmic behavior”

Using the scraping tool, we collected the top 100 most relevantFootnote 15 articles for each of the above queries, along with a set of meta-data and metrics provided by Google Scholar and the tool itself. The query execution was done in October 2021 for a ten-year period, from 2011 to 2021,Footnote 16 and excluded patents and citations. In this way, we examined the last decade of research on the topic of human agency and oversight in algorithmic processes. Then, we removed duplicates and applied a set of eligibility criteria to filter out the resulting article pool. Specifically, the papers must be:

  1. 1.

    Related to our areas of interest (a), (b) and (c). Reports, unpublished manuscripts, and abstracts are eligible for inclusion only when they are highly related to the topics of interest.

  2. 2.

    Accessible (Free or Paid).

  3. 3.

    Non-technical; with an exception to the technical studies that develop mechanisms to be used in black-box systems or services for human agency and oversight.

  4. 4.

    Anglophone works.

  5. 5.

    Empirical studies.

During the screening phase, we filtered the collected articles by reviewing their title, abstract, and keywords for relevancy based on our eligibility criteria. From the 1500 collected articles, we kept only the 550 articles that were found to be relevant during the first screening phase. Table 1, presents the actual articles excluded and kept per query.

Because of the high-multidisciplinary nature of the area on “Algorithmic Fairness, Accountability, Transparency, and Explainability”, we have chosen a number of essential systematic literature reviews for each domain of interest. Alternatively, an exhaustive investigation of this would probably produce significant noise. We list the selected publications here: [3, 7, 13,14,15,16,17,18,19,20,21,22,23]

Table 1 Number of collected articles per query based on the keywords
Full size table

3.2 Phase B: sampling of key multidisciplinary articles

As we observed, the topic of human oversight is being discussed within various scientific domains and from various points of view. Because of the inherent multidisciplinarity of the topic, we aimed to include a number of indicative, influential articles from as many domains as possible for the purposes of this paper. For this, we put a set of eligibility criteria to identify the key candidate articles for our critical synthesis, as presented below:

  1. 1.

    Five articles from multidisciplinary domains, forming a multidisciplinary pool of candidate articles.

  2. 2.

    Articles of high relativity to the matter (based on the judgment of the authors).

  3. 3.

    Articles of demonstrated high influence; more than 160% citation growth between the last two years (2022–2023).

  4. 4.

    Articles that implicitly or explicitly approach the matter from a theoretical or even philosophical viewpoint.

  5. 5.

    Articles focusing on common topics (i.e., such as the reflection of the human impact, consideration of the human involvement and judgment, and ethical delegation and responsibility).

  6. 6.

    Articles that reflect the considerations presented in relevant EU guidelines (e.g., Trustworthy AI, AI Act) regarding human agency and oversight.

The authors inspected the candidate pool of articles by using a random sampling method, assessing their title and abstract for relevancy, keeping in mind the synthesis eligibility criteria described above. Any relevant articles were appended to the pool based on the human judgment of the researcher, while the less relevant ones were excluded.

As a result, we identified five indicative scientific publications; two from the domain of Philosophy and three from multidisciplinary domains of a diverse pool of disciplines from social sciences, humanities and computing to fairness, accountability, transparency and ethics (FATE) in socio-technical systems and computer science. Table 2 presents the selected articles, along with their date of publication and the number of citations received.

3.2.1 Validation

The five articles have demonstrated high impact, having been cited frequently and exhibiting citation growth between 169 to 422%. More specifically, at the time of publication: [24] [CitationsFootnote 17: 1041 to 1759 (169% citation growth)], [25] [Citations: 690 to 1497 (213% citation growth)], [26] [Citations: 261 to 504 (193% citation growth)], [16] [Citations: 85 to 226 (266% citation growth)], and [23] [Citations: 9 to 38 (422% citation growth)]. This evidence provides validation for our methodology – which is prone to human error and understanding—in that the selected articles are of high importance.

3.3 Phase C: the review and synthesis process

Next, we provide further details on the examination process of the five influential articles of our SLR pool. The review process followed a three-step approach. During the first step, the first researcher/author read all five selected articles, took notes and marked any reflected requirements around human oversight presented and discussed, implicitly or explicitly by the article’s authors. Proceeding to the second step, the researcher sketched out a mapping based on the extracted human oversight requirements as a simplified component diagram, stating the extracted requirements along with the corresponding citations. In the third and final step, the researcher inspected the initial mappings in the simplified component diagram, observed if there were any common themes or consensus among the papers on the requirements, and grouped them into high-level concepts accordingly. Figure 2 presents the high-level requirements generated during the third step of the review process, and are exhaustively analyzed and critiqued throughout the paper.

3.3.1 Validation

A second researcher validated the corresponding mappings generated. Any thoughts and concerns were discussed with the first researcher who conducted the mapping during the review and synthesis procedure. Both researchers came into a consensus on the final mappings proposed.

Table 2 Selected publications for review
Full size table

4 Multidisciplinary perspectives

In this section, we briefly discuss the aims and contributions of each selected publication. This will help the reader understand each article’s context and aims, prior to the critical synthesis reporting.

4.1 The ethics of algorithms: mapping the debate

In this work, Mittelstadt et. al [24] focused on clarifying the ethical importance of algorithmic mediation as many tasks are rapidly delegated from humans to algorithms, forming decisions and final outcomes that might affect whole societies. In this line of work, they review the current discussion of ethical aspects of algorithms, providing also a perspective map to organize the debate. In addition, they assess the available literature in an effort to identify areas for improvement around the proper development of algorithmic ethics.

4.2 AI4People—an ethical framework for a good AI society: opportunities, risks, principles, and recommendations

In an EU initiative, Floridi et al [25] aim to lay the foundations for a “Good AI Society”. They present a set of core opportunities and risks of AI for society. Five ethical principles were synthesized for adoption during the development lifecycle of a system. Following this, they present 20 concrete recommendations - to assess, develop, incentivize and support good AI - for direct application from national or supranational policymakers as well as other stakeholders.

4.3 How AI can be a force for good

In this short philosophical paper, Taddeo et al. [26], discuss the potential of an ethical framework to aid in keeping humans in control of AI, and harnessing its power. Furthermore, the authors briefly discuss three areas of interest. First, they discuss delegation and responsibility, which concerns how much we delegate human tasks to AI, and how this strikes back by eroding our human supervision in a way that the unfair impacts of AI become a reality. Second, they discuss the invisibility and influence of AI; the way that AI is widely integrated into our daily reality, its influence on humans and its ability to manipulate human choice. The last area they briefly discuss is translational ethics, where ethical principles depending on different cultural contexts and domains are needed to be applied in these systems, along with a set of foresight methodologies to indicate and prevent any unwanted consequences.

4.4 What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability

This work is a systematic literature review conducted by Wieringa [16] around algorithmic accountability during a period of ten years between 2008 to 2018, consisting of 93 core articles. The author followed an interdisciplinary approach to form an overview of material from data sciences to law, and from computer science to governance studies. This work has three contributions: (1) it connects the accountability theory in the discussion of algorithmic accountability, (2) it provides a cross-sectoral overview of that same discussion viewed in light of accountability theory, with extra attention to accountability risks in algorithmic systems, and (3) it provides a definition of algorithmic accountability based on accountability theory and algorithmic accountability literature.

4.5 Requirements for trustworthy artificial intelligence—a review

A review from Kaur et al. [23] focuses on reviewing various proposed approaches for making algorithmic systems trustworthy. In this work, these approaches were summarized based on the EU principles for trustworthy AI. Furthermore, the review provides an overview of different principles that are important to make AI trustworthy. Their mapping presents four principles such as the principle of respect for human autonomy, the principle of prevention of harm, the principle of fairness, and the principle of explicability.

5 Human oversight requirements

In this section, we present the distillations of the critical synthesis of the selected articles. We detail the insights on the identified requirements for human oversight based on the authors’ findings and perceptions.

We observed that five core components make up the foundation of Human Oversight: Algorithmic Transparency, Explainability, Accountability, Trust and Law & Ethics. Figure 2 presents this foundation and the respective requirements as a set of components. The essential core components able to facilitate a human oversight process are referred to as “foundations”, and can often be perceived as having a bi-directional relation between each other. For instance, oversight should—by design—facilitate the accountability of an algorithmic process as a motivation, while on the other hand, when accountability is not properly defined, human oversight may not be applicable or effective by all means. These foundations, along with their requirements, must be in place to be able to apply a human oversight process properly. Next, we discuss each foundation in detail with its respective requirements.

Fig. 2
figure 2

Foundational components of human oversight requirements as reflected in the selected literature. Oversight can be described as a process that requires both some high-level and baseline foundations

Full size image

Algorithmic transparency. This foundation is often associated with two themes of requirements—Mechanisms and Conditions. The Mechanisms consist of various metrics for trustworthiness [25] and other relevant aspects such as biases (considering also the biases of the interpreter [24]) and fairness [23, 24]. There is a definite need for low- [25] and high-impact auditing mechanisms [24, 25] for monitoring and assessing the behavior of algorithmic systems while keeping them transparent. The operation openness [25], accessibility and comprehensibility [24] of the algorithmic processes are three Conditions reflected from the selected literature that have to be in place for a system to be considered transparent, in addition to the rest.

Trust. Often associated with themes such as the Public Factor, Perception Considerations, and Guidelines. The Public Factor is often referenced as the existence of the public’s trust [24, 25], acceptance and adoption [25] towards the algorithmic systems. To establish and maintain a trustworthy relationship with the stakeholders of a human oversight process, we identified three Perception Considerations that have to be met and clarified. First, the benefits of the system have to be seen as meaningful [25]. Similarly, the risks around the system functionality are meant to be seen as preventable, minimizable, or even as a potential [25] to leverage new opportunities. Last but not least, the condition of a system being harmless to humankind [23] is another important aspect of the whole equation of trust. As for the Guidelines, the authors highlight the need for clear requirements and standards by considering that different application domains follow different principles [23].

Algorithmic explainability. Often described by three themes such as the Conditions, Techniques, and Mechanisms. The authors explain that we need understandable [25] and interpretable [23] socio-technical systems, able to gather meaningful insight [16] using the proper explanations. Another suggestion on the explanations is that it would be beneficial to respect the temporal unfolding of an algorithmic system [16] to assess it in a fair manner. These are basically the Conditions we should consider from the algorithmic explainability perspective to hold the systems explainable. In the literature, it is stated that we need explainable AI techniques [23, 26] for exercising algorithmic explainability on systems. Some of these should act also as outcome prediction techniques [26] to assess the behavior of the system in the long run. As proposed in the literature, one of the mechanisms needed is a dedicated framework for enhancing explicability [23, 25].

Algorithmic accountability. This foundation is the second most frequently referenced in the selected literature. It is often described by five general themes such as Responsibility, Agency, Conditions, Mechanisms and the Modularity of a human oversight process. The modularity consists of the modularity of the socio-technical account and a set of considerations on the types of the decision-making process and accountability (that can be divided into actor-level and actor-forum relation respectively). Responsibility is usually discussed in the context of delegation [25, 26] in which the tasks to delegate to the system should be thoughtfully selected. The authors stated another form of responsibility, distributed responsibility (i.e., alternatively characterized as moral responsibility or distributed morality) [24,25,26], where multiple stakeholders along with the system hold different levels of responsibility on the final outcome. The authors discuss agency as a core consideration of the whole accountable process. They present a form they call distributed agency [26] and they emphasize the need of enhancing human agency [16, 24, 25] often using facilitating frameworks to aid this process [25]. Of course, there needs to be some conditions that have to be met again; an involved (in the accountable process) audience that would be able to (1) understand the subject matter and (2) engage with the material in a critical way [16]. On the same line, we would need to be able to measure this human involvement, which is also referenced in the literature as a mechanism [16]. There is a modularity theme emerging from their authors’ discussion. According to [16] the socio-technical account should not be seen as a checklist of everything, but rather as a modular frame that can help identify and ask the questions crucial in particular contexts. There are also a set of considerations for different types of the decision-making process such as prioritization, classification, association, filtering, randomization, and prediction [16]. The rest of the considerations revolve around the types of accountability. First, the authors [16] talk about the four types of accountability based on the level of an actor such as individual, hierarchical, collective, and corporate. Second, they talk about the three types of accountability based on the relation between the actor and the forum such as vertical, horizontal, and diagonal accountability.

Law and ethics. This is the most frequently referenced foundation in the selected literature. It is often described by seven general themes such as Data Considerations, Values, Ethical Principles, Regulation, Regulatory Bodies, Conditions, and Actions. To begin with, data governance, ownership, and privacy are the ones referenced in the literature as important data considerations during the system operation [26]. The authors, throughout their work, refer to a wide set of values a system should be aligned with. These values consist of human autonomy [24,25,26], human self-determination [25, 26], human self-realization [25], human dignity and flourishing [25], human rights [16, 25], and last but not least the equal and fair treatment of individuals (including protected groups or groups of interest) [23, 24]. Algorithmic systems might follow these values in tandem with a set of ethical principles too. The literature emphasizes that we need to define a set of proper and viable ethical principles for the algorithmic context, as in bioethics [23, 24, 26]. The authors of AI4People [25] propose a redefined set, based on bioethical principles, such as beneficence, non-maleficence, autonomy, justice, and explicability. Another important set of ethical principles is the principles determined by the national, cross-national, or international laws such as the EU Ethical Principles on AI [23]. The EU principles refer to respect for human autonomy, prevention of harm, fairness, and explicability, along with a set of key requirements such as human agency and oversight, technical robustness, privacy and data governance, transparency, diversity and fairness, societal well-being, and accountability. In addition to these, the systems have to consider the ethical sensitivity (of human agents) [16]. As a result, ethical guidance [16] needs to be defined accordingly. However, we need proper regulation and regulatory bodies to be formed and applied. From the regulation side, according to the authors, the world needs design policies and rules [25], self-regulatory codes of conduct (for data and AI-related professions, with specific ethical duties) [25], strict laws that can govern the working of AI systems [23] and new kinds of expertise from lawyers, judges, and legal practitioners [23]. On the side of regulation bodies, there needs to be established an EU Oversight Agency [25] for the EU-deployed algorithmic systems and a European Observatory for AI [25], along with specialized legal instruments and contractual templates for human–machine collaboration (i.e., in the work environment) [25]. Of course, there have to be met some conditions that can aid a successful, aligned-to-the-law and ethical human oversight process. First of all, we need to have widely accessible mechanisms of regulation and redress in place [25]. Second, we need financial incentivization at all levels [25] to move forward with fundamental changes in the way companies prioritize their law & ethics compliance (i.e., a European AI Adjustment Fund). A third point is to achieve a balance between coercing and supporting users’ decisional autonomy [24]. There are some actions the authors suggest, in light of the law & ethics foundation. One has to constantly evaluate the desirability of different courses of action that present conflicts between the involved parties [24]. Also, it has to be considered that the requirements for a moral agency may differ between humans and algorithms [24]. Lastly, it would be beneficial to consider the fact that diverse contexts may be ruled by different moral frameworks [24].

Oversight (in algorithmic processes). Oversight can be achieved when the aforementioned foundations described above are established in the context of use. In addition to these foundations, there are some general themes of requirements when it comes to human oversight in algorithmic processes that have to be aligned and in collaboration with the rest of the foundations. From this perspective, we observed six general themes referenced in the literature such as the Human Factor, Public Factor, Enabling Conditions, Facilitating Mechanisms, Timeframe Considerations, and Cooperation. The human factor basically revolves around human involvement [23] as the center of a system (human-centric) [23]. The authors agree that a system must be able to facilitate human supervision [23,24,25,26]. Some of them even describe the system as an enabler for maintaining the open human choice [25, 26] in the process flow. As stated in the literature, keeping humans in control (i.e., humans-in-the-loop, humans-on-the-loop) [23, 24] and increasing societal capabilities [25] might benefit the behavior monitoring of systems. In addition, meaningful oversight and human intervention capabilities are necessary ingredients for a stakeholder who exercises oversight of the system processes in order to adequately override the final decision [23, 24]. The theme of public factor is discussed in reference to public perception and understanding [16, 25]. Enabling conditions require robust procedures [26] and maintaining a trustworthy relationship [24] around the system. Furthermore, it needs information diversity [23] which can act as an enabler condition for autonomy [24]. The authors point out that exercising oversight requires targeting well-defined, comprehensible, and predictable systems, instead of complex and fluid ones [24]. There is a set of facilitating mechanisms reflected in the literature such as the existence of proper foresight methodologies [26], guiding principles [26], participatory mechanisms to ensure alignment with societal values and understanding of public opinion [16, 25], resource mechanisms for data subjects to question the validity of algorithmic decisions and prescribed norms for humans and algorithms to work in tandem [24] while also making the data subjects able to express their views and to “contest the decision” [16, 23, 24]. Considering the last point, the authors suggest that we also need mechanisms for enabling collaborative intelligence (i.e., Human–AI collaboration) [23], which can aid in empowering human cognitive ability instead of just replacing humans [23]. In any case, they highlight that context-specific solutions are needed [23]. When it comes to timeframe considerations, the authors suggest that exercising oversight throughout the system lifetime would be more beneficial [16, 23, 26]. To achieve a proper level of oversight, we would also need cooperation between various stakeholders of interest. This cooperation should be cross-disciplinary and cross-sectoral [23, 25]. As discussed, there needs to be a diverse multistakeholder group [23, 25] including also a multidisciplinary team of human experts [23].

6 Discussion

Next, we further expand and comment on the conceptual framework based on the extracted knowledge and conditions identified from the selected literature, answering the research question we put forward in Sect. 2. Further, we discuss a methodological approach, based on the foundations described above, as a way of monitoring the behavior of algorithmic processes.

The fundamental components of oversight are Transparency, Explainability, and Accountability. In parallel, these foundations have to be aligned with the requirements as described under Trust, Law & Ethics. When considering applying a newly formed human oversight approach to monitor the behavior of a target algorithmic process, one has to verify its compliance with the basic set of foundations to ensure its alignment with the current regulation prerequisites, ethical considerations, and social norms. To expand, we need transparent procedures in place, not only in the internal workings of algorithmic processes but in conjunction with having transparent human intervention (oversight) procedures to properly scrutinize algorithmic behavior at its core. Explainability can definitely aid humans who monitor a system in justifying its algorithmic outcome and predictions by leveraging well-known and emerging technical strategies, methodologies, and tools. Similarly, explainable human procedures can help delegate accountability in a fair manner. In that way, accountability should be in place, not only from the perspective of the system’s inner workings and behavior where accountability is often being delegated to the developers or organizations involved in their implementation, but also for the human oversight actions required to override cases of unwanted, biased or discriminatory algorithmic decisions.

Both algorithmic and human oversight procedures require reflection on their expert and public trustworthiness. Domain-specific solutions such as medical systems, for example, should not only have the corresponding experts’ (i.e., specialized doctors) acceptance, but also the public’s acceptance, to be adopted in a medical diagnosis or procedure. The benefits to the end-users or affected individuals have to be clearly stated, thoughtfully checked, and monitored regularly to increase trust in a meaningful way. Risks have to be addressed and mitigation strategies should be determined in advance. A human oversight approach can also aid in increasing the system’s trustworthiness when mitigation strategies are infused into the whole procedure, the harmful effects of any improper algorithmic behavior can be minimized or even prevented.

The legal domain still lacks appropriate laws, regulations, policies, and expertise when it comes to handling incidents revolving around improper algorithmic behavior. There is also a lack of prevention guidelines and procedures for enforcing human monitoring over algorithms by respective organizations. The selected literature presents a gap in expertise and governmental oversight bodies to keep those algorithmic processes accountable and aligned with various ethical norms. On the other hand, fundamental ethical norms are still not mature and commonly agreed upon yet, while others are mentioning various ethical values around human rights should be taken into consideration when developing or using algorithmic systems.

Towards a modular oversight methodology (MOM). A human oversight approach must be designed for monitoring the behavior of algorithmic processes while also making them more accountable, trustworthy, and fair. In this subsection, we describe a Modular Oversight Methodology (MOM) we envision. We also connect the authors’ requirements on human oversight, drawing a diode to MOM. In contrast to existing approaches to auditing system behavior (e.g., commercially available toolkits for mitigating bias), MOM offers a means to integrate humans into the process, who provide judgements on the behaviors from their own point of view. Fig. 3 illustrates a black-box system without an applied oversight methodology as of today. In an era where it is crucial to start addressing and dealing with improper algorithmic behavior, there is still a lot of ongoing work in this emerging area. On the contrary, Fig. 4 presents a simplified figure of our envisioned MOM, keeping a black-box system’s behavior fair using human involvement. Next, we briefly describe the key characteristics/requirements of MOM.

Human involvement. An oversight methodology should be human-centric on its own, involving humans in the process, regardless of the human-centricity of the system. Human(s)-in-the-Loop (HITL) can play an important role in checking the algorithmic behavior based on a set of laws and ethical guidelines applied in the respective domain of use. Furthermore, SITL will magnify the socially accepted behavior and expectations from the target algorithmic processes, aiding in scrutinizing the social behavior and effects of their use. It would be preferable if we can follow a SITL approach in which three groups are involved; (1) a (multidisciplinary) team of domain experts, (2) the public/society who do not relate to the effect of the target algorithmic behavior, and (3) a group of people related to or affected by the target algorithmic behavior. Thinking of the concept of having a Social Contract + Human(s)-in-the-Loop as Rawan suggested [1] to form a SITL approach, the Algorithmic Social Contract can act as a crucial asset consisting of the social norms, laws, and ethical guidelines (considering the notions of security, privacy, and fairness) to help HITL in the oversight process and in which the stakeholders agree on which benefits might sacrifice and the costs to pay according to the final decision. In addition, having humans-in-the-loop would clear out many accountability delegation issues while also increasing the system’s trustworthiness.

Modularity is vital. The literature suggests domain-specific and per-context oversight solutions according to the systems’ purpose and characteristics. Among other modular factors, we have to consider (1) the domain in which the algorithmic solution is created and applied, (2) the human actors or entities that interact with or get affected by the algorithmic solution, (3) the types of available applicable technological strategies (methodologies, mechanisms, and tools), and (4) the respective national and international, guidelines and laws which directly affect the domain or application of use. Let’s consider the COMPAS risk assessment tool as an example application. It is created and applied in the legal domain, being used in courts by legal instruments (expert human actors), and its decisions can affect the right of freedom of defendants (affected human actors). In such a system, one could use an oversight methodology that will follow specific auditing approaches of stressing the system prior to a defendant’s trial and/or during the time of the trial based on the defendant’s demographics and other necessary historical information for human supervision. Having a team of experts inspecting the audit results can ensure the systems’ compliance with the respective national and international, guidelines and laws (including applicable ethical considerations), while also in cooperation with another social group (e.g., citizens) can aid in checking its processes for socially improper or harmful algorithmic behavior. Modularity is also applicable to the Algorithmic social contract as parts of it might need per-context tweaks. For instance, there are countries in which different national laws apply compared to each other. Another example is the cultural norms might differ per country of the application use. Modularity is an aspect that requires a dynamic approach that considers various parameters during an oversight process.

Diversity. An emerging aspect that can play a role in different phases of a human oversight procedure is diversity, and the selected literature reflects this need in various ways. Currently, a terminology of diversity can be seen in regard to data or the information gathered and used by algorithmic processes. Similarly though, the same can happen with an oversight process; when people have diverse perceptions and views on the matter/subject - sometimes even contradicting each other - would be more beneficial [1]. When it comes to perceived fairness, research has shown that people perceive “fairness” in different ways [27]. Having SITL describe the social norms in a diverse and meaningful way, would be the heart of such an oversight process. Besides, diversity can also be found in the pool of technical strategies (mechanisms, tools, and methodologies) available for inspecting the target algorithmic processes’ behavior in the specific domain and application context of use. We need diversity in nearly every phase of the oversight process to ensure the validity of our human supervision judgment, decisions, and interventions. Diversity would also act as a facilitator for all the voices to be heard in both societal and scientific (or expert) contexts.

Timeframe. MOM should integrate human oversight over system processes at any time required, as the authors of the selected literature suggested. We identify three basic timeframes in which various actors should be responsible for keeping the algorithmic behavior proper. We position these timeframes on the development phase of the system such as the pre-development, development, and post-development phase. The first, pre-development, is referring to the design of the system user requirements, components, and algorithmic processes. Here, system designers, developers/system analysts, and other design stakeholders are responsible for exercising human supervision of the design artifacts they build according to the Algorithmic Social Contract they agreed upon before this process begins. The system structure and goals should be designed according to the social norms, laws and ethical guidelines apply to the system context. Next, we have the development phase, where developers, production organizations, and end-users are responsible to keep the ongoing system implementation aligned (again) with the Algorithmic Social Contract defined. Any changes to the behavior during the implementation time should be scrutinized for any possible unwanted consequences or harmful effects. Such oversight methodology can be part of an agile development approach. The final, post-development or post-production phase, is a characteristic ongoing and dynamic phase referring to the system after is getting deployed and during its use in the application domain. In this case, the support team of the development team/organization or the individual/organization that uses the system should keep exercising human oversight using MOM to ensure that the system won’t evolve in such ways that can alter its behaviors and start posing harmful effects to the table.

To sum up the observations from the extracted requirements, MOM needs five basic components. It needs (1) to focus on target algorithmic processes (of a black-box system) in a specific context, it also needs (2) clear identification of the input and the corresponding algorithmic output; (3) it needs a human supervisor who can follow and apply the (4) proper guidelines (accompanied by an Algorithmic Social Contract) using a support (5) toolbox which includes the available technical strategies (mechanisms, tools, and methodologies) for exercising oversight over the behavior of system processes (e.g., applying a set of well-known auditing practices). The overall process should be characterized as semi-automated, avoiding manual time–costly solutions or automated ones that might generate more obstacles/issues based on their objective, non-subjective and generalizable nature. The methodology should be able to get tweaked according to the needs of the oversight process, the system, and business goals, the target domain, legal and ethical guidelines, and more, considering the overall oversight process as a set of modular components that cooperate together to achieve a legitimate result.

Fig. 3
figure 3

A black-box decision support system without human oversight (i.e., without MOM)

Full size image
Fig. 4
figure 4

A black-box decision support system with human oversight (i.e., using MOM)

Full size image

7 Limitations and future work

In this paper, we presented a short review of the requirements for a successful application of a human oversight process. Our discussion reflects only a snapshot of the selected literature gathered from our SLR pool of articles, using a critical synthesis approach. Specifically, we present only what is stated in the selected literature and discuss generalized foundations (and their reflected themes) that can be the first step towards a complete human oversight requirements framework. This is not an empirical work, but rather a conceptual work. Further work is needed to enhance the proposed extracted foundational concept to make it more robust by conducting a larger-scale and systematic literature review.

Furthermore, our work does not aim to provide an implemented solution on the matter, but rather focuses on beginning a discussion around the requirements for a successful application of human oversight over algorithmic behavior and, in parallel, we aim shedding a light for future research towards a MOM to facilitate this kind of process. Future work will be needed to carefully define a systematic human oversight process or framework by considering all the necessary prerequisites at the given point in time as reflected in the literature.

Moreover, we do not focus on the micro-level elements and AI design, nor we suggest any changes on improving tech from a design or HCI perspective. Again, through this work, we aim to begin a conversation on the requirements of a successful facilitation of human oversight through a MOM to assess the behavior of algorithmic processes. We aim to help by putting the first step towards an upcoming research on the matter.

Another important obstacle is the fact that the regulation and guidelines proposed by key entities, such as the EU, often provide crucial considerations for a noble vision (e.g., Trustworhty AI), but that are difficult or sometimes even impractical to implement. This is another domain of research where, again, researchers and experts from multidisciplinary domains (such as Laws, Philosophy, Computer Science, etc.) have to address those obstacles and come to a consensus on practical solutions. In the light of the EU AI Act that is anticipated by the end of 2023, it is worth mentioning that we are still not ready yet, because when it comes down to the practical considerations underpinning its implementation, there are still many open questions. For example, what should we consider to be “meaningful human control”, when is it needed and, practically, how should it be facilitated?

Last but not least, as with any critical analysis and synthesis paper, this work is prone to the human error of understanding, interpreting and other forms of the biases of the authors and/or the researcher who applied the methodology and crafted the corresponding mappings when reviewing the selected literature. An integrated SLR on the topic is required to provide more rigor conclusions on the proposed mappings.

8 Conclusion

We reviewed five influential publications from different disciplines to identify the requirements underpinning the successful human oversight of algorithmic processes. We analyzed a number of requirements reflected in the selected literature, many of which were commonly cited across sources. We grouped the requirements into six areas of interest, which we called foundations, constructing a conceptual mapping of the foundations and their associated themes. First, we explain those themes in detail, as reported by the authors. Later, we synthesize the literature, relating the identified requirements/foundations to the SITL approach, and discussing how they can be incorporated into an oversight methodology. This leads us to present a conceptual framework for a Modular Oversight Methodology (MOM), which can aid in monitoring algorithmic behavior. As we explain, future work is needed to enhance our proposed concept and methodological approach. In other words, this is a first step towards identifying the requirements around human oversight by considering diverse, multidisciplinary perspectives.