Beyond Data

Abstract

In a technology context dominated by data-intensive AI systems, the consequences of data processing are no longer restricted to the well-known privacy and data protection issues but encompass prejudices against a broader array of fundamental rights. Moreover, the tension between the extensive use of these systems, on the one hand, and the growing demand for ethically and socially responsible data use on the other, reveals the lack of a framework that can fully address the societal issues raised by AI.

Against this background, neither traditional data protection impact assessment models nor the broader social or ethical impact assessment procedures appear to provide an adequate answer to the challenges of our algorithmic society. In contrast, a human rights-centred assessment may offer a better answer to the demand for a more comprehensive assessment, including not only data protection, but also the effects of data use on other fundamental rights and freedoms.

Given the changes to society brought by technology and datafication, when applied to the field of AI the Human Rights Impact Assessment must then be enriched to consider ethical and societal issues, evolving into a more holistic Human Rights, Ethical and Social Impact Assessment (HRESIA), whose rationale and key elements are outlined in this chapter.

1.1 Introduction

All AI applications rely on large datasets, to create algorithmic models, to train them, to run them over huge amounts of collected information and extract inferences, correlations, and new information for decision-making processes or other operations that, to some extent, replicate human cognitive abilities.

These results can be achieved using a variety of different mathematical and computer-based solutions, which are included under the umbrella term of AI.¹ Although they differ in their technicalities, they are all data-intensive systems and it is this factor that seems to be the most characteristic, rather than their human-like results.

We already have calculators, computers and many other devices that perform typical human tasks, in some cases reproducing our way of thinking or acting, as demonstrated by the spread of machine automation over the decades. The revolution is not so much the ‘intelligent’ machine, which we had already (e.g. expert systems), but the huge of information these machines can now use to achieve their results.² No human being is able to process such an amount of information in the same way or so quickly, reach the same conclusions (e.g. disease detection through diagnostic imaging) with the same accuracy (e.g. image detection and recognition) as AI.

These data-intensive AI systems thus undermine a core component of the individual’s ‘sovereignty’ over information:³ the human ability to control, manage and use information in a clear, understandable and ex post verifiable way.

This is the most challenging aspect of these applications, often summed up with the metaphor of the black box.⁴ Neither the large amounts of data – we have always had large datasets⁵ – nor data automation for human-like behaviour are the most significant new developments. It is the intensive nature of the processing, the size of the datasets, and the knowledge extraction power and complexity of the process that is truly different.

If data are at the core of these systems, to address the challenges they pose and draft some initial guidelines for their regulation, we have to turn to the field of law that most specifically deals with data and control over information, namely data protection.

Of course, some AI applications do not concern personal data, but the provisions set forth in much data protection law on data quality, data security and data management in general go beyond personal data processing and can be extended to all types of information. Moreover, the AI applications that raise the biggest concerns are those that answer societal needs (e.g. selective access to welfare or managing smart cities), which are largely based on the processing of personal data.

This correlation with data protection legislation can also be found in the ongoing debate on the regulation of AI where, both in the literature and the policy documents,⁶ fair use of data,⁷ right to explanation,⁸ and transparent data processing⁹ are put forward as barriers to potential misuse of AI.

Here we need to ask whether the existing data protection legislation with its long and successful history¹⁰ can also provide an effective framework for these data-intensive AI systems and mitigate their possible adverse consequences.

1.2 Rise and Fall of Individual Sovereignty Over Data Use

When in 1983 the German Constitutional Court recognised the right to self-determination with regard to data processing,¹¹ the judges adopted an approach that had its roots in an earlier theoretical vision outlined in the 1960s. This was the idea of individual control as a key element in respect for human personality.

This idea was framed in different ways depending on the cultural context¹² and legal framework.¹³ It also extended beyond the realm of data protection as it could relate to general personality rights however they are qualified in different legal contexts.¹⁴ Regardless of the underpinning cultural values of data protection, the idea of an individual’s power to counter potential data misuse is in line with the European tradition of personality rights.

As with personal names, image, and privacy, for personal data too, the theoretical legal framework aims to give individuals a certain degree of sovereignty regarding the perceivable manifestation of their physical, moral and relational identity. The forms and degree in which this sovereignty is recognised will differ over time and may follow different patterns.¹⁵

Individual sovereignty contains two components: the inside/outside boundary and the need to protect these boundaries. In personality rights and data protection, these boundaries concern the interaction between the individual and society (control) and the need for protection concerns the potential misuse of individual attributes outside the individual sphere (risk). While this does not rule out the coexistence of a collective dimension, the structure of individual rights is based on the complementary notions of control and risk.¹⁶

This has been evident since the earliest generations of data protection regulation, which were based on the idea of control over information¹⁷ as a response to the risk of social control relating to the migration from dusty paper archives to computer memories.¹⁸ Their purpose was not to spread and democratise power over information, but to increase the level of transparency about data processing and guarantee the right to access to information, providing a sort of counter-control over the collected data to the citizen.¹⁹

In these first data protection laws we can see the context-dependent nature of this idea of control, where the prevalence of data processing in public hands and the complexity of data processing for ordinary people led regulators to focus on notification, licencing,²⁰ right to access and the role of independent authorities. There was no space for individual consent in this socio-technical context.

The current idea of control as mainly centred on individual consent, already common in the context of personality rights, emerges in data protection as the result of the advent of personal computers and the economic exploitation of personal information, no longer merely functional data but a core element of profiling and competitive commercial strategies.²¹

These changes in the technological and business frameworks created new demands on legislators by society as citizens wished to negotiate their personal data and gain something in return.

Although the later generations of European data protection law placed personal information in the context of fundamental rights,²² the main goal of these regulations was to pursue economic interests relating to the free flow of personal data. This is also affirmed by Directive 95/46/EC,²³ which represented both the general framework and the synthesis of this second wave of data protection laws.²⁴ Nevertheless, the roots of data protection still remained in the context of personality rights making the European approach less market-oriented²⁵ than other legal systems. The Directive also recognised the fundamental role of public authorities in protecting data subjects against unwanted or unfair exploitation of their personal information for marketing purposes.

Both the theoretical model of fundamental rights, based on self-determination, and the rising data-driven economy highlighted the importance of user consent in consumer data processing.²⁶ Consent was not only an expression of choice with regard to the use of personality rights by third parties, but became a means of negotiating the economic value of personal information.²⁷

With the advent of the digital society,²⁸ data could no longer be exploited for business purposes without any involvement of the data subject. Data subjects had to become part of the negotiation, since data was no longer used mainly by government agencies for public purposes, but also by private companies with monetary revenues.²⁹

Effective self-determination in data processing, both in terms of protection and economic exploitation of personality rights, could not be achieved without adequate awareness about data use.³⁰ The notice and consent model³¹ was therefore a new layer added to the existing paradigm based on transparency and access in data processing.

In the 1980s and 1990s data analysis increased in quality, but its level of complexity remained limited. Consumers understood the general correlation between data collection and the purposes of data processing (e.g. miles and points to earn free flights for airlines or nights and points for hotels) and informed consent and self-determination were largely considered synonyms.

This changed with the advent of data-intensive systems based on Big Data analytics and the new wave of AI applications which make data processing more complicated and often obscure. In addition, today’s data-intensive techniques and applications have multiplied in a new economic and technological world which raises questions about the adequacy of the legal framework – established at the end of the last millennium and having its roots in the 1970s – to safeguard individuals’ rights in the field of information technology.

The current social environment is characterised by a pervasive presence of digital technologies and an increasing concentration of information in the hands of just a few entities, both public and private. The main reason for this concentration is the central role played by specific subjects in the generation of data flows. Governments and big private companies (e.g. large retailers, telecommunication companies, etc.) collect huge amounts of data in the course of their daily activities. This mass of information represents a strategic and economically significant asset, since these large datasets enables these entities to act as gatekeepers to the information that can be extracted from datasets. They can choose to restrict access to the data to specific subjects or to circumscribed parts of the information.

Governments and big private companies are not alone in having this power, but the information intermediaries (e.g. search engines,³² Internet providers, data brokers,³³ marketing companies), which do not themselves generate information, do play a key role in circulating it.³⁴

Even where the information is accessible to the public, both in raw and processed form,³⁵ the concurrent effect of all these different sources only apparently diminishes the concentration of power. Access to information is not equivalent to knowledge. A large amount of data creates knowledge only when the holders have the appropriate tools to select relevant information, reorganise it, place it in a systematic context and the people with the skills to design the research and interpret the results of analytics.³⁶

Without this, data only produces confusion and ultimately results in less knowledge, when information is subject to incomplete or biased interpretation. The mere availability of data is not sufficient in AI,³⁷ it is also necessary to have the adequate human³⁸ and computing resources to handle it.

Control over information therefore not only regards limited access data, but can also concern open data,³⁹ over which the information intermediaries create added value with their analytical tools.

Given that only a few entities are able to invest heavily in equipment and research, the above dynamics sharpen the concentration of power, which has increased with the latest wave of AI.⁴⁰

In many respects, this new environment resembles the origins of data processing, the mainframe era, when technologies were held by a few entities and data processing was too complex to be understood by data subjects. Might this suggest that the future will see a sort of distributed AI, as happened with computers in the mid 1970s?⁴¹

The position of the dominant players in AI and data-intensive systems is not only based on expensive hardware and software, which may get cheaper in the future. Nor does it depend on the growing number of staff with specific skills and knowledge, capable of interpreting the results provided by AI applications.

The fundamental basis of their power is represented by the huge datasets they possess. These data silos, considered the goldmine of the 21st century, are not freely accessible, but represent the main or collateral result of their owners’ business, creating, collecting, or managing information. Access to these databases is therefore not only protected by law, but is also strictly related to the data holders’ peculiar market positions and the presence of entry barriers.⁴²

This makes it hard to imagine the same process of ‘democratisation’ as occurred with computer equipment in the 1980s repeating itself today.

Another aspect that characterises and distinguishes this new concentration of control over information is the nature of the purposes of data use: data processing is no longer focused on single users (profiling), but has increased in scale to cover attitudes and behaviours of large groups⁴³ and communities, even entire countries.⁴⁴

The consequence of this large-scale approach is the return of fears about social surveillance and the lack of control over important decision-making processes, which characterised the mainframe era.

At the same time, this new potentially extensive and pervasive social surveillance differs from the past, since today’s surveillance is no longer largely performed by the intelligence apparatus, which independently collects a huge amount of information through pervasive monitoring systems. It is the result of the interplay between private and public sectors,⁴⁵ based on a collaborative model made possible by mandatory disclosure orders, issued by courts or administrative bodies, and extended to an undefined pool of voluntary or proactive collaborations by big companies.⁴⁶

In this way, governments may obtain information with the indirect “co-operation” of consumers who quite probably would not have given the same information to public entities if requested. Service providers, for example, collect personal data on the basis of private agreements (privacy policies) with the consent of the user and for specific purposes,⁴⁷ but governments exploit this practice by using mandatory orders to obtain the disclosure of this information.⁴⁸ This dual mechanism hides from citizens the risk and extent of social control that can be achieved by monitoring social media or other services using data-intensive technologies.⁴⁹

In addition, the current role played by private online platforms and the environment they create, which also include traditional state activities,⁵⁰ raise further issues concerning the possibility of them having an influence on individual and collective behaviour.⁵¹

In this scenario, the legal framework established in the 1990s to regulate data use⁵² has gone to crisis, since the new technological and economic contexts (i.e. market concentration, social and technological lock-ins) have undermined its fundamental pillars,⁵³ which revolve around the purpose specification principle, the prior limitation of possible uses,⁵⁴ and an idea of individual self-determination mainly based on the notice and consent model.

The purpose specification and use limitation principles have their roots in the first generation of data protection regulation, introduced to avoiding extensive and indiscriminate data collection that might entail risks in terms of social surveillance and control.

In the 1980s and 1990s, with the advent of a new generation of data protection regulation, these principles not only put a limit on data processing, but also became key elements of the notice and consent model. They define the use of personal data made by data controllers, which represents important information impacting users’ choice. Nevertheless, the advent of AI applications makes it difficult to provide detailed information about the purposes of data processing and the expected outputs.

Since data-intensive systems based on AI are designed to extract hidden or unpredictable inferences and correlations from datasets, the description of these purposes is becoming more and more generic and approximate. This is a consequence of the “transformative”⁵⁵ use of data made by these systems, which often makes it impossible to explain all the possible uses of data at the time of its initial collection.⁵⁶

These critical aspects concerning the purpose specification limitation have a negative impact on the effectiveness of the idea of informational self-determination as framed by the notion of informed consent.

First, the difficulty of defining the expected results of data use leads to the introduction of vague generic statements about the purposes of data processing. Second, even where notices are long and detailed, the complexity of the AI-based environment makes it impossible for users to really understand it and make informed choices.⁵⁷

Moreover, the situation is made worse by economic, social, and technological constraints, which completely undermine the idea of self-determination with regard to personal information which represented the core principle of the generation of data protection regulation passed in the 1980s and 1990s.⁵⁸

Finally, as mentioned before, we have seen an increasing concentration of informational assets, partly due to the multinational or global nature of a few big players in the new economy, but also due to mergers and acquisitions that created large online and offline companies. In many cases, especially in IT-based services, these large-scale trends dramatically limit the number of the companies that provide certain services and which consequently have hundreds of millions of users. The size of these dominant players produces social and technological lock-in effects that accentuate data concentration and represent further direct and indirect limitations to the consumer’s self-determination and choice.⁵⁹

1.3 Reconsidering Self-determination: Towards a Safe Environment

In the above scenario, characterised by data-intensive applications and concentration of control over information, the decision to stick with a model based largely on an idea of informational self-determination centred on informed consent is critical to the effective protection of individuals and their rights.⁶⁰

This leads us to reconsider the role of user self-determination in situations where individuals are unable to understand data processing and its purposes fully⁶¹ or are not in a position to decide.⁶² In these cases, the focus cannot be primarily on the user and self-determination but must shift to the environment. A broader view is needed, with human-centred solutions and applications where the burden of assessing the potential benefits and risks for individual rights and freedoms does not fall mainly on the shoulders of the impacted individuals or groups.

Without limiting the freedom of individuals not to be subject to AI-systems – with the exception of cases of prevailing competing interests (e.g. crime detection systems) –, these systems should provide a safe environment in terms of potential impacts on fundamental rights and freedoms. Just as customers do not have to check the safety of the cars they buy, in the same way the end users of AI systems should not have to check whether their rights and freedoms are safeguarded.

AI providers and AI systems users (e.g., municipalities in smart cities), and not end users (e.g., citizens), are in the best position to assess these risks to individual rights and freedoms and to develop or deploy AI systems with a rights-oriented design approach, under the supervision of competent and independent authorities. Furthermore, they are also in the best position to consider all the different interests of the various stakeholders with regard to extensive data collection and data mining.⁶³

Against this background and given the data-intensive nature of the systems involved, a first line of attack might be to consider data protection law as the reference framework for AI regulation, broadening its scope. This has been done in the literature with regard to the GDPR, focusing on open clauses such as fairness of data processing⁶⁴ or promoting the data protection impact assessment (DPIA) as a general-purpose methodology.⁶⁵

However, looking at the big picture and not just specific elements, existing data protection regulations are still focused on the traditional pillars of the so called fourth generation of data protection law:⁶⁶ the purpose specification principle, the use limitation principle and the notice and consent model (i.e. an informed, freely given and specific consent).⁶⁷

These components of data protection regulation struggle with today’s challenges, where the transformative use of data⁶⁸ often makes it impossible to know and explain all the uses of information at the time of its initial collection, or provide detailed information about AI data processing and its internal logic.⁶⁹

The asymmetric distribution of control over information and market concentration⁷⁰ highlighted in the previous section,⁷¹ as well as social⁷² and technological lock-ins,⁷³ further undermines the idea of information self-determination in AI based mainly on the user’s conscious decision on the potential benefits and risks of data processing.⁷⁴

In addition, looking at the potential impact of AI, these data-intensive systems may affect a variety of rights and freedoms⁷⁵ that is much broader than the sphere covered by data protection. This must necessarily be reflected in the assessment methodologies which should go beyond the limited perspective adopted in today’s data protection impact assessment models, which are mainly centred on the processing, task allocation, data quality, and data security.⁷⁶

Although the EU legislator recognises data processing risks such as discrimination and “any other significant economic or social disadvantage”,⁷⁷ and recommends a broader assessment including analysis of the societal and ethical consequences,⁷⁸ Article 35 of the GDRP and the supervisory authorities’ assessment models do not adequately consider potentially impacted rights, their diversity and complexity, or the ethical and social issues.⁷⁹

Finally, the impact on society of several AI-based systems raises ethical and social issues, which have been only touched on in defining the purposes of DPIA and often poorly implemented in practice.⁸⁰

For these reasons, a holistic approach to the problems posed by AI must look beyond the traditional data protection emphasis on transparency, information, and self-determination. In the presence of complicated and often obscure AI applications, focusing on their design is key to ensuring effective safeguarding of individual rights. Such safeguards cannot simply be left to the interaction between AI manufacturers/adopters and potentially impacted individuals, given the asymmetry and bias inherent to this interaction.

Given the active and crucial role in creating a safe environment – from a legal, social and ethical perspective – of those who design, develop, deploy and adopt AI systems, it is crucial to provide them with adequate tools to consider and properly address the potential risks of AI applications for individuals and society.

1.4 A Paradigm Shift: The Focus on Risk Assessment

Risk assessment models today play an increasing role in many technology fields, including data processing,⁸¹ as a consequence of the transformation of modern society into a risk society⁸² – or at least a society in which many activities entail exposure to risks and one that is characterised by the emergence of new risks. This has led legislators to adopt a risk-based approach in various areas of the legal governance of hazardous activities.⁸³

There are different assessment models possible (technology assessment, risk/benefit assessment, rights-based assessment) in different domains (e.g., legal assessment, social assessment, ethical assessment), but the first question we need to ask when defining an assessment model is whether the model is sector-specific or general. This is an important question with respect to AI too, since AI solutions are not circumscribed by a specific domain or technology.

The adoption of a technology-specific approach, for example an IoT impact assessment, a Big Data impact assessment, a smart city impact assessment seems misguided.⁸⁴ From a rights-oriented perspective, all these technologies and technology environments are relevant insofar as they interact with individuals and society, and have a potential impact on the decision-making process.

Regardless of the different software and hardware technologies used, the focus of a human-centred approach is necessarily on the rights and values to be safeguarded. The model proposed here is thus not a technological assessment,⁸⁵ but a rights-based and values-oriented assessment.

In the context of data-driven applications, an assessment model focused on a specific technology appears inadequate or only partially effective.⁸⁶ On the other hand, given the various application domains (healthcare, crime prevention, etc.), different sets of rights, freedoms and values are at stake. A sector-specific approach must therefore focus on the rights and values in question rather than the technology.

Sectoral models concentrate their attention, not on technologies, but on the context and the values that assume relevance in a given context.⁸⁷ This does not mean that the nature of the technology has no importance in the assessment process as a whole, but that it mainly regards the type and extent of the impact.

Adopting a value-oriented approach, the assessment should focus on the societal impact which includes the potential negative outcomes on a variety of fundamental rights and principles, no longer restricted to simple privacy-related risks,⁸⁸ and encompassing the ethical and social consequences of data processing.⁸⁹

A general AI impact assessment, centred on human rights,⁹⁰ ethical and societal issues, can address the call for a broader protection of individuals in the AI context and better deal with the rising demand for ethically and socially oriented AI from citizens and companies.⁹¹

The inclusion of ethical and societal issues is consistent with the studies in the realm of collective data protection⁹² that point out the importance of these non-legal dimensions in the context of data-intensive applications.⁹³ Evidence in this regard comes from predictive policing software, credit scoring models and many other algorithmic decision-support systems that increasingly target groups and society at large rather than single persons, thus highlighting the group and societal scale of the potential adverse impacts.

Although the present absence of a holistic approach to risk in AI is partially filled by a variety of bottom-up initiatives, corporate guidance or ongoing public investigations, the main limitations of these initiatives concern the variety of values, approaches and models adopted.⁹⁴ Similarly the ongoing debate on AI regulation has not yet furnished a clear assessment model.⁹⁵

Against this background, the following sections sketch out a uniform model – whose components are discussed in greater detail in Chaps. 2 and 3 – which provides a common ground for an AI application assessment and, at the same time, offers sufficient flexibility to give voice to differing viewpoints.

1.5 HRESIA: A Multi-layered Process

The main components of the Human Rights, Ethical, and Social Impact Assessment (HRESIA) are the analysis of relevant human rights, the definition of relevant ethical and social values and the targeted application of these frameworks to given AI cases. The HRESIA therefore combines the universal approach of human rights⁹⁶ with the local dimension of societal values.

The first layer of the model is based on the common values found in human rights and related process principles,⁹⁷ whose relevance has also been recognised by Data Protection Authority (DPA) jurisprudence and the courts.⁹⁸ The second layer concerns the social and ethical values which play an important role in addressing non-legal issues associated with the adoption of certain AI solutions and their acceptability, and the balance between the different human rights and freedoms, in different contexts and periods.⁹⁹

The proposed model therefore combines the human rights assessment with attention to the societal and ethical consequences,¹⁰⁰ but without becoming a broader social impact assessment, remaining focused on human rights. In this sense, ethical and social values are viewed through the lens of human rights and serve to go beyond the limitations of legal theory or practical implementation in effectively addressing the most urgent issues concerning the societal impacts of AI.

Moreover, ethical and social values are key to interpreting human rights in the regional context, in many cases representing the unspoken aspect of the legal reasoning behind the decisions of supervisory authorities or courts when ruling on large-scale impacting use of data.¹⁰¹

One option in trying to embody this theoretical framework in an assessment tool focused on concrete cases is to follow the models already adopted in the field of data processing.¹⁰² This is envisaged in the recent proposals concerning AI,¹⁰³ which follows a questionnaire-based approach including, in some cases, open questions concerning human rights and social issues, though with a limited level of granularity.

However, the HRESIA model follows a different approach, in which the focus on human rights exploits different tools to the focus on ethical and social issues: the first relies on questionnaires and risk assessment tools (Chap. 2), while the second is built on the use of experts to address societal challenges associated with the development and implementation of AI solutions (Chap. 3).

Questionnaires and checklists alone are not sufficient to cover the human rights, ethical and societal components of the impact assessment. They can be useful in the HRIA (Human Rights Impact Assessment) planning and scoping phase, as well as in the collection of relevant data, but this is only one part of the assessment procedure, which includes evaluation models, data analysis, and expert evaluation.¹⁰⁴

In the case of ethical and social issues, standardised questionnaires and checklists cannot grasp the specificities of the case, whereas experts interacting with relevant stakeholders can play a crucial role in understanding and exploring important questions. Questionnaires and checklists are just two of the possible tools to be used in fieldwork, along with focus groups, interviews, etc.¹⁰⁵

From a methodological standpoint, an important role is played by participation¹⁰⁶ which makes it possible to get a better understanding of the different competing interests and societal values.¹⁰⁷ Both in carrying out the assessment and in the mitigation phase – where the results of the HRESIA may suggest the engagement of specific categories of individuals –, participation can give voice to the different groups of persons potentially affected by the use of data-intensive systems and different stakeholders¹⁰⁸ (e.g. NGOs, public bodies)¹⁰⁹ facilitating a human-centred approach to AI design.

Participation is therefore a development goal for the assessment,¹¹⁰ since it reduces the risk of under-representing certain groups and may also flag up critical issues that have been underestimated or ignored.¹¹¹ However, as pointed out in risk theory,¹¹² participation should not become a way for decision makers to avoid their responsibilities as leaders of the entire process.¹¹³ Decision makers, in the choice and use of AI systems, must remain committed to achieving the best results in terms of minimising the potential negative impacts of data use on individuals and society.

Finally, given the social issues that underpin the HRESIA, transparency is an essential methodological requirement of this model. Transparency is crucial for an effective participation (Chap. 3) – as demonstrated in fields where impact assessments concern the societal consequences of technology (e.g. environment impact assessments) – and is also crucial in providing potentially affected people with information to give them a better understanding of the AI risks and reduce the limitations on their self-determination.

Along the lines of risk management models, the HRESIA assessment process adopts a by-design approach from the earliest stages and is characterised by a circular approach that follows the product/service throughout its lifecycle, which is also in line with the circular product development models that focus on flexibility and interaction with users to address their needs.¹¹⁴

1.6 The Role of Experts

The combination of these different layers in the model proposed here is intended to provide a self-assessment tool enabling AI system developers, deployers, and users to identify key values guiding the design and implementation of AI products and services. However, general background values and their contextual application may be not enough to address the societal changes when designing data-intensive systems. Although balanced with respect to the context, the definition of such rights and values may remain theoretical and need to be further tailored to the specific application.

To achieve a balance in specific cases, individuals with the right skills are needed to apply this set of rights and values in the given situation. The difficulty of bridging the gap between the theory of rights and values and their concrete application, given the nature of data use and the complexity of the associated risks, means that experts can play an important role in applying general principles and guidelines to a specific case (see Chap. 3).

Experts are therefore a key component of model implementation as they assist AI developers and users in this contextualisation and in applying the HRESIA benchmark values to the given case, balancing interests that may be in conflict, assessing risks and mitigating them.

The need for an expert view in data science has already been perceived by AI companies. The increasing and granular availability of data about individuals gathered from various devices, sensors, and online services enable private companies to collect huge amounts of data from which they can extract further information about individuals and groups. Private companies are therefore now more easily able to conduct large-scale social investigations, which can be classed as research activities, traditionally carried out by research bodies. This raises new issues since private firms often do not have the same ethical¹¹⁵ and scientific background as researchers in academia or research centres.¹¹⁶

To address this lack of expertise, the adoption of ethical boards has been suggested, which may act at a national level, providing general guidelines, or at a company level, supporting data controllers on specific data applications.¹¹⁷ Several companies have already set up ethical boards, appointed ethical advisors or adopted ethical guidelines.¹¹⁸

However, these boards have a limited focus on ethical issues and do not act within a broader framework of rights and values. Such shortcomings highlight the self-regulatory nature of these solutions lacking a strong general framework that could provide a common baseline for a holistic approach to human-centred AI.

On the other hand, committees of experts within the HRESIA framework could build on the human rights framework outlined above, representing a sound and common set of values to guide expert decisions and complemented by the ethical and social values taken into account by the HRESIA.

These aspects will clearly have an influence on the selection of the experts involved. Legal expertise, an ethical and sociological background, as well as domain-specific knowledge (of data applications) are required. Moreover, the background and number of experts will also depend on the complexity of AI use.¹¹⁹

The main task of the experts is to consider the specific AI use and place it in the local context, providing a tailored and more granular application of the legal and societal values underpinning the HRESIA model. In this process, the experts may decide that this contextual application of general principles and values requires the engagement of the groups of individuals potentially affected by AI¹²⁰ or institutional stakeholders. In this sense, the HRESIA is not a mere desk analysis, but takes a participatory approach – as described earlier¹²¹ – which may be enhanced by the work of the experts involved in the HRESIA implementation.

To guarantee the transparency and the independence of these experts and their deliberations, specific procedures to regulate their activity, including stakeholder engagement should be adopted. In addition, full documentation of the decisional process should be recorded and archived for a specific period of time depending on the type of data use.

1.7 Assessing the Impact of Data-Intensive AI Applications: HRESIA Versus PIA/DPIA, SIA and EtIA

When comparing the HRESIA model with the impact assessment solutions adopted in the field of data-centred systems, the main reference is the experience gained in data protection.

The focus on the risks arising from data processing has been an essential element of data protection regulation from the outset, though over the years this risk has evolved in a variety of ways.¹²² The original concern about government surveillance¹²³ has been joined by new concerns regarding the economic exploitation of personal information (risk of unfair or unauthorised uses of personal information¹²⁴) and, more recently, by the increasing number of decision-making processes based on information (risk of discrimination, large scale social surveillance, bias in predictive analyses¹²⁵).

From a theoretical perspective, this focus on the potential adverse effects of data use has not been an explicit element of data protection law. The main purpose of many of the provisions is the safeguarding of specific values, rights and freedoms (e.g. human dignity, non-discrimination, freedom of thought, freedom of expression) against potential prejudices, adopting a procedural approach that leaves in the shadows these interests, which are encapsulated in the broad and general notion of data protection.

Moreover, compared to other personality rights, such as right to image or name, data protection has a proteiform nature, as data may consist of name, numbers, behavioural information, genetic data or many other types of information. The progressive datafication of our world makes it difficult to find something that is not or cannot be transformed into data. The resulting broad notion of data protection covers different fields and has partially absorbed some elements traditionally protected by other personality rights.¹²⁶

Against this background, the idea of control over information was used to aggregate the various forms of data protection and to find a common core.¹²⁷ The procedural approach is consistent with this idea, as it secures all stages of data processing, from data collection to communication of data to third parties. Nevertheless, control over information describes the nature of the power that the law grants to the data subject, not its theoretical foundations.

In this regard, part of the legal doctrine has emphasised the role of human dignity as the cornerstone of data protection in Europe.¹²⁸ However, the interplay with the non-discrimination principle¹²⁹ and the role of data protection in the public sphere and digital citizenship¹³⁰ suggest that a broader range of values underpin data protection.

Although, over the years, data protection regulations¹³¹ and practices¹³² have adopted a more explicit risk-based approach to address the varying challenges of data use, they still focus on the procedural aspects. Data management procedures therefore represent a form of risk management based on the regulation of the different stages of data processing (collection, analysis and communication) and the definition of the powers and tasks of the various actors involved in this process.

This procedural approach and the focus of risk assessment on data management have led data protection authorities to propose assessment models (Privacy Impact Assessment, PIA) primarily centred on data quality and data security, leaving aside the nature of safeguarded interests. Instead, these interests are taken into account by DPAs and courts in their decisions, but – since data protection laws provide limited explicit references to the safeguarded values, rights and freedoms – the analysis of the relevant interest is often curtailed or not adequately elaborated.¹³³

Data protection authorities and courts prefer arguments grounded on the set of criteria provided by data protection regulations.¹³⁴ The lawfulness and fairness of processing, transparency, purpose limitation, data minimisation, accuracy, storage limitation, data integrity and confidentiality are general principles frequently used by data protection authorities in their argumentations.¹³⁵ However, these principles are only an indirect expression of the safeguarded interests. Most of them are general clauses that may be interpreted more or less broadly and require an implicit consideration of the interests underpinning data use.

Moreover, the indefinite nature of these clauses has frequently led to the adoption of the criterion of proportionality¹³⁶, which amounts to a synthesis of the different competing interests and rights by courts or the DPAs. In fact, this balancing of interests and the reasoning that has resulted in a precise distinction between them is often implicit in the notion of proportionality and not discussed in the decisions taken by the DPAs or only discussed in an axiomatic manner.¹³⁷

Against this scenario, it is difficult for data controllers to understand and acknowledge the set of legal and social values that they should take into account in developing their data-intensive devices and services, since these values and their mutual interaction remain unclear and undeclared. Nor is this difficulty solved by the use of PIAs, since these assessment models merely point out the need to consider aspects other than data quality and data security, without specifying them or providing effective tools to identify and enlist broader social values.

Equally, the recent requirements of the GDPR – according to the models proposed by the DPAs – fail to offer a more satisfactory answer. Despite specific references in the GDPR to the safeguarding of rights and freedoms in general as well as to societal issues,¹³⁸ the new assessment models do nothing to pay greater attention to the societal consequences than the existing PIAs.¹³⁹

The HRESIA fills this gap, providing an assessment model focused on the rights and freedoms that may be affected by data use¹⁴⁰ offering a more appropriate contextualisation of the various rights and freedoms that are relevant to data-intensive systems. The latter are no longer limited to data protection and should therefore be considered separately rather than absorbed in a broad notion of data protection.

Moreover, the HRESIA makes explicit the relevant social and ethical values considered in the evaluation of the system, while data protection laws, as well as proposed AI regulations, use general principles (e.g. fairness or proportionality) and general clauses (e.g. necessity, legitimacy¹⁴¹) to introduce non-legal social values into the legal framework. Legal scholars have also highlighted how the application of human rights is necessarily affected by social and political influences that are not explicitly formalised in court decisions.¹⁴²

From this perspective, a HRESIA may be used to unveil the existing interplay between the legal and the societal dimensions,¹⁴³ making it explicit. It is important to reveal this cross-fertilization between law and society, without leaving it concealed between the lines of the decisions of the courts, DPAs or other bodies.

Finally, a model that considers the social and ethical dimensions also helps to democratise assessment procedures, removing them from the exclusive hands of the courts, mediated by legal formalities.

This change in the assessment analysis can have a direct positive impact on business practices. Although courts, DPAs and legal scholars are aware of the influence of societal issues on their reasoning, this is often not explicit in their decisions. Product developers are therefore unable to grasp the real sense of the existing provisions and their implementation. Stressing the societal values that should be taken into account in human rights assessment helps developers to carry out self-assessments of the potential and complex consequences of their product and services, from the early stages of product design.

Some may argue that one potential shortcoming of the proposed approach concerns the fact that it may introduce a paternalistic view to data processing. In this sense, a HRESIA model necessarily encourages system designers, developers and users to rule out certain processing operations due to their ethical or social implications, even though some end users may take a different view and consider them in line with their own values. The model may therefore be seen as a limitation of self-determination, indirectly reducing the range of available data use options.

The main pillar of this argument rests on individual self-determination, but this notion is largely undermined by today’s AI-driven data use.¹⁴⁴ The lack of conscious understanding in making decisions on data processing, and the frequent lack of effective freedom of choice (due to social, economic and technical lock-ins), argue for a slightly paternalistic approach as a way to offset these limitations on individual self-determination.¹⁴⁵ Moreover, HRESIA is not a standard but a self-assessment tool. It aims to provide a better awareness of the human rights, ethical and social implications of data use, including a bottom-up participatory approach and a context-based view, which give voice to different viewpoints.

Finally, the publicity surrounding the HRESIA (in line with the HRIA) may help to reinforce individual self-determination, as it makes explicit the implications of a certain data processing operation and fosters end users’ informed choice. Publicity increases not only the data subject’s awareness, but also the data controller’s accountability in line with a human rights-oriented approach.¹⁴⁶

There are cases in which full disclosure of the assessment results may be limited by the legitimate interests of the data controller, such as confidentiality of information, security, and competition. For example, the Guidelines on Big Data adopted by the Council of Europe in 2017¹⁴⁷ – following the opinions of legal scholars¹⁴⁸ – specify that the results of the assessment proposed in the guidelines “should be made publicly available, without prejudice to secrecy safeguarded by law. In the presence of such secrecy, controllers provide any confidential information in a separate annex to the assessment report. This annex shall not be public but may be accessed by the supervisory authorities”.¹⁴⁹

Having highlighted the difference between PIA/DPIA and HRESIA, it is worth noting how closely HRESIA stands to the SIA (Social Impact Assessment). They share a similar focus on societal issues and the collective dimension,¹⁵⁰ an interest in public participation, empowerment of individuals and groups through the assessment process, attention to non-discrimination and equal participation in the assessment, accountability procedures and circular architecture. Important similarities also exist with the EtIA (Ethical Impact Assessment) models¹⁵¹ and the focus on the ethical dimension.

However, despite the similarities, there are significant differences that set the HRESIA apart from both the PIA/DPIA and the SIA and EtIA models. The main differences concern the rationale of these models, the extent of the assessment and the way the different interests are balanced in the assessment. The HRESIA aims to provide a universal tool that, at the same time, also takes into account the local dimension of the safeguarded interests. In this sense, it is based on a common architecture grounded on intentional instruments with normative force (charters of fundamental rights). The core of the architecture is represented by human rights, which also play a role in SIA models but are not pivotal, as the SIA takes a wider approach.¹⁵²

In fact, the scope of the SIA model encompasses a wide range of issues,¹⁵³ broad theoretical categories and focuses on the specific context investigated.¹⁵⁴ The solutions proposed by the SIA are therefore heterogeneous and vary in different contexts,¹⁵⁵ making it difficult to place them within a single framework, which – on the contrary – is a key requirement in the context of the global policies on AI.

By contrast, a model grounded on human rights¹⁵⁶ is more closely defined and universally applicable. Moreover, the SIA is designed for large-scale social phenomena, such as policy solutions,¹⁵⁷ while the HRESIA focuses on specific data-intensive AI applications.

Finally, the HRESIA is largely a rights-based assessment, in line with the approach adopted in data protection (PIA, DPIA), while both the SIA and the EtIA (Ethical Impact Assessment) are risks/benefits models.

On the comparison between HRESIA and EtIA,¹⁵⁸ the same considerations made with regard to SIA can be made in relation to EtIA.¹⁵⁹ In the forms proposed in the context of data use, there is a clearer link in the EtIA model with the ethical principles already recognised in law.¹⁶⁰ However, a purely ethical assessment does run the risk of overlap between ethical guidance and legal requirement.

1.8 The HRESIA and Collective Dimension of Data Use

Shifting the focus from the traditional sphere of data quality and security to fundamental rights and freedoms, the HRESIA can be of help in dealing with the emerging issues concerning the collective dimension of data processing.¹⁶¹

Data-intensive applications and their use in decision-making processes impact on a variety of fundamental rights and freedoms. Not only does the risk of discrimination represent one of the biggest challenges of these applications, but other rights and freedoms also assume relevance, such as the right to the integrity of the person, to education, to equality before the law, and freedom of movement, of thought, of expression, of assembly and freedom in the workplace.¹⁶²

Against this scenario, the final question that the proposed model must address regarding its interplay with data protection concerns the compatibility of the collective dimension of data protection and the way human rights are framed by legal scholars. To answer to this question, it is necessary to highlight how the notion of collective data protection tried to go beyond the individual dimension of data protection and its focus on data quality and security, suggesting a broader range of safeguarded interests and considering individuals as a group.

An impact assessment focussing on the broader category of human rights, which also takes into account the ethical and societal issues related to data use, can provide an answer to this need. This broader perspective and the varied range of human rights makes it possible to consider the impacts of data use more fully, not only limited to the protection of personal information. Moreover, several principles, rights, and freedoms in the charters of human rights directly or indirectly address group or collective issues.

However, in the context of human rights¹⁶³ as well as data protection, legal doctrine and the regulatory framework focus primarily on the individual dimension. Furthermore, in some cases, human rights theory provides little detail on the rights and freedoms threatened by the challenges of innovative digital technology.¹⁶⁴

In this regard, for example, the approach to classification adopted by modern algorithms does not merely focus on individuals and on the categories traditionally used for unfair or prejudicial treatment of different groups of people.¹⁶⁵ Algorithms create groups or clusters of people with common characteristics other than the traditionally protected grounds (e.g. customer habits, lifestyle, online and offline behaviour, network of personal relationships etc.). For this reason, the wide application of predictive technologies based on these new categories and their use in decision-making processes challenges the way discrimination has usually been understood.¹⁶⁶

Additionally, the nature of the groups created by data-intensive applications poses challenging issues from the procedural viewpoint, which concern the potential remedies to the need for collective representation in the context of algorithmic-created groups.¹⁶⁷ Indeed, people belonging to groups that are the traditional targets of discriminatory practices are aware of their membership of these groups and they know or may know the other members of the group. On the contrary, in the groups generated by algorithms, people do not know the other members of the group and, in many cases, are not aware of the consequences of their belonging to a group. Data subjects are not aware of the identity of the other members of the group, have no relationship with them and have a limited perception of their collective issues.

Hard law remedies in this field may not be easy to achieve in the short run and the existing or potential procedural rules often vary from one legal context to another.¹⁶⁸ In this scenario, an assessment tool may represent a valid alternative to address these challenges. For these reasons, a model based on a participatory approach and in which human rights are seen through the lens of ethical and social values can provide broader safeguards both in terms of the interests taken into account and the categories of individuals engaged in the process.

Finally, providing a framework for a collective and societal impact assessment of data-intensive applications is also in line with the ongoing debate on Responsible Research Innovation¹⁶⁹ and the demands of the data industry and product developers for practical self-assessment tools to help them address the social issues of data use. Tools should be more flexible, open to new emerging values, easily reshaped and applicable in different legal and cultural contexts. At the same time, it should be pointed out how the HRESIA model differs from the Responsible Research Innovation assessment, where the latter takes into account a variety of societal issues, which do not necessarily concern fundamental rights and freedoms¹⁷⁰ (e.g. interoperability, openness).¹⁷¹

1.9 Advantages of the Proposed Approach

The positive features of the proposed model for assessing the impact of data use can be briefly summarised as follows:

• The central role of human rights in HRESIA provides a universal set of values, making it suited to various legal and social contexts.

• The HRESIA is a principle-based model, which makes it better at dealing with the rapid change of technological development, not easily addressed by detailed sets of provisions.

• The proposed model follows in the footsteps of the data protection assessments, as a rights-based assessment in line with the PIA and DPIA approaches. However, it is broader in scope in that individual rights are properly and fully considered, coherent with their separate theoretical elaboration.

• The HRESIA emphasises the ethical and social dimensions, giving a better understanding of the human rights implications in a given context, and as spheres to be considered independently when deciding to implement data-intensive AI-based systems affecting individuals and society.

• By stressing ethical and social values, the HRESIA helps to make explicit the non-legal values that inform the courts and DPAs in their reasoning when they apply general data protection principles, interpret general clauses or balance conflicting interests in the context of data-intensive systems.

• In considering ethical and social issues, this model makes it possible to give flexibility to the legal framework in dealing with AI applications. A human rights assessment that operates through the lens of ethical and social values can therefore better address the challenges of the developing digital society.

• Finally, as an assessment tool, the HRESIA fosters the adoption of a preventive approach to product/service development from the earliest stages, favouring safeguards to rights and values, and a responsible approach to technology development.

1.10 Summary

The increasing use of AI in decision-making processes highlights the importance of examining the potential impact of AI data-intensive systems on individuals and society at large.

The consequences of data processing are no longer restricted to the well-known privacy and data protection issues but encompass prejudices against groups of individuals and a broader array of fundamental rights. Moreover, the tension between the extensive use of data-intensive systems, on the one hand, and the growing demand for ethically and socially responsible data use on the other, reveals the lack of a regulatory framework that can fully address the societal issues raised by AI technologies.

Against this background, neither traditional data protection impact assessment models (PIA and DPIA) nor the broader social or ethical impact assessment procedures (SIA and EtIA) appear to provide an adequate answer to the challenges of our algorithmic society.

While the former have a narrow focus – centred on data quality and data security – the latter cover a wide range of issues, employing broad theoretical categories and providing a variety of different solutions. A human rights-centred assessment may therefore offer a better answer to the demand for a more comprehensive assessment, including not only data protection, but also the effects of data use on other fundamental rights and freedoms (such as freedom of movement, freedom of expression, of assembly and freedom in the workplace) and related principles (such as non-discrimination).

Moreover, a human rights assessment is grounded on the charters of fundamental rights, which provide the common baseline for assessing data use in the context of global AI policies.

While the Human Rights Impact Assessment (HRIA) is not a new approach in itself¹⁷² and has its roots in environmental impact assessment models and development studies,¹⁷³ HRIA has not yet been systematically applied in the context of AI.¹⁷⁴

However, given the enormous changes to society brought by technology and datafication, when applied to the field of AI the HRIA must be enriched to consider ethical and societal issues, evolving into a more holistic model such as the proposed Human Rights, Ethical and Social Impact Assessment (HRESIA).

The HRESIA is also more closely aligned with the true intention of the EU legislator to safeguard not only the right to personal data protection, but also the fundamental rights and freedoms of natural persons.

Furthermore, ethical and social values, viewed through the lens of human rights, make it possible to overcome the limitations of the traditional human rights impact assessment and help to interpret human rights in line with the regional context. The HRESIA can in this way contribute to a universal tool that also takes the local dimension of the safeguarded interests into account.

To achieve these goals the HRESIA model combines different components, from self-assessment questionnaires to participatory tools. They help define the general value framework and place it in a local context, providing a tailored and granular application of the underlying legal and social values.

On the basis of this architecture, such an assessment tool can raise awareness among AI manufacturers, developers, and users of the impact of AI-based products/services on individuals and society. At the same time, a participatory and transparent assessment model like the HRESIA also gives individuals an opportunity for more informed choices concerning the use of their data and increases their awareness about the consequences of AI applications.

This assessment may represent an additional burden for AI industry and adopters. However, even in contexts where it is not required by law,¹⁷⁵ it could well gain ground in those areas where people pay greater attention to ethical and social implications of AI (healthcare, services/products for kids, etc.) or where socially oriented entities or developers’ communities are involved. Moreover, as has happened in other sectors, a greater attention to human rights and societal impacts may represent a competitive advantage for companies that deal with responsible consumers and partners.

Finally, the focus of policymakers, industry, and communities on ethical and responsible use of AI, and the lack of adequate tools to assess the impacts of AI on the fundamental rights and freedoms, as called for by the proposals under discussion in Europe,¹⁷⁶ also make the HRESIA a possible candidate as a mandatory assessment tool.