Keywords

Introduction

“Data justice” is a broad paradigm that encompasses and supersedes legal issues of data ownership and privacy regulations to account for complex power imbalances and injustices that are brought about by big data collection and use (Taylor, 2017). The concept is multifaceted in its applications. Data justice can aim towards “just decisions” and “just procedures” in administrative, legal, contractual, and other situations, where these decisions and procedures are made based on increasingly larger amounts of data. It can also mean aiming towards just decisions in sociotechnical systems’ design when it relies on and pertains to data. Broader understandings of data justice can refer to justice on the level of policies, institutions, and societal structures pertaining to (big) data collection and use.

The relationship between data and justice as separate components is marked by complex considerations. For one, drawing on the metaphor of the classical image of Justitia as blindfolded, justice is envisioned through not having or not using certain types of data, such as those related to categories of race, class, gender, and so on. Simultaneously, just decisions require other types of data related to a person’s societal attributes and actions. Justitia’s blindfold is not meant to prevent her from seeing the relevant facts of a specific case. On the contrary, covering only one decisive fact may lead to arbitrary and unjust results. Thus, having and using certain types of data is seen as a prerequisite of justice, invoking a need to distinguish between relevant and irrelevant data to make just decisions.

Navigating the tension between having/using and not having/using certain types of data is crucial for more equitable, just, non-discriminatory futures. The underrepresentation of certain groups, people, contexts in datasets has problematic effects in proving injustice because inequalities need to be made visible. Not having data about diverse populations in engineering, computing, medicine, design, and architecture also leads to services and products that are unusable or inaccessible for some (Criado-Perez, 2019). Data collection, however, can be problematic, especially for vulnerable and marginalised groups. Concerns of privacy as well as underrepresentation and over-surveillance have been raised by people of colour, feminists, and LGBTQI communities (Browne, 2015; Shephard, 2016, 2018; Weinberg, 2017). The question of decision power over what counts as relevant and irrelevant data for making just decisions, whether answered “by design” or left open, is nonetheless particularly significant.

This points to aspects of data justice that concern well-being, participation, and broader contextual and societal aspects—what is commonly known as social justice. Reconfiguring data justice to include social justice means drawing attention to deeper structural imbalances in a given society and using data justice as a guide for legal and policy frameworks that actively work towards eliminating injustices. For algorithmic decision-making systems, this would entail paying attention to who is unduly overserved and underserved in regard to the use of these systems (Benjamin, 2019). This ties data justice to equity, equality, and fairness, as well as to “design justice”, a concept promoting participation, power in design processes, and justice built into scenarios, systems, and infrastructures (Costanza-Chock, 2020). Such an understanding of justice has been prominent in intersectional feminist thinking that points out interlacing effects of structural inequalities.

Digitalisation has an enormous impact on all of the above-mentioned aspects of data justice. Algorithms and data processing have always been at the core of computing but today’s pervasiveness of information technology and, especially the rise of machine learning and algorithmic decision-making (ADM) technologies with their reliance on large data sets, pose new threats and provide new opportunities. Support in decision-making is one of the most prominent and controversially discussed applications of machine learning. While this chapter pertains to information systems in general, we specifically developed the arguments with ADM technologies in mind.

A key threat posed by ADM technologies is that they stimulate further adverse effects on vulnerable and marginalised populations, increasing opacity and challenges in controlling the use of data in decision-making, and increasing difficulty in locating and ascribing accountability and responsibility. Critical research in systems design has shown that the pervasive universalisation of technological, legal, and policy solutions might also hinder context-specific and community-attuned approaches to data justice (Couldry & Mejias, 2019; Dourish & Mainwaring, 2012; Say Chan, 2018; Thatcher et al., 2016). These risks call for a concept of data justice that is both normatively robust to provide benchmarks for legal decision-making, practically applicable to address design justice concerns as well as conceptually flexible enough to accommodate or at least serve as common ground to bring in broader questions of social justice that go beyond regulation and appeal to historically formed sociopolitical structures and contexts. This chapter proposes just such an account of data justice as an interdisciplinary, multidimensional concept. First, we interrogate data justice through the lenses of feminist and legal studies and second, we investigate pathways towards realising data justice as a multidimensional practice in IT-design aided by these perspectives. We start by looking at how data justice is framed in feminist research and feminist-informed critical data and design perspectives. We then describe how it is conceptualised in law, particularly in the context of the GDPR and legal debates around privacy in Europe. On this basis we provide suggestions for what kind of concepts and tools are required to implement design justice in systems design and legal frameworks. We show that legal understandings of data justice can help generate data justice tools through regulatory frameworks, while feminist critique and design approaches can provide fruitful interventions towards more just information systems that take structural inequalities and context into account. Lastly, we discuss the extent to which data justice as a normative concept can be operationalised and highlight the potential challenges an interdisciplinary approach poses to the implementation of data justice. This includes questioning the limits of such implementations and tracing the convergences and divergences between feminist and legal approaches in regard to possible interventions towards data justice.

Ultimately, this chapter sets out to advance the design of data-driven computational systems that are most just for all. Our approach means rethinking basic assumptions of data justice from feminist and legal perspectives and suggests how this could impact IT-design and open up pathways towards more interdisciplinary approaches in critical data studies. Understanding data justice as multidimensional and interdisciplinary provides a conceptual ground that serves both the needs of legal formalisation and the feminist imperatives of contextualisation and specificity. While this chapter concludes with specific recommendations for design, its main aim is to rework the theoretical basis of where those recommendations should stem from.

We acknowledge that our perspective on data justice is unavoidably affected by our situatedness in a Western European educational institution and our observations pertain mostly to a European legal framework and justice-related issues that emerge in technology design and use.

“What’s in a Name?”—Data Justice as a Concept in Feminist and Legal Scholarship

Debates around digitalisation, its effects, and the process of digital technologies becoming essential infrastructure, form a backdrop to understanding and implementing data justice as a normative concept and as a practical concern. As a broader normative concept, data justice points to understanding data as necessarily situated in sociopolitical context and practices (Dencik et al., 2019). This expands concerns regarding data beyond the domains of security, privacy, and data protection to incorporate imbalances of power. This also accounts for effects of digitalisation and datafication and relating those to social justice, citizenship, and political participation (ibid.).

Different definitions of data justice point to different contexts within which data justice can be discussed and applied. Dencik et al. (2019) distinguish between academic/conceptual definition (data justice as analysis of data that pays specific attention to structural inequalities and the different implications of datafication for different parts of society), the design dimension (data justice as pertaining to design conditions and processes through which data infrastructures emerge), and activist definition (whereby data justice is employed at the intersection of political and social activism and technology that challenges the status quo).

As a normative concept, we propose an understanding of data justice that zooms in on the imbalances of power within IT-design and use and opens a space for deliberations on how such injustices could be ameliorated. Such a concept would incorporate both legal and design perspectives as well as provide space to include concerns around data collection and use as well as the design and use of data-intensive technological systems, such as algorithmic decision-making systems.

Feminist Accounts of Data Justice

Data justice points to power discrepancies in data-related systems. Feminist approaches to data generally argue that data is entangled and laden with politics, which feminist analyses and tools can unveil and intervene in (D’Ignazio & Klein, 2019). Here, both data and justice, as well as data justice, connect several important aspects. In “Data Feminism”, D’Ignazio and Klein argue that both the embodiment of data subjects and the effects of data on differently embodied people are important; that data are historical, contextual, and political as well as always processed and interpreted and that their presumed neutrality and objectivity should be interrogated; that what kind of data is collected, for which purposes, and how data is used is a political matter which should be critically discussed.

Such concerns regarding data orient justice towards embodied and contextualised understandings of it. This is particularly important not only for feminist but also anti-racist and social-justice oriented concepts. Anti-racist feminist legal scholar Kimberlé Crenshaw—following many black scholars and activists outside of a legal discipline such as Audre Lorde, bell hooks, Combahee River Collective, and others—using the metaphor of traffic intersections, coined the term “intersectionality” and pointed out that specific positions that index access to rights, resources as well as the possibilities of claiming justice are affected not by one but often by several social categories at the same time (1989). Intersectionality allows the rethinking of discrimination through multiple layers of meaning.

Following anti-racism activism and thought is particularly important for understanding social justice as well as data-related initiatives such as Data For Black Lives.Footnote 1 Among others, Eubanks (2018), Benjamin (2019), and Noble (2018) have pointed out that data inequalities and the lack of data justice have detrimental effects on communities of colour and other marginalised populations. People of colour are over-surveilled and generally over-served in terms of data accumulation for the purposes of evaluation and control, including predictive policing and other algorithmic prediction systems (Singelnstein, 2018; Thurn & Egbert, 2019; Hofmann, 2020), while they are simultaneously underserved when it comes to benefiting from algorithmic and data-driven systems (West et al., 2019; Hart, 2017; Yarger et al., 2019). Calls for justice from collectives such as Data For Black Lives or authors of Feminist Data Manifest-NoFootnote 2 centre contextualised justice that is attentive to differential conditions, impacts, and effects experienced by different populations, and that requires a broad scope of socio-culturally embedded, nuanced, and specific solutions that pertain to legal, policy, and design measures.

Legal Framework: Justice, Data, and the Challenges of Digitalisation

While data justice is debated within a feminist framework in relation to contextualisation and differential conditions, in law, as it pertains to digitalisation, the expression of justice is centred around two essential elements: the principle of equality and procedural justice. The principle of equality requires that the state treat things equally if they are the same or at least substantially the same. Despite many problems around the question of what is “substantially the same”, this notion of equality in Western legal tradition is probably the oldest and most agreed part of justice (Willoweit, 2012), dating back to ancient Greece; expressed in Ulpian’s Digest I. 1.10: “honeste vivere, neminem laedere, suum cuique tribuere”. Procedural justice assumes that if a transparent, inclusive procedure is used to make a decision, then that procedure ensures that a sufficiently fair and legitimate decision will be made (Luhmann, 2001).

Digitalisation has ambiguous effects on justice (Schliesky, 2019; Härtel, 2019). It could improve decision-making, leading to better and fairer decisions, if the decision-maker is provided with more data which is relevant for a decision, if there are new algorithms which are able to better deal with these relevant data (e.g. accelerate processing, visualise results or individualise data use through personalisation), or if there are new algorithms which help to distinguish between relevant and irrelevant data (e.g. addressing the problem of human biases).

But digitalisation poses new risks for legal concepts of justice and legal proceedings aiming at just decisions. This may be due to (1) enormous amounts of personal data about data subjects, specific group characteristics such as race, class, and gender, or data on individual behaviour (Hoffmann-Riem, 2018). This poses the risk of enabling decision-makers to discriminate (more efficiently) against specific persons, specific groups, or specific types of behaviour if they wish to do so or are unconsciously driven by prejudice. Big data analysis will also not be a tool for everybody, as those equipped with the necessary hardware and software will be able to derive additional knowledge (raising e.g. questions of antitrust law—Körber, 2016; Louven, 2018), leading to additional power imbalances (Taeger, 2019).

Digitalisation may (2) also create conditions for data-powered findings, which lead to the conclusion that certain characteristics are (statistically) connected with certain groups or behaviours. Depending on the level of statistical significance, this may also influence decisions, without encouraging investigation into possible deeper structural factors that led to such statistical connection and thus divesting attention away from structural solutions. These types of empirical findings as well as new algorithms used in current and future legal proceedings also carry (3) inherent risks of non-transparency. The difficulties with the interpretation, explainability, and transparency of algorithmic systems make it very difficult or at times impossible, at least within current legal procedures, to monitor the results of such systems.

Data Protection Law

Basic constitutional law principles such as democracy and the rule of law as well as fundamental rights need to be freshly concretised to better address the aforementioned risks (Unger & von Ungern-Sternberg, 2019). In the German legal tradition, concepts of democracy and justice have been connected to the fundamental right to self-determination since the population census decision of the German Federal Constitutional Court of 1983 (Bundesverfassungsgericht, 1983), whereas other legal systems have a concept of privacy that focuses solely on the individual and her “right to be let alone”, which was first described by Warren and Brandeis in 1890 and lacks the society-oriented part of its European counterparts (Klar & Kühling, 2016; Gusy, 2018).

Being enshrined in fundamental rights (e.g. Art. 20–26 of the Charter of Fundamental Rights of the EU, CFR), equality and fairness are also essential parts of the regulatory framework on personal data. Concepts of data justice may, then, build on existing regulations, namely the GDPR, the EU e-Privacy Directive, and applicable national laws. In particular, the GDPR has taken up the idea that data protection does not only protect personal privacy, but also autonomy and self-determination. At the same time, we need to address the loopholes which the non-regulation of non-personal data creates.

According to Art. 4 (1) GDPR, “personal data” means any information relating to an identified or identifiable natural person (the “data subject”); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier, or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural, or social identity of that natural person. This is a very broad concept (confirmed by the European Court of Justice, 2016) and the main reason for the ever-growing applicability of data protection law.

According to Art. 1 (2) GDPR, the regulation protects fundamental rights and freedoms of natural persons and “in particular their right to the protection of personal data”. While the latter refers to Art. 8 CFR, it is important to note that data protection law also protects other fundamental rights (Hornung & Spiecker gen. Döhmann, 2019), particularly equality before the law (Art. 20 CFR), non-discrimination (Art. 21 CFR), cultural, religious, and linguistic diversity (Art. 22 CFR), and equality between women and men (Art. 23 CFR), as well as the rights of the child and the elderly (Art. 24 and Art. 25 CFR, respectively). These rights must be considered when applying the GDPR, which is particularly relevant when there is a need for balancing conflicting interests.

Issues of non-discrimination also play a role in the protection of “special categories of personal data” (Art. 9 GDPR, including personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, and data concerning health or a natural person’s sex life or sexual orientation). Although experience shows that the sensitivity of personal data depends, to a large extent, on the purpose and circumstances of data processing (Simitis, 1990) and there is thus “no insignificant data anymore” (Bundesverfassungsgericht, 1983), the data categories in Art. 9 GDPR are specially protected as experience shows that there is a high risk of discriminating against people on this basis.

The notion of “procedural justice” is expressed in the GDPR by extensive procedural requirements for the processor. Following the principle of accountability (Art. 5 (2) GDPR), the processor is obliged to comprehensively inform the data subject, to implement appropriate technical and organisational measures to demonstrate compliance with the GDPR, to maintain records of processing activities, and to carry out a data protection impact assessment. The GDPR has considerably increased procedural requirements, calling for accountable and just data processing procedures as well as data protection by design and by default (Art. 25 GDPR).

The Justice Aspects of Non-personal Data

While non-personal data may be regulated by other fields of law, current data protection law has nothing to say on this type of data. At first glance this appears unproblematic as anonymity works as a shield against discrimination. This shield is however becoming weaker as big data analysis allows for (sudden or subtle) re-identification (Hornung & Wagner, 2019). Looking closer, identifiability is in no way a requirement for unjust decisions. It will often suffice to discriminate if the decision-maker or the ADM processes note the fact that people belong to certain groups or show certain attributes. While belonging to bigger anonymous groups may work as an effective shield against the risks of personalised data processing, the discrimination of these groups also poses risks for judicial remedy. At least in some cases it will be hard to prove that specific individuals are adversely affected. If equality laws do not include mechanisms of collective redress, these cases of “victimless discrimination” remain invisible to the legal system or at least not addressed properly (Lahuerta, 2018).

Furthermore, examples such as racial profiling (Angwin et al., 2016) and other developments within digitalisation, big data, and AI (Boehme-Neßler, 2008; Unger & von Ungern-Sternberg, 2019) have brought about new risks. Algorithmic personal pricing may be based on information about upscale IT equipment or behavioural analysis, teasing out the last bit of individual willingness to pay (Paal, 2019; Zuiderveen Borgesius & Poort, 2017). Using specific search terms (such as “maternity leave”) in internet search engines may add to personal profiles on which discriminatory decisions may be based without knowing the specific person that is being discriminated. Safeguards such as anonymous job applications may not protect against these mechanisms, as big data and AI algorithms could still produce power imbalances to the disadvantage of non-identifiable persons. In addition, many protected categories of personal data can be coded by proxy (Dwork et al., 2012; Barocas & Selbst, 2016), as is the case, for example, with ethnicity and postal codes or specific search terms and maternity status.

These cases show that belonging (or even only being assigned) to a certain group as well as showing a certain feature or behaviour suffices to enable unjust decisions. Current data protection law, therefore, may be an important building block in a regulatory data justice framework but there is a need to address these further issues as well.

Expanding Data Justice Through Feminist and Legal Perspectives

We have identified how (data) justice has been approached conceptually and normatively from feminist and social justice perspectives that highlight context, structural inequalities, and the differential distribution of the negative and positive effects of digitalisation. We showed that justice is put into practice in law through substantive justice (the notion of equality) and procedural justice, and how these concepts are put into place in the specific European context of data protection law. Furthermore, we identified that non-personal data can be a basis for causing injustice and discrimination. In this section, we delve deeper into possibilities regarding data justice from feminist and legal perspectives, namely what kinds of alterations to the understanding and implementation of data justice are offered by both.

Feminist Avenues Towards Rethinking Data Justice

Feminist scholarship has generated important critiques of systems of oppression and inequality. One major take is that the interlinking of social categories and power relations expands through societies concerning individual, structural, and symbolic dimensions (Harding, 1986). Feminist research problematises knowledge structures such as categorisation and classification (Bowker & Star, 1999), which relates to the above-mentioned ambiguous character of having and not having data. In this way, feminist critique is at least twofold: On an empirical level, it provides a lens that shows how social inequalities are interwoven in the society’s socio-economic-political fabric, making threads of injustice visible. This visibility is crucial for informing IT-design for social change (see section “An Overview of Technical Education, Higher Education, and Unemployability in India”). On a conceptual level, feminist thought highlights the productive and performative power of social categorisation itself. Cutting through these levels is a tension between relying on categories in order to point out injustices, criticising existing categories, expanding categorisations, and also challenging the very notion of stable categorisations (Benhabib et al., 1995).

This tension has been explored in feminist and queer legal thinking as it relates to perspectives on difference, equality, and justice. Feminist legal scholars have expressed varying positions on what constitutes gender equality in law, which is defined through gender neutrality or through gender specificity (Baer, 2011; Fineman, 2009). Both have faced critique: the former for the lack of acknowledgement of the different social, material, and cultural positions of women, while the latter is criticised for presenting universalist, essentialising, white- and Euro-centric views on women (ibid.). Queer scholars have also pointed out that any sort of categorisation fails those that fall outside of the normative definitions of legal subjects, as is often the case with transgender, intersex, and non-binary people (Spade, 2015). Relatedly, an ongoing debate in feminist and queer legal scholarship entails questioning the possibilities and the limits of identity-based discourses that focus on individual rights and binary categories of gender (Spade, 2015; Fineman, 2009).

Acknowledging that feminist thinking and feminist and queer legal theorising is diverse, we nonetheless highlight key avenues offered by research in these fields towards rethinking data justice:

Intersectionality: Intersectional analysis understands the position of a person to be the node of different socio-cultural categories (gender, race, class) that index their access to resources as well as their experiences of disenfranchisement (Crenshaw, 2019). Intersectionality links structural oppression, individual experience, and the symbolic order. Building upon black feminist activism, writing, and scholarship (see hooks, 1981, 1990), intersectionality as a concept informed (and informs) international policy making and discussions on human rights at the UN and for NGOs; historically, in countering violence against women of colour (Yuval-Davis, 2006). Intersectionality shows why making injustice visible while simultaneously interrogating social categories with which injustice is commonly addressed is so important. Intersectionality remains a crucial and still largely absent perspective that could contribute to understanding and designing more just information systems through the interrogation of intersecting structural causes, effects, and the processes of discrimination and oppression.

Contextualisation: Contextualisation refers to the understanding of social categories as interdependent with the context they are produced in or in which they have meaning (Lykke, 2010). Different contexts, situations, and locations hold different consequences depending on how a person is categorised (Davis, 1983; Evans & Lépinard, 2019). Justice is taken to acquire meaning in the broader context of social relations, histories of violence/oppression, and relations of power. This means that the definition of justice, or what might constitute just action, is context-dependent, re-orienting the definition of data justice towards social justice that is concerned with overall just and fair relations in society in regard to the distribution of wealth, power, and other resources (Gangadharan, 2020).

Relationality: Interconnectedness and relationality are inherent in understanding justice as social justice. Critical scholarship and activism (including feminist thinking—Sander-Staudt, 2016, de la Bellacasa, 2017, and disability justice activism and theory—e.g. Mingus, 2010), as well as feminist legal scholarship (see Hunter, 2018) have highlighted that viewing the subject as abstract, individual, independent, autonomous, and rational is tied to the history of Enlightenment and its formation of the modern state. This legacy has until relatively recently denied subjectivity to women and children, racialised, and naturalised subjects as well as disabled people (Braidotti, 2007), based on the idea that they lack the rational, individualist, and self-defining attributes of subjecthood. Feminist thought understands individuals as shaped by relations. This is particularly relevant when thinking about discrimination that arises not out of specific personal conditions but because of being ascribed to a certain community. Feminist legal scholars have explored how feminist epistemology can introduce transformative shifts in legal reasoning or methodology in general through stressing contextualisation and relationality (Baer, 1999; Szablewska & Bachmann, 2015).

Distributed relational responsibility and accountability: Contextuality and relationality already imply a reconfigured understanding of responsibility and accountability. Feminist philosophy, particularly posthumanist new materialism, conceptualises agency as relational and emerging from within specific situational contexts (Braidotti, 2013; Barad, 2007). For data justice this would mean regarding responsibility and accountability as embedded within IT-design and sociotechnical infrastructures (Draude, 2020; Busch, 2018). From this follows a relational understanding of accountability that—given the distributed character of data collection, processing, use—could help implement a more realistic and just take on accountability in information systems that acknowledges the role of design, ownership, and use, as well as the unequal power that different actors possess. This is important because distributed responsibility should not lead to an avoidance of accountability or the offloading of responsibility from corporations to users, for example. Data protection law has recently made steps towards incorporating this idea, stipulating in Art. 5 (2) GDPR that data controllers shall be responsible for, and be able to demonstrate compliance with the data protection principles in Art. 5 (1) GDPR (principle of accountability). Data protection by design is evenly tied to these principles in Art. 25 (1) GDPR, building a strong nexus between accountability and system design.

Deconstructing the private/public distinction: From a Euro-Western perspective the struggle to open up the private as a realm for political struggle has been at the core of feminism (e.g., Elshtain, 1993), for example, in cases of sexual and domestic violence—a field which for a long time was not addressed by the legal system because it was seen as a “private” domain, and which, to this day, often pressures women who raise claims of sexual harassment to present their personal sexual history as proof of character. Similarly, cases of online sexual harassment are often distributed along the lines of sexual and racial marginalisation (Shephard, 2016). However, the deconstruction of the public/private dichotomy when it also wants to do justice to people of colour and other marginalised communities must interrogate its context and history and relate to intersectional perspectives (Collins, 1991).

Feminist Suggestions for IT-Design Towards Data Justice

The concepts outlined above invite the re-positioning of justice towards a more structural, expanded, and social understanding of justice, contextualising it and tying it to questions of equality and fairness, beyond individual rights and towards the concerns of communities. This resonates with questions of algorithmic and data justice such as: How is access to just outcomes and just procedure different for communities? How does big data-based decision-making disproportionately affect already marginalised communities? In what ways does the understanding of privacy as an individual domain impede the understanding of discrimination and disenfranchisement through categorisation and inference? The feminist considerations outlined above can provide a conceptual reorientation of data justice. Here, we provide some recommendations on how these conceptual reorientations can be applied towards the implementation of data justice as a pragmatic approach within IT-design. The idea is to not create foolproof design approaches but, rather, to treat data justice as a normative direction, along which processes of technology design could be refocused in order to design more accountable and contextualised systems.

First, the contextualisation of justice towards intersectional and social justice would allow those involved to pay attention to both issues around non-personal data and related power imbalances, and to ways in which collective decision-making and the power of self-determination could be enhanced. The diversification of data sets is often recommended to counter social bias in ADM and machine learning systems (Zou & Schiebinger, 2018). Intersectionality brings to the fore the challenges of such diversification. One problem of dealing with data and intersectionality is data disaggregation. According to Crenshaw (1989), the segmentation of discrimination has a counter-beneficial effect for women of colour. Since a lot of data is gathered or broken down into separate categories, intersectional positioning in the world is not represented. Here, the explicit sampling of intersections could be recommended. If diversification of data is pursued it is not enough to bring in diverse groups or features. Instead, the active oversampling of marginalised groups is recommended. Also, while the diversification of data might enhance the performance of the machine learning system, how people are affected by the system outcomes or what it means to become visible in a data set must also be considered.

This leads to the second point: To design more accountable sociotechnical systems it is important to situate the information systems and their design processes (Draude et al., 2020). This means that it is important to understand such systems as embedded in political, social, and other relevant contexts as well as in structural power relations. Draude et al. propose several guidelines for how such situatedness could be achieved. One of them is the systematic and iterative attention to a “4P set of questions”, which stands for people affected and involved; place, for example, of data collection, of use, of affected areas; power relations; and participation of human/non-human actors (ibid., pp. 336–337).

Third, for the implementation of feminist values into data systems, approaches that question power relations and foster collaboration are promising. Participatory Design (PD) has a long tradition in information systems and human-computer interaction (Simonsen & Robertson, 2013; Sundblad, 2011). In contrast to its historical origin focusing on industry workers, today’s PD approaches are mostly used in local, often smaller-scale projects addressing community building, neighbourhood projects with local youth or the elderly as user groups. Inherent to PD is interrogating power setting in the development process, such as between developer and user, democratic and emancipatory values, and participation throughout all stages of development, ideally by all people affected. To meet challenges brought upon by AI and big data, existing PD research and methodology must be updated. This is increasingly recognised in computing (Bannon et al., 2018). For data justice, PD offers the potential to transfer the claim of not just having a voice but, furthermore, of “having a say” (Kensing & Greenbaum, 2013) in overall system design to the new challenges brought about through (big) data collection, gathering, labelling, use, and storage. Juliane Jarke (2019a, 2019b), in her work on co-creating digital public services with older adults, has shown how participatory practices can be employed for data curation, self-determination in data ownership, and the shifting of power relations through collaborative methods such as “data walkshops”.

In computing, norms, standardisations, and algorithms have so far acted as processes of exclusion when it comes to implementing social diversity. But they also provide the gateway for bringing in normative claims (Friedman & Kahn Jr, 2003; Roßnagel et al., 2018), such as non-discrimination or gender equality. To be implementable in information system development, critical knowledge has to be made operationalisable. Some approaches to design or to software engineering have taken this up. To name just two: Anti-Oppressive Design translates Patricia Hill Collins’ racial justice work into a framework for HCI (Smyth & Dimond, 2014); the Gender-Extended Research and Development Model (GERD) takes up intersectional gender studies and reconstructs software engineering cycles enriched with feminist reflections and examples (Draude & Maaß, 2018).

To sum up, feminist perspectives offer some promising avenues for implementing data justice through conceptual and design means. Data justice here gets reconfigured as a more contextual, situated, and systemic approach to justice. These conceptual considerations can then be translated into systems design approaches that prioritise contextualisation through situating, participatory design methods and methods that are sensitive to power relations. These approaches are both resonating as well as contrasting with specifically legal approaches. In the next section, we will explore how data justice can be intervened into and implemented from a legal perspective in the current legal framework, particularly within the GDPR.

Legal Interventions for Data Justice in the Current Legal Framework

While feminist scholarship allows for structural design interventions, applying existing laws first of all requires the utilisation of data justice as a pragmatic and normative tool. In the GDPR, data justice questions play out regarding having/using and not having/using specific personal data. The GDPR provides space for interpretation when it comes to data justice questions, and it is, therefore, important to discuss how the GDPR and the tools that it offers could be used.

First, it is possible to understand several provisions of the GDPR as being duties to respect personal autonomy.Footnote 3 Art. 5 (1) (a) GDPR emphasises that personal data shall be processed “fairly”. Little use has been made of this requirement so far (Roßnagel, 2019), but it could work as a gateway to incorporate notions of justice, equity, and equality—in particular, factoring in vulnerable groups and social inequalities. As there is so far no legal precedent at all, the term “fairly” offers space for interventions, including perceptions from feminist research on IT-design (cf. above). To this end, there is the need to translate into legal practice an understanding of intersectionality (and intersectional discrimination) as well as the relationality and situatedness this brings. This could mean giving up an absolute, fixed identity-based concept of discrimination in favour of understanding discrimination as related to location, context, time, and changing with societal circumstances.

Autonomy is also emphasised in the definition of the data subject’s consent in Art. 4 (11) and Art. 7 GDPR. Such consent is only valid if it is a freely given, specific, informed, and unambiguous indication of the data subject’s wishes, prohibiting pre-checked checkboxes which the user must deselect to refuse her consent (European Court of Justice, 2019). Feminist scholarship has established the concept of “relational autonomy” (Mackenzie & Stoljar, 2000). As elaborated upon in section “Introduction”, feminist thought emphasises contextualisation and relationality. The data subject’s consent, the capability for self-determination and assessing the impact of one’s consent become highly challenging in times of complex, networked, distributed information systems.

Data justice also relates to and is operationalised through several rights that are detailed in the GDPR, namely in the rights of the data subject to access (Art. 15), rectification (Art. 16 GDPR), erasure (Art. 17 GDPR), restriction of processing (Art. 18 GDPR), data portability (Art. 20 GDPR), and object (Art. 21 GDPR). The right to data portability, an important innovation of the GDPR (de Hert et al., 2018), provides for the right of the data subjects to receive the personal data concerning them which they have provided to a controller, in a structured, commonly used, and machine-readable format and the right to transmit those data to another controller without hindrance by the controller to which the personal data have been provided. Aiming at reducing lock-in effects, particularly in applications with high network effects, could in the future be a powerful tool to decrease the data power of these providers. As already mentioned, the GDPR considers the sensitivity of the data to be processed (Art. 9 GDPR). The Regulation also addresses the risks of automated individual decision-making, including profiling, in Art. 22 GDPR (Malgieri, 2019). Such decision-making shall only take place upon explicit consent of the data subject, following a contract with her or if authorised by Union or Member State law, if this law lays down suitable measures to safeguard the data subject’s rights and freedoms, and legitimate interests.

Several procedural requirements should also be noted. The rights of the data subject are tools for procedural interventions for the data subject. The general principle of transparency (Art. 5 (1) (a) GDPR) is specified in many new and extended provisions. This is not restricted to reactive duties, such as granting access to data upon the request of the data subject. There are also obligations to proactively inform the data subject (in general, before processing is initiated) in Art. 12 ff. GDPR, and to notify personal data breaches to both the supervisory authority and the data subject (Art. 33, 34 GDPR). These duties are of utmost importance because transparency is a conditio sine qua non for almost every other right: Without knowing the controller and the details of the data processing, data subjects will not feel the need to control the actions of powerful controllers. As the German Federal Constitutional Court put it in its famous population census decision of 1983 (cf. Hornung & Schnabel, 2009), if individuals cannot oversee and control which or even what kind of information about them is openly accessible in their social environment and if they cannot even appraise the knowledge of possible communication partners, they may be inhibited in making use of their freedom.

Other procedural requirements form direct interventions within organisations processing personal data. Each controller must maintain a record of processing activities (Art. 30 GDPR), including the purposes of the processing, the categories of personal data and recipients, and even, where possible, the envisaged time limits for erasure. Art. 32 GDPR contains the duty to implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk evocated by data processing. If that risk is likely to be high, Art. 35 GDPR imposes the duty to carry out a data protection impact assessment, including measures envisaged to address the risks (Bieker et al., 2016; Raab, 2020). Data protection officers (Art. 37, 38 GDPR) are important tools of mandatory internal self-control. The GDPR has also strengthened self-regulation in data protection law by voluntary codes of conduct (Art. 40, 41 GDPR), and certification (Art. 42, 43 GDPR). All in all, these requirements enforce, and the voluntary instruments offer important self-learning mechanisms for controllers and processors.

Towards Operationalising Data Justice

Following the aim of protecting fundamental rights of equality (Art. 1 (2) GDPR, cf. above), the idea of data justice—including the knowledge social sciences and humanities can add to this concept—should be kept in mind when interpreting the GDPR, particularly its sweeping clauses. Examples include the principle that personal data shall be processed “fairly” (Art 5 (1) (a) GDPR), the question of if there are “interests or fundamental rights and freedoms of the data subject” which override legitimate interests pursued by the controller (Art. 6 (1) (f) GDPR), the “appropriate measures to provide information” (Art. 12 (1) GDPR), the implementation of “suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests” when automated decision-making takes place (Art. 22 (3) GDPR), and every provision which forces controllers or processors to assess the specific risk of the data processing (e.g. appropriate technical and organisational measures, Art. 24 (1) GDPR; data protection by design, Art. 25 (1) GDPR; security of processing, Art. 32 (1) GDPR; data breach notification, Art. 33, 34 GDPR; data protection impact assessment, Art. 35 GDPR).

Our general assumption is that there are sufficient possibilities to address issues of data justice within the current legal framework. The main task will be to make decision-makers familiar with this idea, including the knowledge provided by feminist research, such as the specific discriminatory risks that intersectionality brings forth (cf. section “Introduction”) as well as the broader understanding of data justice including structural policy measures and best practices for technology design. Data protection practitioners could profit from the feminist perceptions that any realistic concept of discrimination must include not only individual but also structural understandings of injustice—and that, following the idea of intersectionality and situatedness, we need to shift the understanding of a person’s identity as a fixed position in society towards a more interactive, dynamic understanding.

Particular attention should be drawn to issues of profiling, which have been identified as being particularly risky not only for the individual, but also from the perspectives of social inequalities and social justice (Büchi et al., 2020). Furthermore, industry ethical standards as well as possible policy-oriented regulations could take the aforementioned legal as well as more conceptual feminist re-framings of data justice into account.

Some of these considerations are already implemented through ideas such as addressing quality and diversity of data as well as employing various sociotechnical design approaches. Nonetheless, a more explicit challenge to stark structural inequalities related to digitalisation could be considered as normative elements of data justice that could play a role in industry standards. Future policy making should also include issues of intersectionality, perhaps. Through procedural requirements for the integrated work of the respective bodies supervising issues of sectional equality (such as gender equality officers and disabled-employee officers).

Considering data justice and the rights of the data subject, it seems unclear how effective they are in practice. This is related to the general problems of the respective rights (such as procedural obstacles, unwillingness of controllers and processors to answer and to react to complaints), but also to issues of group-related discrimination, which “law in the books”-thinking is not able to address. Data protection rights form a comprehensive set of claims that will work as valuable tools for those who are fit and willing to fight against unjust processing and discriminatory effects. Those who lack these capabilities or who are (or feel) deterred because of their situation may face considerable difficulties in doing so.

While supervisory authorities usually report to the public the number of complaints they receive in their activity reports (Art. 59 GDPR), there is no information about the sociodemographic characteristics of those making use of this right. Regarding the rights against controllers and processors (access, rectification, erasure, restriction of processing, data portability, and object), there is not even any general statistics. There also appears to be no research at all on the question of which groups use their data protection rights in practice and whether the current procedural rules and requirements disfavour vulnerable groups.

The idea of making a data subject’s rights workable for everyone also calls for digital literacy and education, a task that yet again poses problems of equally addressing different social groups. Data controllers should be urged to commit themselves to implementing measures to make rights workable particularly for vulnerable groups. These measures should then be subject to review by the supervisory authorities, as they have, inter alia, the tasks of promoting public awareness of data processing risks (Art. 57 (1) (b) GDPR) and providing information to data subjects concerning the exercise of their rights (e). In fulfilling these tasks, it would be possible to connect to representatives from vulnerable groups and enable forms of citizen participation.

Ideally, these activities should—together with research on the implementation of data subject’s rights—lead to specific best practices and technological tools that are usable and affordable (for all) and enable to effectively exercise the respective rights. The need for such tools will become even more urgent, as ADM poses serious additional risks for the practical use of rights aiming at transparency. Best practices and tools could later form part of codes of conduct for specific processing sectors (Art. 40 GDPR).

The GDPR has seriously increased the duties of controllers to assist data subjects: Art. 12 (1) GDPR stipulates that the controller shall take appropriate measures to provide any information and communication in a concise, transparent, intelligible, and easily accessible form, using clear and plain language. According to Art. 12 (2) GDPR, the controller shall also “facilitate the exercise of data subject rights under Articles 15 to 22”. This obligation could be interpreted to include a duty of controllers who implement ADM systems to also ensure that they implement effective technical tools for data subjects who want to make use of their rights.

Regarding applicable law, data protection law is restricted to personal data. Even considering the wide definition given in Art. 4 (1) GDPR, many cases of using data and the power of algorithmic decision-making are not covered by the GDPR. When considering appropriate possibilities for addressing this issue, regulators may be able to learn from the instruments data protection law provides—an approach which the German data ethics commission (Datenethikkommission, 2019) appears to follow in its expert opinion of 2019. Using justice and solidarity as ethical and legal principles (pp. 46 f.), it attempts to formulate data rights and corresponding data responsibilities (Datenrechte und korrespondierende Datenpflichten), arguing that a right to digital self-determination (Recht auf digitale Selbstbestimmung) should apply to legal persons as well as collective groups (85 ff.).

Many rules in the GDPR relate to specific data subjects and may not be used where those subjects do not exist. The general notion of a data subject’s rights could however be transferred, for example, by granting persons affected by automated anonymous decisions a right not to access their personal data, but to access the algorithms and training rationales of AI decision-making. Following the idea that clandestine data power is particularly dangerous for data justice, new laws for mandatory pro-active transparency could be introduced as well. A recent example is the duty to inform consumers before distance and off-premises contracts about the fact that the price was personalised based on automated decision-making.Footnote 4 Such rules surrounding transparency could also include feminist research’s critique that social inequalities and injustice are often not visible enough (cf. section “Introduction”).

Many procedural instruments are also transferable to non-personal data and its algorithmic use. Records of processing activities could enhance transparency and allow for external administrative and judicial control. “Data justice impact assessments” or “AI impact assessments” could initiate the awareness within organisations, particularly in regard to the impact on vulnerable groups. There could even be an obligation to involve and consult representatives of these groups. Such participatory approaches would not only be a way of taking into account suggestions for inclusive participatory design and co-creation (cf. section “State of Research: The QS and Maker Movements’ Organisational Elites”), but also strengthen the legitimacy of design decisions. In bigger organisations, an ombudsman (“data justice protection officer”) could oversee risky data processing and serve as a contact person for those affected by automated decision-making.

Certification and audits could be used to demonstrate compliance of existing products and procedures—either voluntary or in the case of risky algorithms and use cases, compulsory. They may disburden individuals from having to understand IT systems and data processing structures in detail in order to understand risks and personal implications (Hornung & Hartl, 2014; Rodrigues & Papakonstantinou, 2018; Hornung & Bauer, 2019). Given the ever-increasing complexity of ADM, this preventative control by experts will become more important and could again strengthen accountability. It could also become part of a wider system of external control, including elements like an obligation to disclose training data.

Potentials and Limitations of Integrating Feminist and Legal Perspectives for Data Justice in IT-Design

In lieu of a conclusion, we follow the potentials and limits of integrating and operationalising feminist and legal perspectives towards data justice in IT-design and what this means for critical data studies. Instead of combining the multiple perspectives provided into one unified framework, we map data justice as a multidimensional concept that has its normative-legal and pragmatic-design aspects, both of which can be intervened in from the perspectives of feminist and legal scholarship. This we see as both an advantage and a limitation. It is a potential drawback because a unified normative framework would be useful for IT-system design. On the other hand, the lack of such a unification allows for both interpretative flexibility regarding the concept of data justice as well as space for specific, situated, and contextualised translations of what data justice might mean in concrete cases, communities, and situations.

Feminist perspectives push the concept of (data) justice away from universalism and towards relational contextualised justice. This repositions data justice as a more systemic understanding of justice where the tools needed to ensure data justice are also systemic and broad, encompassing pragmatic legal interventions but also measures for policy and design requirements. The legal perspective meanwhile operates within the idea of a universal, formal definition of justice that ensures the broadest common denominator of justice for all, resonating with similar ideas of universality and broad applicability in IT-design.

This tension can be productive and does not necessarily need to be immediately resolved. Both perspectives are needed, particularly because they can be operationalised differently. The former, more conceptually inclined, feminist perspective can be particularly well suited for instigating broader conceptual and structural change when it comes to understanding what data justice might mean and also what should be included within normative and value-based considerations in technology design. More pragmatic legal approaches instead point to possible changes in the existing legal framework and can perhaps be integrated more easily with formal and model-based approaches prevalent in design.

In this chapter, we have outlined the broad conceptual changes needed with a feminist perspective, more concrete interventions from a legal perspective, and what kind of implications they both might bear on IT-design. Nonetheless, some questions remain that are significant both for IT-design and critical data studies as an interdisciplinary research field that purports to not only research but also to intervene in data-based systems.

First, how can we envision what “data protection” and “data justice” in IT-design entails? For one, this requires the more precise investigation into how the translations can be made between a more conceptual normative level and the more pragmatic levels of design—or put differently, what norms actually entail and how normative aspects are realised in IT-design. There is so far no common European understanding in regard to the exact content of the fundamental right to data protection enshrined in Art. 8 CFR (Marsch, 2018). This means that besides the new methods of information system design, fundamental rights innovations (Hornung, 2015) could also play a role in the shaping of data justice. Here feminist legal scholarship is still important, as it continues to investigate what “gender equality”, “equity”, and “privacy” might mean for different legal subjects.

Second, and relatedly, the question remains: At which points might we need to put effort into translating feminist critique (Simitis 1990) into formalisable and generalisable legal regulations? This is not a new question but has been explored by feminist computer scientists as well as feminist and queer legal scholars as pointed out earlier. Nonetheless, it highlights the need to investigate the conceptual possibilities of “translation” as well as existing best practices and to provide the space for activist voices and the voices of those constituencies that are directly suffering from “data injustice”.

Last but not least, it is important to note that there already exists a plethora of sociotechnical approaches that can be used in designing more just data systems (such as participatory design). However, they are employed relatively rarely. This means that the question of how to make sure that feminist and legal justice-oriented design recommendations (particularly if they are not formalised in law) are taken up more extensively remains open as well.

To conclude, new perspectives in critical data studies require closer interdisciplinary collaborations. It is not only the analysis but pragmatic interventions that can originate in critical data studies that require understanding and work into the tensions that interdisciplinary perspectives bring. Our invitation is to embrace those perspectives and continue to expand their reach through possible structural regulation, policy regulations, voluntary industry standards, and other measures, with the hope that these different approaches can be brought into “strategic resonance” with each other: a kind of resonance that leaves space for tensions as sources of conceptual possibility. We hope that this chapter contributes to the creation of such a methodological and conceptual open space for considerations of data justice and interdisciplinary approaches in critical data studies.