REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets

Abstract

Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. Our work tackles this issue and enables the preemptive analysis of large-scale datasets. REvealing VIsual biaSEs (REVISE) is a tool that assists in the investigation of a visual dataset, surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based. Object-based biases relate to the size, context, or diversity of the depicted objects. Person-based metrics focus on analyzing the portrayal of people within the dataset. Geography-based analyses consider the representation of different geographic locations. These three dimensions are deeply intertwined in how they interact to bias a dataset, and REVISE sheds light on this; the responsibility then lies with the user to consider the cultural and historical context, and to determine which of the revealed biases may be problematic. The tool further assists the user by suggesting actionable steps that may be taken to mitigate the revealed biases. Overall, the key aim of our work is to tackle the machine learning bias problem early in the pipeline. REVISE is available at https://github.com/princetonvisualai/revise-tool.

Appendices

Appendices

1.1 Gender Label Inference

An additional person-based metric we consider is gender label inference. Specifically, we note two especially concerning practices of assigning gender to a person who is too small to be identifiable, or no face is detected in the image. This is not to say that if these cases are not met it is acceptable to assign gender, as gender cannot be visually perceived by an annotator, but merely that assigning gender when one of these two cases is applicable is a particularly egregious practice. For example, it’s been shown that in images where a person is fully clad with snowboarding equipment and a helmet, they are still labeled as male (Burns et al., 2018) due to preconceived stereotypes. We investigate the contextual cues annotators rely on to assign gender, and consider the gender of a person unlikely to be identifiable if the person is too small (below 1000 pixels, which is the number of dimensions that humans require to perform certain recognition tasks in color images Torralba et al., 2008) or if automated face detection (we used Amazon Rekognition (“Amazon Rekognition” , n.d.), but note that any other face detection tool can be used) fails. For COCO, we find that among images with a human whose gender is unlikely to be identifiable, 77% are labeled male. In OpenImages,⁵ this fraction is 69%. Thus, annotators seem to default to labeling a person as male when they cannot identify the gender; the use of male-as-norm is a problematic practice (Moulton, 1981). Further, we find that annotators are most likely to default to male as a gender label in outdoor sports fields, parks scenes, which is 2.9x the rate of female. Similarly, the rate for indoor transportation scenes is 4.2x and outdoor transportation is 4.5x, with the closest ratio being in shopping and dining, where male is 1.2x as likely as female. This suggests that in the absence of gender cues from the person themselves, annotators make inferences based on image context. In Fig. 17 we show examples from OpenImages where our tool determined that gender definitely should not be inferred, but was. Because attributes like skin tone can be inferred from parts of the image, such as a person’s arm, we do not consider that attribute in this analysis.

This metric of gender label inference also brings up a larger question of which situations, if any, gender labels should ever be assigned (Scheuerman et al., 2020; Hamidi et al., 2018). However, that is outside the scope of this work, where we simply recommend that dataset creators should give clearer guidance to annotators, and remove the gender labels on images where gender can definitely not be determined. We note that while we picked out two criteria of when a person is too small and when there is no face detected to be instances in which gender inference is particularly egregious, there are many other situations that users may wish to delineate for their own purposes.

Fig. 17

Examples from OpenImages where annotators assigned gender to the person, but they should not have. The criteria used are that the person is either too small or has no face detected

1.2 Validating Distance as a Proxy for Interaction

In Sect. 5.1, Instance Counts and Distances, we make the claim that we can use distance between a person and an object as a proxy for if the person, p, is actually interacting with the object, o, as opposed to just appearing in the same image with it. This allows us to get more meaningful insight as to how genders may be interacting with objects differently. The distance measure we define is $dist = \frac{\text {distance between p and o centers}}{\sqrt{\text {area}_{\mathrm {p}}*\text {area}_{\mathrm {o}}}}$, which is a relative measure within each object class because it makes the assumption that all people are the same size, and all instances of an object are the same size. To validate the claim we are making, we look at the SpatialSense dataset (Yang et al., 2019); specifically, at 6 objects that we hope to be somewhat representative of the different ways people interact with objects: ball, book, car, dog, guitar, and table. These objects were picked over ones such as wall or floor, in which it is more ambiguous what counts as an interaction. We then hand-labeled the images where this object cooccurs with a human as “yes” or “no” based on whether the person of interest is interacting with the object or not. We pick the threshold by optimizing for mean per class accuracy, where every distance below it as classified as a “yes” interaction and every distance above it as a “no” interaction. The threshold is picked based on the same data that the accuracy is reported for.

Table 4 Distances are classified as “yes” or “no” interaction based on a threshold optimized for mean per class accuracy

As can be seen in Table 4, for all 6 categories the mean of the distances when someone is interacting with an object is lower than that of when someone is not. This matches our claim that distance, although imperfect, can serve as a proxy for interaction. From looking at the visualization of the distribution of the distances in Fig. 18, we can see that for certain objects like ball and table, which also have the lowest mean per class accuracy, there is more overlap between the distances for “yes” interactions and “no” interactions. Intuitively, this makes some sense, because a ball is an object that can be interacted with both from a distance and from direct contact, and for table in the labeled examples, people were often seated at a table but not directly interacting with it.

1.3 Pairwise Queries

In Sect. 4.2, another claim we make is that pairwise queries of the form “[Desired Object] and [Suggested Query Term]” could allow dataset collectors to augment their dataset with the types of images they want. One of the examples we gave is that if one notices the images of airplane in their dataset are overrepresented in the larger sizes, our tool would recommend they make the query “airplane and surfboard” to augment their dataset, because based on the distribution of training samples, this combination is more likely than other kinds of queries to lead to images of smaller airplanes.

However, there are a few concerns with this approach. For one, certain queries might not return any search results. This is especially the case when the suggested query term is a scene category, such as indoor cultural, in which the query “pizza and indoor cultural” might not be very fruitful. To deal with this, we can substitute the scene category, indoor cultural, for more specific scenes in that category, like classroom and conference, so that the query becomes something like “pizza and classroom”. When the suggested query term involves an object, there is another approach we can take. In datasets like PASCAL VOC (Everingham et al., 2010), the set of queries used to collect the dataset is given. For example, to get pictures of boat, they also queried for barge, ferry, and canoe. Thus, in addition to querying, for example, “airplane and boat”, one could also query for “airplane and ferry”, “airplane and barge”, etc.

Fig. 18

Distances for the objects that were hand-labeled, orange if there is an interaction, and blue if there is not. The red vertical line is the threshold along which everything below is classified as “yes”, and everything above is classified as “no”

Fig. 19

Screenshots of top results from performing queries on Flickr that satisfy the tags mentioned. For train, when it is queried with boat, the train itself is more likely to be farther away, and thus smaller. When queried with backpack, the image is more likely to show travelers right next to, or even inside of, a train, and thus show it more in the foreground. The same idea applies for pizza where it’s imaged from further in the background when paired with an indoor cultural scene, and up close with broccoli

Fig. 20

Screenshots of top results from performing queries on Flickr that satisfy the tags mentioned. For bed, sink provides a context that makes it more likely to be imaged further away, whereas cat brings bed to the forefront. The same is the case when the object of interest is now cat, where a pairwise query with sheep makes it more likely to be further, and suitcase to be closer

Another concern is there might be a distribution difference between the correlation observed in the data and the correlation in images returned for queries. For example, just because cat and dog cooccur at a certain rate in the dataset, does not necessarily mean they cooccur at this same rate in search engine images. However, our query recommendation rests on the assumptions that datasets are constructed by querying a search engine, and that objects cooccur at roughly the same relative rates in the dataset as they do in query returns; for example, that because train cooccurring with boat in our dataset tends to be more likely to be small, in images returned from queries, train is also likely to be smaller if boat is in the image. We make an assumption that for an image that contains a train and boat, the query “train and boat” would recover these kinds of images back, but it could be the case that the actual query used to find this image was “coastal transit.” If we had access to the actual query used to find each image, the conditional probability could then be calculated over the queries themselves rather than the object or scene cooccurrences. It is because we don’t have these original queries that we use cooccurrences to serve as a proxy for recovering them.

To gain some confidence in our use of these pairwise queries in place of the original queries, we show qualitative examples of the results when searching on Flickr for images that contain the tags of the object(s) searched. We show the results of querying for (1) just the object (2) the object and query term that we would hope leads to more of the object in a smaller size, and (3) the object and query term that we would hope leads to more of the object in a bigger size. In Figs. 19 and 20 we show the results of images sorted by relevance under the Creative Commons license. We can see that when we perform these pairwise queries, we do indeed have some level of control over the size of the object in the resulting images. For example, “pizza and classroom” and “pizza and conference” queries (scenes swapped in for indoor cultural) return smaller pizzas than the “pizza and broccoli” query, which tends to feature bigger pizzas that take up the whole image. This could of course create other representation issues such as a surplus of pizza and broccoli images, so it could be important to use more than one of the recommended queries our tool surfaces. Although this is an imperfect method, it is still a useful tactic we can use without having access to the actual queries used to create the dataset.⁶