In this section, we share the different “lessons learned” from the project development, specifically in regard to how we engaged with diverse Wikipedia communities to design solutions that addresses the needs of any Wikipedian willing to bridge the content gaps.
In the first subsection, we explain the general community needs we identified throughout the different interactions with the Wikimedia community members (6.1). In the second subsection, we detail the insights we obtained from each of the main iterations that have been fundamental to extend the data framework and build the tools and visualizations (6.2). Finally, in the third subsection, we reflect on the development trade-offs and limitations implicit to the approach we followed (6.3).
Community engagement is presented as one central point in each of the two intertwined processes, as it is a phase of validation of the current iteration and definition of the following one. This phase involves presenting the new research findings, tools, and visualizations in various formats as varied as a conference talk, an edit-a-ton workshop or video-call interviews. In the past 2 years, there has been a good deal of dissemination events.Footnote 46 In return, it is expected that other community members express their opinions, interests, understandings, and experiences to nurture the Diversity Observatory discourse or tools.
Each of these interactions and collaborations with other community members is essential to collect feedback in the same exact scenarios in which the tools will be used. Requests for features or improvements on the User Interface have been common. Other community members’ expectations are usually set on learning and improving the tools, but always in the most transparent and incremental manner - the “wiki-way”.Footnote 47 Community engagement has both the function of “generative research” and “evaluative research” . Generative to find new opportunities (e.g., requests for inclusion of new diversity categories or new features and visualizations), and evaluative, to validate current elements of the tools and refine the current approach.
This latter is essential at the levels of clarifying the discourse (framing the problems and simplifying the language), prioritizing some analysis (showing the key aspects of the problem), and making the tooling (improving its functionalities and usability). The feedback provided at each interaction is documented and shared across the different active members of the Observatory. The repetition of themes, requests, or concerns and the complexity and associated costs are key to decide the focus of the following iteration.
From the different conversations all over these years, we have identified that other community members needs can be classified into five different groups:
Understanding the situation and progress for a specific kind of content gap (situation and progress)
Being able to compare different Wikipedia language editions’ coverage of some kind of content gap (comparison)
Obtaining discourse and graphical material to explain it to others (communication)
Distinguishing the value in a content gap, i.e., which article should I create (prioritize)
Bridging content gaps more easily and being more efficient at it (efficiency)
The project Wikipedia Diversity Observatory has been developed in three different phases since 2018. We will briefly explain the outcomes presented in each of these iterations, the venues where they were presented and asked for community engagement, and the most common feedback received during these interactions.
First iteration: culture (2018–2019)
In this first iteration, the project was focused on studying Cultural Diversity (its original name was Wikipedia Cultural Diversity Observatory). The goal was to collect the “local content” of each of the more than 300 Wikipedia language editions and create a tool to retrieve lists of relevant articles. In this way we aimed to validate the research results from previous studies [12, 13] with more language editions and raise awareness on the need for increasing cultural diversity, providing some quantitative indicators as it was done for the gender gap.
In the first community interactions, we presented some data tables, simple visualizations, and the first version of the “Top CCC” lists in the regional African communities’ conference WikiindabaFootnote 48 (Tunis, Tunisia), the Central and Eastern European communities’ conference Wikimedia CEEFootnote 49 (Lviv, Ukraine) and WikimaniaFootnote 50 (Cape Town, South Africa).
Website approach validation
The idea of dividing the website into “visualizations” and “Tools” was rapidly validated by community members. This way, the first would address the needs of understanding the situation and progress, allowing for comparison and providing evidence, while the second would be focused on organizing action by providing lists of high-priority articles.
Top CCC lists usefulness
The Top CCC articles lists were embraced by African communities, given that the lists of relevant articles (e.g., Vital articlesFootnote 51 or List of articles every Wikipedia should haveFootnote 52) manually created by Wikipedians in English Wikipedia or Meta-wiki tend to over-represent the Western world.
By having lists of relevant articles according to different criteria but always centred on each cultural context, it was now possible to convince every other language to cover cultural diversity more easily. The discovery was that 127 Wikipedia language editions did not even contain 100 articles geolocated in their corresponding territories or 100 articles on their cultural context.
Exporting the lists
The Top CCC article lists were used in the annual contest CEE SpringFootnote 53 (March, 2019) to select the articles for the 2019 edition. The contest encourages the different languages of the region to create articles about each other’s context, including geography, culture, and people. Some Wikipedians pointed out the need for flexibility in providing ways to export the lists to Wikitext or Excel format, so that Wikipedians are able to use them within their daily work.
These were easy upgrades to the tool that were addressed along with some new lists. The Observatory was used to create lists of articles for every country from which participants could choose.Footnote 54 Rather than directing participants to the website, they preferred having the lists integrated as tables in the contest page in the Meta-wiki.
Creating “Common CCC article lists”
One of the participants to the CEE Spring 2019 contest requested to have a list of articles including common aspects of the whole region or at least common to a few countries. This gave place to the tool “Common CCC”,Footnote 55 which allows searching for articles belonging to the local content of two or more language editions at once.
Monitoring changes over time
Community members who participated in the CEE Spring 2019 and in Intercultur 2019Footnote 56 found the analyses of cultural diversity coverage as something interesting, but observed that without the possibility of monitoring their evolution over time, the metrics were not encouraging progress.
Integrating further categories
Among all the Wikipedians who used the tools and were inquired to give their opinion about the project, there was consensus on the request for understanding the intersection between gender and cultural contexts, as well as other topics.
Second iteration: gender, geography and minoritized languages (2019–2020)
In the second iteration, the main goal was to address the temporal dimension of the analyses and to expand the data for including gender and geography. We improved the repertoire of visualizations in the Observatory, adding “Diversity Over Time” and dedicated dashboards on gender and geography.
In matters of discourse, we made an important effort to disseminate the value of contributing “local content” and not only cover content from other language editions. In this sense, we presented guidelinesFootnote 57 giving examples and reasons to create “local content,” a chapter on its importance that would be published in the Wikipedia 20 anniversary book.Footnote 58
Creating the “Missing CCC dashboard”
One of the lessons learnt from the previous iteration was that many language editions do not cover their own local content. This inspired the creation of the Missing CCC articles lists, which provide articles about a language’s cultural context that exist in other Wikipedia language editions. These articles lists were very suitable to Indian Wikipedias, which in 2019 organized a contest named “Project Glow”Footnote 59 (formerly Project Tiger). This contest originally used lists of topics provided by Google that were generated using the list of local queries in the search engine.Footnote 60 However, sometimes the resulting topics could be considered popular (e.g., a smartphone new model) but not necessarily relevant to their context. The Missing CCC articles lists created in 14 Indian languages were provided to the participants to help them identify relevant missing articles that existed in English or other larger Wikipedias.
More granularity in time analysis
After several discussions at the Wikimania 2019 (Stockholm), it appeared as necessary to be able to have a finer granularity in the analysis than the one provided by the “Diversity Over Time” dashboard. This dashboard was useful to evaluate the impact of the previous Wikimania 2018. However, beyond seeing the distribution of new articles created in the last month, as allowed by the tool, Wikipedians wanted to monitor the edits in the last 24–48 h. This would encourage the creation of the “Recent Changes Diversity” dashboard.
Another requested improvement was to have the possibility to configure one language-based dashboard for their language, including all the analyses of interest. This appeared as a valuable add-on to the website, after the Diversity Observatory was presented at the regional conference WikiArabia (Marrakech, Morocco).Footnote 61 This language-based configurable dashboard would be a valuable step forward in terms of usability, that has not been implemented yet but should be considered.
LGBT+ and ethnic groups
Some underrepresented groups like LGBT+ and ethnic groups were recurrently in the focus of debates at the Wikimedia Movement Strategy 2030 conversations dedicated to uncovering possible actions and plans to increase diversityFootnote 62 in the movement. The degree of organization around these groups is lower than for gender or geography. The requests for including them in the Observatory were aimed at increasing the visibility of these topics and raising awareness on the barriers that prevent their coverage.
Third iteration: LGBT+, ethnic groups and time (2020 - current)
In the third iteration, the main goals were to expand the framework and complete the main topics including LGBT+, Ethnic groups and time, as well as providing new dashboards and features and other improvements to address the needs detected in the previous iteration. The analysis of data on the additional topics would be more experimental, as the research on it is scarcer and the level of engagement of the community lower.
Ethnic groups data is incomplete
We found that data on ethnic groups is incomplete for an extensive and accurate selection of all the content related to them. However, we could still achieve a collection of articles that may serve as a reference point for comparison. Building on this data, we created dashboards like “Ethnic Groups Topic Articles,” where biographies or more general topics around each group are listed as a starting point for bridging content gaps across languages. This would be especially useful for events like the “International Roma Day edit-a-thon”.Footnote 63
LGBT+ content and categorization
For the LGBT+ content, we collected all the biographies with a non-heterosexual orientation. Also, we collected all the articles that were categorized as LGBT+ in at least one language edition. For the Wikimedia LGBT+ editors, not only it matters that LGBT+ content exists, but also that it is categorized as such.Footnote 64 In fact, only 93 Wikipedia language editions have the “LGBT” related category.Footnote 65 We created the dashboard “LGBT+ Articles” which provides a way to retrieve articles related to LGBT+ and sort them according to the number of languages in which they are categorized as such. The feedback received in online events was positive.
During 2020, community engagement was reduced to online edit-a-thons, and the conversations were mostly focused on different aspects of the use of the tools. We collected some specific requests around their integration in Wikipedia language editions and within other existing tools. For example, enabling third-party applications via data API was requested to update specific metrics in Wikipedia pages using bots. Tools like Fountain toolFootnote 66 used in the contest Asian MonthFootnote 67 to count the number of articles or Bytes added during a contest would also benefit from querying the Diversity Observatory API to be able to see the diversity of articles created. Generally, addressing editors’ needs does not only imply making the dashboards more usable, but also providing ways to incorporate metrics and knowledge into other tools and spaces.
Diversity in Wikimedia education
Identifying content gaps is relevant to those organizations of the movement that are aimed at fostering partnerships or introducing Wikipedia as part of the education system. For example, in the last quarter of 2020 we were contacted by the Wikimedia Foundation education department to create three modules that would be used to explain how to read and edit Wikipedia in the classrooms. We created in the teaching materials a specific section called “Diversity Observatory”,Footnote 68 encouraging the creation of local content, and translated it to English, Spanish, Arabic, and Tagalog. Along with other texts, the module “allowed teachers to reflect on the importance of cultural representation online, the challenges in accessing sources of information, and building community knowledge responsibly.” As of April 2nd, 2021, more than 7000 teachers engaged with the content of the program by accessing the resources and joining live training sessions.
Prior to every new iteration, we acknowledged the different development trade-offs. Most typically, the question is between addressing the new opportunities (generative research) and addressing the improvement of the tooling (evaluative research). In the first iterations, we prioritized fulfilling the requests for expanding the analysis on more topics (e.g., gender, geography, LGBT+, ethnic groups, time, etc.) rather than aiming at giving a polished end-product.
We took note of every usability issue and potential new functionalities. Still, we focused on uncovering the advantages of having one framework to measure the different content gaps, considering that there are some specialized tools for gender. Even though there are always potential new topics, the Diversity Observatory addresses all the topics of interest of the non-geographical Wikimedia user groupsFootnote 69 (e.g., Wikimedia LGBT+, Gender-related, etc.)
Focusing on developing the framework has been a conscious choice on this particular trade-off between exploring new data and polishing the product. Similarly, when analysing the data in the search for valuable insights, we also preferred exposing many of the visualizations on the website as a matter of openness. The feedback we received allowed us to discard some of them. We realized that the website not only plays a role in providing solutions or insights, but is also an experimental space to invite other community members to reflect on content diversity.
We have noticed that this approach may overwhelm some users, who expect a finished product instead of a research prototype. However, this is a cost for staying open to everyone’s feedback, which is important when we are all still learning from the data and what is valuable to Wikipedians. In the future, given that the main topics are already covered and analysed, we expect that it will be possible to reduce the complexity of analyses and metrics to those few that users find particularly valuable.