Gender differences in participation and reward on Stack Overflow

As coding skills find their way into the basic requirements of many well paying jobs (Glass 2016), the underrepresentation of women in technical fields is becoming an increasingly salient issue (Republic 2014). Recent efforts to reduce this discrepancy are multi-faceted: while communities aimed at teaching girls or women to code focus on issues related to self-confidence and gender stereotypes (Cohoon and Aspray 2006; Ahuja 2002), more IT companies and schools are promoting diversity and fighting discrimination (Clayton and Lynch 2002). Online resources also provide significant and informal opportunities for people who want to learn how to code, from free courses to entire communities for learning, discussing, and collaborating (Glass 2016; Wired 2014; Lerner and Tirole 2002). Two prime examples of the latter are Stack Overflow and GitHub. One would hope that the digital nature of these new “knowledge marketplaces” could democratize knowledge and help to mitigate existing inequalities. Yet, research finds that the opposite has happened. Contribution rates for women in open-source programming communities such as GitHub or Stack Overflow are even lower than their overall presence in the IT labor market.¹ These trends align with findings from studies on other open-source communities and knowledge creation platforms, including Wikipedia and OpenStreetMap (Stephens 2013; Ford 2016; Horvath 2014). These studies formulate a variety of hypotheses to explain this effect, including the impact of gender roles and stereotypes, lower confidence and risk aversion among women, and the asymmetric threat of harrassment.

In this study we explore reasons behind low participation and success rates of women on Stack Overflow, the largest Q&A platform for programming and an important resource in the open source IT world. Over time, Stack Overflow has grown into a large database of knowledge which people use at all stages of learning how to program. Questions vary in difficulty and specificity, and the coverage of topics evolves essentially in sync with coding itself. Beyond being a knowledge base, the site also serves as a social platform, job search site, and recruiting tool - it is an important hub in the IT ecosystem.

We collect a gender-balanced sample of over 20,000 user profiles, and use them to investigate the differences in the participation and success of men and women on the site. We frame our analysis in terms of the following questions. Do men and women have different levels of success on Stack Overflow? If so, is it because of differences in how they participate on the platform? We find that the answer to both questions is yes, and follow up by probing the differences in rewards for different kinds of participation on the platform.

More specifically, we find significant gender gaps in activity: women are more likely to ask questions, while men provide more answers and cast more votes. Votes are positive or negative evaluations of other user’s questions and answers. Users gain and lose reputation points, the primary measure of success on the platform, for receiving up and down votes. We also observe that men are significantly more successful on the site, measured by their collection of reputation points. Using the Oaxaca-Blinder decomposition (Oaxaca 1973), a method from economics that to the best of our knowledge has not yet been previously applied to measure gender disparities in online communities, we decompose the outcome differences between men and women in terms of differences in their activity. While our models show that question and answer behaviors and other user- and community-level features explain a large portion of the success gap. 11% of the reputation gap remains unexplained.

In the final part of the paper, we explore the consequences of a hypothetical redesign of the site’s reward system. The proposed alternative scoring system equalizes the rewards for well-liked questions and answers, a simple and justifiable change which does not penalize any group of users in absolute terms. We find that the median woman is marginally more successful than the median man under this revised success measure, reversing the situation under the current system. However even with our recommendation men are still significantly more successful on average due to their overrepresentation among the top users. The recommendation may alter site dynamics as users will be incentivized to ask more and better questions. Given Stack Overflow’s stated aim to build a universal knowledge base, we believe that such a shift in the dynamics is in line with the spirit and goals of the platform.

In general however, our findings suggest that fundamental remedies may be needed in order to encourage women to participate more and in different ways. Given the increasing importance of Stack Overflow and similar sites in both the labor market and knowledge creation, our findings underscore the importance of design decisions and interventions even in well-intentioned and organically grown online communities.

In this section we outline recent work on gender gaps in IT and on the web. We also survey studies which investigate online platforms to detect and measure inequalities.

In this section we describe Stack Overflow as a website and a community. Stack Overflow, founded in 2008, is the largest Q&A site for computer programming. Today, the site hosts over 16 million questions and 24 million answers, and it has a global Alexa rank of 63 (Stackoverflow traffic statistics. https://www.alexa.com/siteinfo/stackoverflow.com). Previous work on Stack Overflow has highlighted its importance to the programming community as a hub of knowledge-sharing (Vasilescu et al. 2014). According to creators, their goal was to design a free access platform serving users with a high quality knowledge base (The stack overflow age. https://www.joelonsoftware.com/2018/04/06/the-stack-overflow-age). Indeed, today programmers of any level or type turn to Stack Overflow as part of their daily routine and the site is usually among the top results in Google searches for programming related queries. In this way, the knowledge shared on Stack Overflow is reused beyond the initial exchange between question-asker and answer-giver. Its user-base also significantly overlaps with that of popular code repositories such as Github (Vasilescu et al. 2013).

Stack Overflow also has influence on hiring/recruiting in the IT sector. Users can search for employment opportunities on the site’s job boards. Moreover, Stack Overflow provides opportunities for them to demonstrate credible, verifiable expertise. Indeed, IT companies and recruiters often look for Stack Overflow profiles when trying to fill positions (Fawcett 2012). This is facilitated by a recently developed resume service on the site: users can turn their profile and activity into a standardized, searchable resume, ready for inclusion in the site’s database of jobseekers. In this way Stack Overflow as a platform is a becoming a significant labor market matching service.

Stack Overflow’s knowledge base (namely the questions and answers that have been posted) are freely available to the public - no registration is required. In order to create content, however, users have to sign up using an email address or social media account. Every account has an associated profile page which tracks a user’s activity history and accomplishments on the site. Users can also enhance their profiles with biographical information, contact information, and an image. The large disparity between the number of unique monthly visitors (estimated by Quantcast to be 50 million in March 2018) and the number of active accounts made in the history of the site (less than 10 million as of March 2018, with far fewer active accounts), indicates that the vast majority of the site’s users are passive: using the site’s knowledge without making contributions of their own.

Registered members can post questions and answers, vote on or edit the questions and answers of other users, and interact with posts using comments. The up and down voting functionality serves as a natural content moderation, users can boost useful questions and answers and subsequent visitors have an easier time finding them. There are also elected official moderators among the community members, who can delete, modify content, and merge repeated questions into one topic.

Aside from the potential for its open and public-facing nature, Stack Overflow also has a gamification aspect (Anderson et al. 2012). Specifically, participation leads to users earning various reputation points and badges. Reputation points are primarily received for upvoted questions and answers. Users receive some moderation privileges when they accumulate enough reputation points. Bronze, silver, and gold badges can be earned through a variety of activities, for example for receiving some number of upvotes on a question or answer, editing posts for clarity, or even for visiting the site for a number of consecutive days. Gamification is a common method used both to increase user engagement and to steer users towards specific behaviors deemed to benefit the community. Indeed, previous studies show that the badge system of SO has a motivating effect on the community. However, as past research indicates that men may respond more to gamification than women, this may exacerbate gender inequalities in participation (Pedro et al. 2015; Herzig et al. 2015). We note that a user’s reputation and badge counts are immediately visible next to any question they ask or answer - offering a signal of the user’s presence and participation on the site to others.

In this section, we present a more detailed overview on how the Stack Overflow website works and the data we gathered about users. We also give a thorough outline of the features we extracted or created and that will serve the basis of the upcoming analyses. We now present our data collection and labeling methodology. Additionally, we introduce our dataset, focusing specifically on how the data breaks down along gender lines.

4.2 Feature Creation

Several of the features we use in our analysis can be extracted directly from user profiles. First, we note users’ meta data, including their biography text, sign-up date, and whether they link to a personal website or social networking accounts such as Twitter, Linkedin, or Github. We operationalize these features in our models as a self-promotion index which takes a value between 0 and 1 depending on the proportion of self-promotion fields that the user has filled out. Within the biography field we check the text for the substrings “senior”, “lead”, “head”, and “manage”, assigning a dummy to each user taking the value 1 if they list any of these leadership or senior position indicators in their bio.

Second we quantify their activity on the site by how many questions they ask and answer, how often they edit posts, how many upvotes and downvotes they cast, and how often they make posts with which tags. Finally, we have information about user’s outcomes and success on the site from their reputation scores and the number and types of badges they receive.

Gender inference

Inferring gender of individuals from their online profiles is a complex problem. We apply a two-step approach to infer user gender, first using genderComputer (Vasilescu et al. 2014), a tool specifically created to infer gender of Stack Overflow users from their given usernames and location. GenderComputer considers a variety of string manipulations (for example reversing “Nohj” to get “John”) to expand the scope of the inference. Location can provide additional accuracy by distinguishing, for example, between an Andrea from the UK (likely a woman) and one from Italy (likely a man). This method classified the users from our sample into 238,150 male, 24,717 female, and 302,304 unidentified users. In order to evaluate the quality of this classification, we manually examined 100 users classified as men and 100 classified as women. We found that while the method performed very well on men (97% agreement with our manual check), our manual check agreed only in 44 out of 100 cases of women. This replicates the recent finding by Ford et al. (2017) that genderComputer sacrifices precision for greater recall when inferring women users.

The second step of our inference seeks to correct this bias by applying a more conservative method based only on first names and location called Gender Guesser. By considering only users rated as likely male or likely female by both methods, we are left with a smaller but more accurate sample. 10,571 users are rated as highly likely women by both methods. We randomly choose 10,571 likely men (again classified as such by both methods) to obtain a balanced sample. We repeated our manual check of a random sample of accounts finding 96% agreement with our classification of men, and 84% agreement with our classification of women. This ensemble approach resembles Ford et al.’s modification of genderComputer to focus on the detection of first names within the username (Ford et al. 2017).

We acknowledge several limitations and drawbacks to our approach to inference. First, we make the simplifying assumption that gender is binary. We argue that this is a fundamental limitation of examining questions about gender differences using harvested data. Second, discarding alias usernames builds on the assumption that men and women are equally likely to adopt user names that can be mapped to their respective gender, and that this mapping does not substantially impact our hypotheses. However, previous research has shown that anonymity impacts behavior (Robertson et al. 2017), and it is possible that some users utilized an anonymous name in order to establish an independent identity. Such name selection is highlighted by the literature on gender swapping in online communities (Bruckman 1996; Szell and Thurner 2013), where, for example, women may pose as men if they feel that they will be taken more seriously or to avoid harassment. We also note the limitations of the geographic component of our inference: a minority of users include location data, and a given location may not reflect a user’s origin (for example if an Italian man named Andrea moved to the UK). Finally, name-gender databases have been shown to have significantly less accuracy when used to infer gender for non-European names (Karimi et al. 2016).

Despite these limitations, we argue that our focus on identifiable names provides the best possible data to test our hypotheses of gender behavioral and outcome differences on Stack Overflow. By limiting our dataset to users where we are highly confident about our gender inference, we gain greater confidence in the estimates of our econometric models. Moreover, our analysis includes robustness checks with 5, 10, 20, and 50 percent of our gender labels in the balanced sample randomly shuffled. These test help us better understand the effect of potential classification errors on our results. See details in Section 5.2.

Detecting user communities

Given the size of the site and the diversity of topics that its users discuss, we consider that coherent communities of users may exist with significantly different patterns of behavior, norms, and outcomes for men and women. For example, users active in a more diverse community may be less likely to leave the site (Vasilescu et al. 2015), while women encountering other women are more likely to engage in a thread (Ford et al. 2017). Using a similar approach to Bosu et al. (2013), we grouped users in communities by building a network where two users are connected if their posts often share the same tags. Specifically, we created a similarity measure between users by calculating a weighted Jaccard similarity measure, defined as

$$s(u,v)= \frac{{\sum}_{t \in T} \min (t_{u},t_{v})} {{\sum}_{t \in T} \max(t_{u},t_{v})} $$

where T is the collection of all tags used at least 200 times, and t_u denotes the number of times user u made a post with tag t. We then filtered the edges using Serrano’s disparity filter (Serrano et al. 2009), which, for each node, checks the weights on all its adjacent links against the null hypothesis that they are uniformly distributed. Each observed weight then has a p-value. We filter edges using this p-value (p<.01). The resulting network has approximately 150,000 edges connecting the roughly 21,000 users. We use the Louvain algorithm (Blondel et al. 2008) to detect communities in this network. We tune the method’s resolution parameter to find larger communities to facilitate a qualitative understanding of the communities found.⁴ We plot the network, visualized using a force-layout algorithm, in Fig. 1. The nodes are colored by community membership.

Open image in new window

We manually checked the most commonly used tags in each community and found many clearly interpretable communities. The prominent programming languages and frameworks we identify in the largest communities coincide with those found in other analyses of programming language use, for instance on GitHub (Celinska and Kopczynski 2017). We describe the 10 largest communities, accounting for 80% of our users, in Table 2. Note that we sampled the males to achieve a 50-50 male-female ratio in our dataset. We see small, occasionally statistically significant gender differences. We find that the C#/asp.net, a Microsoft-developed software framework, and Ruby/Rails, a web development framework, communities have the highest representation of men, while Android, a programming language for mobile phone applications, has more women. We find that Ruby/Rails is the community with the lowest incidence of downvoting.

Table 2

Descriptive statistics of the 10 largest user communities based on a network of tag-use similarity among our balanced sample of users

Description	# of users	% male	% downvotes	% Rep. last year
C#/asp.net	2900	54%	5.5%	6%
Java	2605	49%	5.6%	7.2%
PHP	1941	51%	7.4%	6.5%
Android	1856	45%	5.5%	8.3%
Python	1665	49%	5.1%	9.3%
iOS	1548	49%	5%	6.9%
Javascript	1526	48%	7%	8.9%
C++	1390	51%	5.8%	5.9%
Angular/Node	886	51%	5.3%	14.3%
Ruby/Rails	873	55%	4.2%	6.8%

We describe each user community by interpreting the most frequently used tags in posts by its members. The last column refers to the share of the community’s total reputation gained in the last year

As past work indicates, community structure has a significant impact on user behavior and the possibilities for gaining reputation (Bosu et al. 2013). For instance, it may be easier to ask a new question or post answers in a newer community, for example on Angular/Node related questions, than in a long established community such as on C++. Therefore subsequent models explaining gender differences (see Results section) include fixed effects for user community. We also control for the size of the community, the percentage of the community that is male, and the percent of reputation generated by users in the community in the last year as a proxy for how new the community is.

In this section we discuss one potential way to mitigate the gender differences in outcome and success on Stack Overflow: modifying the reward system. Our results suggest that differences in the rate at which men and women post answers account for the largest part of the reputation gap. On the other hand, women tend to ask more questions than men. As reputation points are almost entirely a function of the number of upvotes received on questions and answers posted, and because receiving an upvote on a question results in half of the reputation gain (+ 5) that receiving an upvote on an answer does (+ 10), we investigate what happens to the distribution of reputation scores of men and women if these rewards were equalized. In other words, we check if equalizing the rewards for good answers and good questions decreases the gender gap.

Independent of the issue of gender disparities, we argue that the proposed change has merit when considered against these points, especially when considering the site’s increased importance as a knowledge resource since 2010. We do not agree, for example, with the value judgement that answers have more intrinsic value than questions: without the question there would be no answer. Stack Overflow distinguishes itself from Wikipedia or textbooks as a knowledge resource by providing applicable answers to real-world user-generated questions. The service of asking a genuine question, the answer to which may seem simple to an expert, is part of what gives Stack Overflow its appeal above and beyond the example cookbook solutions available in many programming references or textbooks.We also note that a single question can generate multiple useful answers - suggesting that any one person answering a question can still learn from other answers. Finally, improvements in site moderation and semi-automated detection of duplicate questions (Zhang et al. 2015) likely reduces the prevalence of reputation mining by asking repetitive questions.

Increasing rewards to good questions may help to make the site more inclusive by offering a less competitive and speed-oriented way to build one’s reputation. For example, recent work on user strategies for gaining reputation on the platform focuses entirely on answers, finding that answering questions quickly (ideally first) is one of the best ways to quickly collect reputation (Bosu et al. 2013). The authors also find that focusing on areas where there are fewer experts, or answering questions on off-peak hours are productive strategies. Such time pressures do not play the same role when one is asking a question.

To calculate the revised reputation score, we collect additional data on each user’s activity. We calculate the revised reputation scores of all users by counting question upvotes as being worth 10 points. Recall that women have on average 55% of the reputation of men, and the median woman has 73% of the reputation of the median man. Using the revised reputation, women have 71% of the reputation of men on average, and the median female has 16% more reputation than the median male. We see this shift in Fig. 6, where we plot the distribution of log reputation and log revised reputation by gender. Indeed women have much lower variance: the low and high end of the distribution are proportionally much more male.

Open image in new window

If additional reputation for asking questions even slightly increases the engagement of some women on Stack Overflow, peer effects might encourage other women to post (Ford et al. 2017). Yet we emphasize that this potential change is only one strategy to address gender disparities on Stack Overflow. Indeed, one should also be interested in why men are so much more likely to give answers, and how one might encourage women to give more answers. Furthermore, one must consider how all users would change their behavior if the scoring system stopped favoring answers. As men might respond more readily to gamification, it is possible that they would begin to ask more questions.

In order to understand why the revised reputation scores do not change trends among the most successful users, we investigated how users of different reputation levels contribute to the site. Figure 7 shows the contribution rates broken down by activity type for each reputation decile. While questions are relatively evenly distributed across the groups, most answers are given by a small number of “experts”. More precisely, 68% of answers are given by the top decile of users, while only 28% of questions are. While this result is somewhat intuitive given that the motivation for asking a question is a lack of knowledge, we think it brings attention to an important issue. Increasing rewards for questions encourages users from a broader spectrum of the population, most critically the learners who form an essential part of the site.

Open image in new window

In this paper we investigated gender differences in the activity and success of users on Stack Overflow. Women are highly underrepresented in the data we collected, not only compared to male users of the website but also compared to their presence in the IT labor market more broadly. Moreover, according to our success measures they are also less successful on the site than men. These differences can partly be explained by different participation or activity behavior. While women ask more questions than men, men are more active in all other forms of activity. Given that Stack Overflow’s current reward system favors popular answers over popular questions, these differences in activity leads to large differences in outcome. Excluding this effect and controlling for important user and community level explanatory features, only 11% of the reputation gap, which may be due to “perceptional” discrimination (among other confounders), remains unexplained. Given our finding that the current rewards system favors “male behavior”, we propose a new reputation system which equalizes the reward for upvotes on questions and answers. We find that the new scoring system reduces differences in the group means, and in fact leaves the median woman with slightly higher reputation than the median male.