Keywords

1 Introduction

Innovation in large companies is essential but is often harder to implement than in startups, where it is a natural and necessary part of the regular work. Large companies have many employees, many of them creative, but teams and team members are often bound to work only on the company strategy [1]. Creating an environment that fosters innovation and creativity requires employees to be motivated, and management needs to provide time and space for innovation to occur [1]. Therefore, to address the challenge of bringing new products to the market, many large software companies tend to approach innovation in a systematic way [1]. One example of such an approach is implementing an innovation program for internal startups [2]. For example, Google and Atlassian had their 20% time program [3], where developers are given 20% of their time to work on a project or initiative of their choosing. Another approach is to allocate days when all software developers in a company work on delivering a software product improvement of their choice. Innovation can be facilitated in a hackathon, a short and time-bound event, where participants work together to develop solutions, software or explore ideas. Large-Scale Agile organizations have increasingly utilized hackathons as a mechanism for innovation in the last 20 years [3,4,5].

Due to Covid19, company activities such as hackathons had been first postponed and eventually organized through virtual platforms such as MS Teams, Zoom, and Slack. Further, as many companies (Twitter, Spotify, Facebook, Salesforce) have announced their Work from anywhere (WFA) strategy, it is likely that structured innovative processes like hackathons will continue to be organized virtually in the future.

This unique situation has provided an excellent opportunity to understand how the shift from physical to a virtual hackathon affects the benefits and challenges of hackathons, but a andrge-scale agile organizations can embrace a WFA future. Even though there has been an increasing amount of literature on hackathons [7] and some literature on virtual hackathons [6], the literature regarding virtual hackathons for software companies, and especially the shift from physical to virtual, is scarce.

Motivated by the importance of innovation in large-scale agile and how large-scale agile organizations can embrace the WFA future, our research questions are:

  • RQ1: What are the benefits and challenges of hackathons?

  • RQ2: How are benefits and challenges affected by moving to a virtual hackathon?

Our chosen way to investigate this question was to perform a case study in a multinational software company utilizing large-scale agile methodologies. We observed the company employees during their virtual hackathon demo and inquired about the recognized benefits and challenges of hackathons for the company and their employees and how the virtual hackathon compared to the previous physical hackathons.

2 Background

Hackathons have been around for over 20 years, first appearing with OpenBSD and SUN microsystems in 1999 [8]. The word originates from the combination of the words “hack” and “marathon” [8]. There are multiple alternative names for hackathons, such as hackfest, jam, codefest, bug bash [5], as well as other more obscure names such as Delivery Day [4] and FedEx Day [3]. These events all contain similar elements, such as a limited and defined amount of time and the goal to create a minimum viable product (or at least something to show at a demo). Hackathons can be open events organized by universities, cities, municipalities, or internal corporate events.

Hackathons vary in how they are organized and executed, how ideas are structured, where it takes place (physical or virtual), and whether or not it is a competition [6]. These characteristics of hackathons provide different benefits and challenges. Falk Olesen and Halskov [7] emphasize that hackathon organizers need to tune its characteristics in order to achieve the wanted benefit for the target group.

Hackathons are not only a benefit for the organization, but also for the individual participants, providing an opportunity for individual development, goals and learning skills. There are also different categories of benefits identified in hackathons. Falk Olesen and Halskov [7] provide three categories, structuring learning, structuring processes, and enabling participation.

While hackathons provide benefits such as improving and fostering internal innovation, expanding competencies, and networking [3, 4, 6, 9,10,11,12], several challenges have been reported. Examples of challenges are prototypes that do not get sufficient follow-up work [7, 9, 11], stress by putting regular work aside [4] and issues associated with hacker culture and low self-esteem reported by Paganini and Gama [12].

Contradictions and tradeoffs are something apparent in the hackathon phenomenon, regardless of its formats. For example, diversification is both a benefit for learning and networking and a potential hinder for effective work during the hackathon [9]. Another contradiction is that the individual participants want to learn one specific technology or skill, disregarding whether or not that is useful for the organization [13].

3 Research Methodology and Approach

Our exploratory case study was conducted in a multinational agile software company identified here as “Ares” due to confidentiality. The company was founded in the early 2000s and develops content distribution software for mobile devices. Ares’s developers are located in two locations and are utilizing agile methodologies. The number of company employees at the time of conducting the study was 39 people. Ares organizes a voluntary hackathon every year and has done so for many years. Ideas are submitted and put up for voting based on managerial input. The employees then vote for three ideas they would like to work on. The hackathon teams are constructed based on the votes and managerial input to ensure suitable team sizes and diversity. The hackathon lasts two days, during which the teams self-organize, and at the end, all teams present a demo of their work. There are no awards or jury involved.

In this case study, we first analyzed a demo from a recent hackathon in the company. Based on the demo and previous research [3, 4], we developed an interview guide containing different areas for questions: Background, Motivation, Idea generation, Cooperation, Hackathon organization, Expectations, Benefits, Challenges, and Virtual.

Seven semi-structured interviews with people from four different development teams were conducted. They represented different skills and roles within the company, including developers, lead architects, senior engineers, and an advertisements operations manager, an overview of the informants can be seen in Table 1. The interviews were conducted in Norwegian or English, recorded, transcribed and qualitatively coded using descriptive coding, followed by a holistic overview and a thematic analysis [13].

The results were presented back to the company during an all-hands meeting to validate the qualitative findings and potentially elicit more information. In addition, during the meeting a survey was administered to the attendees asking them to rank the findings and add comments. Twenty-three persons responded to the survey.

Table 1. Informant overview
Full size table

4 Results

4.1 Hackathon Characteristics

In this chapter, we present our results by starting with a general description of hackathons, physical and virtual, which provide a context for understanding our findings.

How Often? – Once a Year.

To our surprise, even though the participants were very positive towards hackathons, they did not want to conduct them too often. One participant explained, “It’s such a nice activity. It’s something like your birthday. If it would be every month, it would become a bit more boring. It would be just another Hackathon.”

How Long? – Time Boxed.

This is an inherent part of the hackathon, which often means that participants do not have time to finish the project. Opinions about time boxing differ. As one states, “You should stop working when you are finished with the hackathon. Either you prove that something works, or you hopefully had fun.” This contrasts with those who said that the continuation of the hackathon idea was essential to them.

Which Ideas? – Whichever.

The freedom to pitch and develop any idea gives a motivational boost and incentive to participate. As one explains, “… with a more open approach, you can create ideas which might sound cool, but you can’t really integrate it into your company. However, on the other side, those ideas might be something new which your company could work on.” Also, the opportunity to explore personal interests was important, one example being the idea of developing a guitar tuner popular among guitar players.

With Whom? – Whomever.

The informants noted that the learning and teambuilding across disciplines and skills was a vital aspect of participating in the hackathon, indicating that a diverse team with different skill levels was good. However, this was contrasted by the importance for the participants to succeed. For example, one of the participants expressed, “If I constructed the teams, I probably would have chosen different people.”

What’s at the End? – Demo, no Competition.

One informant explained, “The demo is everything about the hackathon. The hackathon is not successful if you do not have something to show at the demo.” That being said, the focus on the demo can also make some people reluctant to join, as one stated: “I haven’t participated in hackathons earlier. I worried that I would not contribute enough because you need to deliver a product and present it at the end.”

4.2 Benefits of Hackathons in Large-Scale Agile

Based on the interviews, we will now present the general benefits of hackathons categorized into individual and organizational. Finally, we report the benefits of most importance, based on the rating provided by the survey respondents (see Table 2).

Table 2. Rated benefits of hackathons
Full size table

Our findings show that innovation is a key benefit both for the individuals and the organization, recognized as the ability to test new ideas fast and take advantage of emerging market opportunities. One informant said, “It was a product idea I had wanted to develop for a while, and finally there was a hackathon so I could check if it worked.” Other interviewees also mentioned that given the proper business case, hackathon products could be launched. One explained: “Two years ago, we made a feature that was put directly into production, actually four weeks later, it was done properly. Hackathon proved it was possible, and it fits well into the existing product line.”

Having a break from everyday work was highly valued by individuals. One interviewee explained: “It’s kind of relaxing for the people, for the company. Well, for the team itself because you are not doing exactly what you are paid for.”

Teambuilding was rated as the top organizational benefit. Having fun from being with colleagues was rated third as the individual benefit. One informant explained, “In hackathons, what normally happens is that the main outcome, at least in our experience, tends to be the social aspect of it, the teambuilding.” Another stated, “you will remember other important parts of the hackathon than just the work or who was the winner, for example, the fun you had with your colleagues and the knowledge you gained.”

Increased marketability and talent attractiveness are achieved due to the individual benefits and because the hackathon in itself is a company practice that gives an edge during the recruitment process. The individual expands their skills and expertise by working on topics and technology where the participant did not have much prior knowledge and thus utilizing the hackathon for learning, providing the organization with higher and broader skilled employees. Building network among colleagues and having employees that know who knows what, increasing the inter-team communication, and give employees new insights into and respect for one another’s role and work.

4.3 Challenges of Hackathons in Large-Scale Agile

Even though there are clear benefits of hackathons, employees also mentioned some challenges. Therefore, we mention the challenges corroborated by the survey. The complete list of challenges is shown in Table 3.

The top individual challenge was related to hackathons adds stress. Even though hackathons are mostly a welcomed break, the timing for running a hackathon is not always good. As someone explained, “you have pressure during your workday, and you need to get something finished and you look at the hackathon as something in the way that is just disturbing you from important work.” The two organizational challenges echo this challenge – stopped production for a time and increased employee stress.

Mismatch in skills and desires due to the team diversification was another top challenge. The optimal team structure was not always obvious. One participant described that the team members did not always have the same desires and motivation for participating in the hackathon. One interviewee explained, “It was challenging that we had different goals we wanted to achieve. I wanted my product to be launched while the others wanted to expand their skills on machine learning.”

Being unable to complete the hackathon projects is somewhat frustrating. Even though the employees are well aware that the hackathon is timeboxed, they still desire to finish their products. A participant explained, “I am not allowed to work on it, I have other work tasks. That’s a downside with hackathons.” This results in developers feeling dissatisfied as their goals remain unfulfilled. As someone explained, “You know, the worst feeling you can have is when you write code and after some time it gets forgotten and no one ever uses it.”

4.4 Virtual Hackathons – What Are the Changes?

A virtual hackathon does not impact the ability to take advantage of emerging market opportunities achieved through hackathons. The idea generation and idea voting are similar to the previous hackathons, the difference is that the teams work and collaborate virtually. As one says, “But from a technical point of view, to just develop something, you can always use Slack or whatever, or just Google meet and call.” In addition to this, our findings show that the number of hours spent on virtual hackathon is less than when it is held physically. “During the first hackathon we stayed in the office up until midnight.” In contrast, in the virtual hackathon, the employees started by dividing the work, then worked focused individually, continuously coordinating through Slack and using virtual meeting rooms for discussions and planning: “We did the first meeting when we split the work. So, after a few hours we met again. We have this approach, let’s just give it a try. Then a few hours later, we had another meeting, and we tracked our progress quite often.”

A break from everyday work through hackathons is especially welcome after stressful periods such as migrations, or as one says in a pandemic,” Especially in these pandemic times. You know we are working from home. So, we want to meet people in real life and hackathons are one way to just, you know, meeting and collaborating”.

We see a reduction in the benefit of having fun with colleagues: “It was not as fun as it used to, hackathons are usually more fun than over video during Covid, in my own home.” In addition, most employees mentioned that they would prefer to have the hackathon co-located in the same physical location in order to socialize and have a good time. Lastly, disagreements due to mismatches in individual goals seem to be harder to solve through virtual meetings.

Informants note that there is less tendency to work after hours during a virtual hackathon. This was explained due to people logging off after their part of the project was done, “Some years, when we had pizza and social interactions, I worked to midnight. This year, I worked to 9 pm, other team members (designers) only worked to 4 pm, and it did not make sense for them to hang around on Slack after they were finished. In the office it’s different because you can be social”.

5 Discussion

Noting the lacking research on internal corporate hackathons [9], we will now discuss our research questions. Our first research question was, “What are the benefits and challenges of hackathons?” Consonant with [3] and [4], we found that providing dedicated days for developers may lead to innovation and new product development and that there are benefits both on the individual and company levels, as seen in Table 1. Developer involvement, productivity, competence, and wellbeing are thus important factors for large-scale agile companies that foster innovation. We also found that learning and networking are important benefits, as corroborated by [7, 12]. In addition, just providing time to take a break from regular work and having fun is important for the wellbeing of the employees. In addition to the different levels of benefits, we also see a clear link between the characteristics of hackathons and the benefits that emerge from them. We also find tradeoffs between some of the benefits. For example, building new stuff vs. teambuilding as our findings show that tailoring the teams for diversification might take away the team’s ability to perform during the hackathon, corroborating Falk Olesen and Halskov [7] where the skills wanted by the individual did not match that of the organization. Our findings showed that some employees were frustrated when their product was not continued. This confirms a challenge with hackathon continuation, as reported in previous research [5, 11]. We also found additional challenges such as taking time away from everyday work, whether it was good timing or not, corroborated by [4]. Personal desires and goals could also pose a challenge where teams could not agree on common goals. Fear of not contributing to the team during the hackathon was also found due to low self-esteem, this corroborates Paganini and Gamas [12] findings, albeit our findings are provided by male developers with limited experience, showing that low self-esteem is not only found with female participants but also in-experienced participants.

Our second research question was, “How are benefits and challenges affected by moving to a virtual hackathon?” Our results suggest that the shift from physical to virtual hackathons in large-scale agile does not seem to change the innovation benefits or worsen the challenges that much. We found that the number of hours spent on the hackathon decreased compared to physical hackathons, and the work was more divided and focused between the participants. The employees were producing good quality demo prototypes and were happy with the outcome. The findings suggest that having fun with colleagues has decreased in virtual hackathons since physical hackathons provide more encouragement and room for socializing, which was not provided in the virtual hackathon.

5.1 Limitations

This study only has one sample for physical to virtual hackathons. In addition, all the interviewees were male, and we did not elicit information about female participants. Qualitative studies are also subjective by nature, however, the survey we administered provided some quality assurance of the validity of the findings.

6 Conclusion and Future Work

Our results suggest that virtual hackathons in large-scale agile organizations benefit individuals’ opportunity to test new ideas, take a break from regular work and acquire new skills as well as broader skillset. In addition, hackathons provide the organization with new products, refreshed employees, and a more competent workforce. Virtual hackathon teams develop good technical solutions; however, the social and fun aspects are diminished compared to physical hackathons.

Practitioners should also be aware that challenges can include additional stress and dissonance in hackathon teams due to differences in experience and personal goals.

Future research should focus on collecting more data about virtual hackathons in other large-scale agile organizations to develop further insights into how virtual hackathons can be organized and improved. When doing this, one could consider using the SPACE framework for developer productivity suggested by Forsgren et al. [14]. The framework recognizes multiple productivity dimensions for large-scale agile organizations, both on an individual, team, and organizational level, that are similar to our findings on hackathons.