Introduction

Humanity is faced with complex environmental and social problems such as climate change, loss of biodiversity, and poverty (Hertel and Rosch 2010; Hoang et al. 2023; Leal Filho et al. 2022; Letcher 2021). Solving complex problems requires the collaboration of scientists from different disciplines (Gain et al. 2020; Gardiner 2020). Sustainability science is a research field that deals with complex problems and is inherently interdisciplinary (Miller 2013; Spangenberg 2011). However, such collaborative work is not easy. Researchers may not know how to collaborate (Freeth & Caniglia 2020). Other common problems are a lack of resources (time and money), disciplinary language, jargon and perceptions (epistemologies and ontologies), spatial dispersion of researchers from different disciplines and faculties, and power imbalances (Cairns et al. 2020; Freeth & Caniglia 2020).

Literature on sustainability and sustainability transition calls for alternative ways to manage projects. Traditional linear (waterfall) project management approaches might be ill-suited to the context of complex problems. Sustainability-related challenges require more adaptability and flexibility (Ely & Marin 2021; Leach et al. 2010). Writing about sustainability science, Spangenberg (2011, p. 282) states: „The structure of any sustainability science project should be conceptualized not as static, but as a dynamic process with a clear vision but no final state defined in detail. This combination of clear objectives and flexibility in structure and content should be backed by a number of contingency rules […].” Although sustainability science is problem-oriented, projects in the field of sustainability science might not only focus on problem-solving but also on learning (Freeth & Caniglia 2020). Thus, project structures might need to create learning spaces and thus be more iterative in nature. Accordingly, in sustainability research, project management approaches need to accommodate iteration and support interdisciplinary collaboration.

Some literature describes interdisciplinary collaboration as a messy process in which problem definitions and solutions, as well as the collaborative process itself, are self-emergent. Thus, the process may involve many negotiations, which “ […] emphasize the need for reflexivity to make these processes visible and transparent “ (Cairns et al. 2020, p. 1719). Due to the character of interdisciplinary projects, an Agile approach might inherently be required (Lanier & Sukop 2016). Hidalgo (2018), builds on the challenges of collaborative research projects and illustrates how Agile Project Management could be a useful approach to address the challenges of collaborative research projects.

Agile is a project management approach that is based on a set of Agile values (Beck et al. 2001a) and principles (Beck et al. 2001b). Agile originates from the IT sector (Hidalgo 2019; Perkin 2020, p. 31) and has been developed to facilitate iterative processes that necessitate reflection, learning, adaptation, and creativity. Though, in more general terms, it responds to business environments that are increasingly influenced by digitalization, fast-changing technologies, and increasing uncertainty (Perkin 2020). The application of Agile is no longer only limited to the IT sector (Perkin 2020; Pope-Ruark 2017), and within the corporate world, it has shown to be a successful approach in delivering aspired results (Serrador & Pinto 2015). While the corporate world is not necessarily comparable to academia, one may wonder whether Agile could also be successfully applied in collaborative research projects.

The literature on applications of Agile in interdisciplinary research projects is scant.Footnote 1 Hidalgo has published some articles reporting on agile project management in academia (Hidalgo 2018, 2019; Hidalgo & Morell 2019). Ochoa et al. (2021) report on adopting Agile in a faculty through two pilot projects. The application of Agile in collaborative academic aeronautics research projects is discussed by Vecchia et al. (2018). In the book The Agile Faculty, Pope-Ruark (2017) gives valuable insights about how to apply Agile in research projects, research endeavors, or teaching. Apart from this, Agile literature covers teaching Agile (Scott & Campo 2021), using Agile for academia-industry collaborations (Guillot et al., 2017; Sandberg & Crnkovic 2017), Agile in IT (Palopak & Huang 2022; Tolfo et al. 2018), or Agile in business and business management contexts (Küffner et al. 2022; Sindhwani et al. 2019).

Although Agile could be useful in complex, collaborative research projects (Hidalgo 2018; Pope-Ruark 2017), it might not be applied, as researchers may not be aware of this approach or might not have the resources to learn more about it. While studies providing new frameworks for collaborative research are useful (König et al. 2013; Tobi & Kampen 2018), it can be asked what researchers can learn from the well-tested Agile approach (Ochoa et al. 2021). The reviewed literature indicates that information about applying Agile in academia is sparse.

To address this gap this article showcases how Agile principles can be applied in interdisciplinary research projects in academia. The Agile principles are translated to the academic context and used to illustrate how they can be applied within an interdisciplinary research program in the field of sustainability transition.

In the next section, the research program (RP) to which Agile principles have been applied is presented. Before expanding on the Agile principles, the method used to collect the needed information is outlined. Subsequently, the Agile principles are introduced. Thereafter, five main aspects of the Agile principles will be explored in more detail using the RP as an example. Before concluding, the challenges of applying Agile in the RP are summarized.

The research program

The application of Agile principles is illustrated within an RP. To explore potential future research avenues, a European university provides funding for three-year RPs that university consortia can apply for. One of the successful RP applications envisions investigating the materials transitionFootnote 2 facilitated through inter- and transdisciplinary collaboration. The RP explores the transition in technical, economic, and social terms. Additionally, it investigates how collaborations among researchers and with stakeholders can support the materials transition. Thus, the RP aims to study its own activities.

The RP’s proposal emphasizes that knowledge from different disciplines is needed to understand the materials transition. Though, it is not clear if and how interdisciplinary collaboration could support the materials transition. Accordingly, to better understand the needed parameters for successful interdisciplinary collaboration, the RP also studies the collaborative process within the RP.

To facilitate the envisioned research goals, three conditions need to be met: (1) setting up interdisciplinary teams, (2) installing processes and structures that permit interdisciplinary collaboration within and between teams, and (3) developing and implementing means to observe, analyze and evaluate the processes and structures. The author of this article is in charge of point three.

It needs to be highlighted that the RP did not consciously choose to apply Agile principles. Rather the research program managers (RPM) made intuitive choices in terms of project organization, structure, and management that would allow for adaptability and flexibility throughout the research process. These choices were made as a reaction to earlier experiences of the RPMs. In these experiences, they were part of research projects, which followed top-down management practices that did not permit sufficient freedom. This was understood as a hindrance to the creative research process, and thus, the RPMs implemented a different approach.

After onboarding and getting in charge of observing, analyzing, and evaluating the research process, I realized that the RP was applying Agile principles. The post-rationalization of processes is not an isolated phenomenon. In one of the few articles about Agile management in academia Hidalgo (2019) provides a direct quote from one of his interviewees: “We were already working with an agile approach but we had not called it ‘agile.’ Later on we started formalizing things and picked up more and more scrumFootnote 3 tools and techniques to improve the way we manage our projects.” Furthermore, as indicated in the introduction, the perception of the RPMs that research programs need to be adaptive is neither unique.

It can be assumed that other interdisciplinary projects in the field of sustainability science follow Agile principles without knowing. It might be of service to these projects if they can apply a well-established frame to their way of doing research. Therefore, this article will outline how Agile principles can align with interdisciplinary projects that embrace reflexivity, creativity, learning, and innovation.

Method

As indicated above I am in charge of observing, analyzing and evaluating the processes and structures of the RP. Thus, based on the RP design I was assigned with an observant role. Though I am not just an external observer, but I am part of the RP. That means that I actively participate in meetings. To collect information about internal processes, I am following the ethnographic method of participant observation (Guest et al. 2013, chapter 3; Seim 2021). To not influence the processes too much, I abstain from influencing the working process in major ways, such as pushing methods or the selection of certain case studies. Thus, my participation is mostly limited to sharing thoughts or reflections.

Since the application of Agile principles was not a conscious act, the reporting on the application of Agile principles is based on observation. The process is described in Fig. 1. I have observed, how interactions take place, how information is shared, how feedback is organized, or how decisions are made. Within meetings, I take notes and reflect on the meetings afterwards. I do not only reflect on the content of the meetings, but also on the dynamics within the meetings. For example, I reflect on how decisions are made. Was it a democratic discussion or a top-down decision? I also reflect on more subtle aspects like attitude. For example, do researchers behave competitively or collaborative? Furthermore, I also observe and reflect on information that I do not get. Since I am part of the team, not getting certain information gives me insights about cut-off information streams. Observations and conversations with other team members can then help me to understand if, e.g., the lack of information is only my observation or a general issue.

Fig. 1
figure 1

Research process

Full size image

Observations started with my onboarding (01st of September 2022), the cut-off date for data entering the current analysis is the 1st of June 2023. The RP’s core team consisted of 14 researchers. However, more researchers were involved in the individual teams. The involvement of additional researchers fluctuated as e.g., master students were involved to perform certain tasks. All researchers that were part of the RP within the above-mentioned timeframe were part of the observations. As common in observational research, I did not steer the participation of researchers (Jorgensen 2005). Thus, I did not control the frequency or intensity of interactions with the RP’s researchers.

The collected information was matched with the Agile principles (discussed and introduced below). Figure 1 illustrates that the identification of Agile principles was conducted simultaneously with observation and processing. This is owed to the fact that Agile was applied by accident. Thus, the Agile frame was only fully explored when it appeared to be a viable frame for understanding the RP’s processes. How the RP adopted the principles is reported in “Applying Agile Principles” section. The challenges that the RP faced are summarized in the “Discussion” section. This open-ended approach suits the research method (participant observation), which is frequently used “[…] to explore and examine research questions that emerge with investigation rather than preconceived hypotheses; […]. The definition of study questions, issues, and problems, as well as the formulation of concepts and development of appropriate indicators often must be refined during the course of the investigation while participating and observing in the field” (Jorgensen 2005, p. 8).

My role within the RP was not clear from the start. Though, that I would take the role of a participant observer was first discussed and agreed upon with the RPM. In these discussions, it was decided that my role would not be shared immediately. This decision was based on the awareness that an observer might disturb the group dynamic. It can be assumed that an external observer might have changed researchers’ behavior, and thus, I might not have had the possibility to make the observations that I made due to my disguised role (Seim 2021). Ten months after my onboarding, the information about my role was shared in a meeting in which I also presented more details about Agile. While the presentation was received with great interest, the disclosure of my role was received with reservation. One team member stated that they now have to be careful how they act since I would observe them. When I revealed my role, I could show that my observations are not to the team’s detriment and that my role is ultimately to facilitate a learning process, by analyzing and reflecting on the overall processes and structures.

What is Agile?

Hidalgo (2019, p. 3) defines Agile as: “Agile project management (APM) or “agile methods” [which represent] a team management approach and a productivity framework that supports continuous and incremental progress on work priorities, even in the face of changes.” Agile is more than just a project management approach or a set of methods. Agile literature talks about not only doing but also being Agile (Perkin 2020). As stated, Agile stems from IT and was formulated as a Manifesto (Beck et al. 2001a). The Agile Manifesto consists of four values, which are further split up into eleven principles (Beck et al. 2001b). Hence, the Agile Manifesto does not simply describe a project management approach but values and principles that should be applied when working together.

Arguably, it is necessary to understand the Agile process, which has a specific flow. However, literature indicates, that the Agile process is not set in stone and should be adapted to the respective situation (Hidalgo 2019; Ochoa et al. 2021). Business literature discussing the transformation to Agile highlights that the way one company may apply Agile successfully may not lead to success in other businesses (Perkin 2020; Rigby et al. 2020). There is also a variety of tools and methods (e.g., Scrum, Kanban, XP, DSDM, Cristal, etc.) that help facilitate the Agile process. However, these tools neither have to be applied one-to-one and should be adapted as needed. One can even develop new tools if they support the Agile process in a specific situation.

Thus, the Agile process setup within one context might not fit in another one. Accordingly, one cannot provide a perfect blueprint of how to do (and be) Agile (Perkin 2020). Nevertheless, there is guidance in the form of the Agile principles (Beck et al. 2001b).

Agile principles

The Agile Manifesto has been written for the IT context. Thus, it needs to be translated for the research context. Tables 1 and 2 provide the Agile values and principles translated to the research context. Perkin (2020) translated the Agile principles to the general business context, and Pope-Ruark (2017) translated the Agile values to the academic context. Following these examples, the Agile values and principles have been translated for the research context. Since this translation is interpretative, other translations might be possible. For example, Pope-Ruark (2017) translated the Agile values differently.

Table 1 Right column: Agile manifesto (Beck et al. 2001a), middle and right column: translation of Agile values for the research context
Full size table
Table 2 Left column: Agile Principles from Beck et al. (2001b), right column: translation of Agile Principle for the research context
Full size table

Summarizing Agile values (Table 1) and principles (Table 2), Agile focuses on:

  1. 1.

    Providing research output that is valuable for society (value creation)

  2. 2.

    Involving stakeholders throughout the process (transdisciplinary)

  3. 3.

    Collaboration among researchers with different profiles (inter- and multidisciplinary)

  4. 4.

    Learning, reflection, adaptability (learning-centered, adaptive management)

  5. 5.

    An organizational structure that supports point 4 (autonomy and self-organization)

  6. 6.

    An organizational culture that supports point 4 (supportive culture, in-person collaboration, people-centric, close collaboration)

This article focuses on how Agile principles can be applied in multi- or interdisciplinary research projects. However, Agile principles are also suitable for transdisciplinary research projects, since Agile is customer-centered. It focuses on investigating and satisfying customers’ needs and integrating customers throughout Agile processes. Obviously, in the context of transdisciplinary research, the customer needs to be reframed as a stakeholder. However, expanding on Agile within transdisciplinary settings requires a separate discussion that, hopefully, future articles will provide.

The Agile process is rudimentarily captured in point 4. Focusing on the process only would fall short of applying Agile. Additionally, one needs to address structures (point 5), culture (point 6), and people (point 3). Furthermore, leadership needs to be considered. Leadership is not an aspect that is deducted from the Agile Manifesto. However, Agile literature illustrates that despite flat hierarchies in Agile settings, leadership is key (Perkin 2020). Therefore, leadership has been added. Hence, the Agile process is supported by structures, culture, people, and leadership (see Fig. 2). These five aspects summarize the Agile principles and are used for the subsequent analysis.

Fig. 2
figure 2

Agile comprises of processes, structures, culture, people, and leadership

Full size image

Applying Agile principles

In this section, I delve deeper into the five aspects (process, structures, culture, people, and leadership) that summarize the Agile principles and outline how they have been implemented in the RP.

Process

The Agile process revolves around reflexivity and learning. Instead of having a project laid out in detail in work packages, milestones, and deliverables, and illustrated in a Gantt chart, Agile processes work with bigger goals that are split up into small tasks and worked on in a circular fashion.

Goals are expressed in full sentences of a specific structure that convey a specific narrative. Accordingly, big goals are called, epics. These epics are then broken down into smaller stories and finally into tasks. “Epics can be written to articulate the purpose, object and goals of a project […]” (Pope-Ruark 2017, p. 42). Stories do the same but on smaller aspects of that project. Table 3 provides examples of how epics and stories can be formulated. Note that the epics and stories are formulated in a specific manner. The reader gets information about the identity (as a team exploring…), the task (we want to…), and the purpose (sot that ….). Such formulation provides more information than a simple bullet-point list, which only reveals the task but does not remind of why one wants to do something (Pope-Ruark 2017).

Table 3 Example of a vision statement, Epics, and Stories
Full size table

It might be counterintuitive to formulate epics and stories and it might not be necessary to stick to the exact format of these epics and stories. However, it is a good exercise to connect the upcoming task to a specific purpose. That helps individual teams not to get lost in the task. For example, consider Stories 2.1. and 2.2 in Table 3. A baseline report could have many different purposes, and thus, the content of the report will differ. It might seem trivial, though, especially, when working in self-organizing, autonomous, interdisciplinary teams, it is vital to remind everyone involved of the purpose of a specific task. Since the RP is Agile by accident, we did not strictly formulate epics and stories. Nevertheless, the notion of epics and stories was present. When the baseline was discussed, the team regularly returned to the purpose (selecting cases and having a good knowledge foundation).

The vision is a key aspect of Agile projects, as it allows teams to work in an autonomous fashion. A clear vision builds the foundation for “aligned autonomy” (Perkin 2020, p. 217). The RP’s vision (see Table 3) provides the overall incentive, the guiding light through the whole process. As already noted, Agile builds around reflection, learning, adaptation, and creativity. Without keeping an eye on the vision, one could get lost in the smaller tasks. Thus, with every revision round, the team should reflect on whether the current action(s) contributes towards reaching the greater vision.

Figure 3 illustrates the Agile process, which has been broken down into six steps. First, the stories and tasks are defined and collected in the backlog. Within this backlog, the stories and tasks are ordered based on their priority. The involved teams discuss the priorities and tasks, and each team commits to a specific story or task. These tasks or stories are then worked on in a sprint. The sprint ends with a review of the activity and a discussion about the next sprint. The results from the sprint inform future work and thus create a feedback loop (step 6 feeds back into step 2 in Fig. 3). Each sprint takes about one to four weeks (Perkin 2020; Pope-Ruark 2017). Thus, representatives of each team meet every four weeks to weekly to discuss what they have been working on (Pope-Ruark 2017). Of course, within the individual teams, work might need to be further discussed. To this aim, the Agile process often includes daily sprint check-ins. Following this structure, the Agile process can be described as a succession of feedback loops.

Fig. 3
figure 3

Agile process

Full size image

To facilitate the revision, the RP has installed monthly core team meetings. Additionally, progress meetings were organized every two to four months. The core team meetings served to give updates about ongoing tasks and to discuss issues ahead. Since these meetings were, at times, not sufficient to discuss all issues, additional progress meetings were held when needed. More about how the RP is organized can be found in the “Structure” section. To illustrate the Agile process within the RP, two examples are provided below. These two examples link to the epics and stories provided in Table 3.

Example pathways

Story 1.2: As a team exploring the synergies between recycling, biobased production, carbon capture and utilization, and dematerialization, we want to define each of these fields with respect to the materials transition, so that we have a good working basis.

This story needed to be split up into several tasks. The materials transition has been framed by the RP as a combination of (1) circular, (2) bio-, (3) carbon economy, and (4) dematerialization. These four aspects are called pathways and needed to be defined individually within the context of the materials transition. Additionally, the relation among the pathways needs to be carved out. The relevance of this task became obvious while other teams worked on another story, which required at least a working definition of the pathways. Providing a detailed description of the pathways and their interrelation was a time-consuming task. To not block the work of the other teams (working on Story 2.1), it was decided to provide the information in an iterative manner.

While the industry teams worked on their task, questions about the pathways emerged. Instead of postponing the work on Story 2.1., the RPM committed to providing responses to the emerging questions. Importantly, due to the interdisciplinary character, the RPM could not have anticipated the industry teams’ questions. Thus, the emerging questions helped the RPM to refine their work on Story 1.2. For example, the relationship between biobased production and carbon capture was questioned. Some industry team members argued that using biobased resources such as wood can be categorized as carbon capture. This discussion led to a clearer definition of the biobased and the carbon capture and utilization pathway. In contrast, if a waterfall approach would have been used, the work of the industry teams would have been delayed, until the RPM provided a description of the pathways. Thus, while the process involved much feedback and discussions, it allowed to provide pathway descriptions that the industry teams could work with, without delaying the work of these teams.

Example communication strategy

Story 3.1: We, as a research program, want to develop a communication strategy, to identify our clear communication goals, our target group(s), and the content we want to communicate.

An obvious question the RP deals with is its legacy. That question is closely linked to how we want to communicate what we are doing. In our monthly core team meetings, we first brainstormed about what end results should look like and who our potential target group should be. Through these discussions, it crystalized that we needed a communication strategy. Hence the formulation of Story 3.1. was the result of regular check-in meetings. In the following, the communication strategy was not simply developed but built through sprints and revisions. The result is a communication strategy that responds to everyone’s goals and not only to the goals of, for example, a project manager. Further communication streams (e.g., social media, policy briefs, scientific articles) could be selected that the researchers felt comfortable with.

The communication plan distinguishes between overarching goals, which are related to the goals of the RP, which themselves connect to certain target audiences (scientific community, students, stakeholders). These goals are then broken down into smaller tasks, respective target groups, purposes, outlets (scientific journal, policy brief, webpage, course), etc. After agreeing on the rough outline of the communication strategy, the tasks were ranked following their priority, and team members had the chance to take the lead and form smaller teams around specific communication plan tasks.

In contrast to waterfall process management, the communication strategy was not developed upfront, but its development was made an organic part of the RP itself, permitting the strategy to accommodate the needs of the RP. Allowing researchers to sign up for specific tasks promotes ownership and permits them to take over tasks that support their strengths or allow them to develop skills.

Structure

Not only projects but whole companies (Perkin 2020), faculties, or universities (Pope-Ruark 2017) can follow Agile principles and thus should apply suitable organizational structures. However, this article focuses on Agile organizational structures limited to research project settings.

To facilitate Agile processes, specific organizational structures should be implemented. Teams should be small (6–8 peopleFootnote 4) and comprise of diverse team members (Perkin 2020, p. 68f.). The teams self-organize and work autonomously. It is understood that a diversity of people and the collaboration between them is the key to solving problems. Further, the literature indicates that all skills necessary to solve a respective problem should be found within the team (Perkin 2020, pp. 69, 98). Accordingly, Agile teams are multi- and interdisciplinary in nature. This also alludes to the people-centeredness of Agile settings, which underscores the creative potential of people (Perkin 2020, p. 65).

As visible in Table 3, a project might have several epics and stories that run in parallel (e.g., Stories 1.2. and 2.1). Not all teams are working on the individual stories at the same time, but individual teams commit to specific stories and tasks. For example, the industry teams worked on Story 2.1, while another team worked on Story 1.2.

Agile projects have a flat organizational structure. Nevertheless, leads can be assigned within each team, who keep track of their backlog and the alignment with the overall vision.Footnote 5 Since several epics might be worked on at the same time, it makes sense to have a backlog for each epic. To not lose the overview of all epics and their backlogs, it is advisable to have an overall leader in place.Footnote 6 However, this person is not a project manager in a classic sense (more on leadership in the “Leadership” section). Neither the team leads nor the overall leads “[…] directly tell the team what to do – the team decides the best way to solve the challenge” (Perkin 2020, p. 33). Thus the Agile organization is a network see (Fig. 4), not a hierarchically organized structure (Perkin 2020, p. 98).

Fig. 4
figure 4

Agile organizational structure: small teams commit to working on tasks during sprints. Representatives of each team regularly meet to reflect on the activities, discuss priorities and commit to new activities

Full size image

The organigram (see Fig. 5) of the RP was developed to support specific research goals (see Table 3). While separate teams were created, these teams should remain in close contact throughout the process to facilitate interdisciplinary collaboration. Thus, not only people within each team should have different disciplinary backgrounds, the collaboration with other teams should further stimulate interdisciplinary learning.

Fig. 5
figure 5

Research program organigram

Full size image

In the initial setup, we had two industry teams, one methodology team, and one team dedicated to piloting newly developed methodologies or technologies (see Fig. 5). Furthermore, we installed a synthesis team that had the task of overseeing all backlogs and the work of the individual teams. Above, it was noted that we had monthly core team meetings as well as regular progress meetings. Usually, these were bigger meetings where a representative from each team should be present. Apart from these meetings, the teams met (also virtually) as needed. Additional meetings across teams were organized whenever a collaborative task required it.

Story 2.1 and 2.2 (see Table 3) are about writing a baseline report, which had to be prepared by the two industry teams. However, the report needed to include specific information so that cases could be selected at a later stage. Thus, the RP needed to make sure that both reports cover the same topics and provide the needed insights. To assist the industry teams in this endeavor, collaboration with the methods team was initiated. In monthly meetings, upcoming issues were discussed and further worked on in the following sprint. Both teams could suggest activities. For example, following the request of the industry team, the methods team investigated if a literature query could give insights about relevant past and ongoing projects at the university. The results were fed back in the next sprint review meeting to inform the next sprint. Thus, instead of providing detailed instructions on writing the report upfront, content and structure were developed through an Agile process.

Due to the Agile nature of the RP, the organizational structure of the program is always up for debate. Agile does not only apply to processes but also to organizational structures. Thus, if realized throughout the process that certain structures are not optimal, they can be changed. Since the organizational structure impacts the output (Perkin 2020, p. 82), we had discussions about having pathway instead of industry teams. The baseline report is one output, and these reports are based on the industry team division. An alternative would have been to provide four reports using the pathways as starting point. If this would have been decided at some point, it could be assumed that we might have changed our organizational structure as well. Apart from the industry team structure, we also discussed the merging of methodology and piloting teams, which they did.

Vision

As mentioned above, Agile teams work autonomously. The stories and tasks provide goals that need to be achieved (e.g., writing a baseline report or writing a communication strategy). However, how these goals are fulfilled is up to individual teams (Perkin 2020; Pope-Ruark 2017). Thus, Agile processes should not support top-down dictation but rather facilitate creativity and autonomy. Though, if teams are free to find their own solutions, one needs to make sure that the solutions align with the overall vision. Accordingly, to make Agile processes and organizational structures work, the vision needs to be clear. Perkin (2020, p. 118) summarizes: “Stubborn on vision, flexible on details.”

A vision statement is part of the RP’s research proposal. However, the question of the bigger picture remained present. Researchers within the individual teams frequently requested a discussion about the vision as this would help them design their operations. Perkin (2020, p. 119) highlights: “The point is that a vision should be strongly directional, but it should also be repeated again and again, articulated, referenced and modelled at every opportunity by the leaders in the organization as a way to reinforce, embed, compel.” The statement of Perkin (2020, p. 119) proved to be correct in the RP. The processes within the RP strongly indicated the relevance of being clear on the vision. The need for a clear vision has also been reported in other interdisciplinary projects (Cairns et al. 2020; König et al. 2013, p. 266) and in a large-scale survey on the success of Agile projects, Serrador and Pinto (2015) found that a lack of a clear vision can have a negative impact on the project’s success.

Interdisciplinarity and diversity

Agile literature refers to multidisciplinary or diverse teams (Pope-Ruark 2017, p. 62). Multidisciplinary work is encouraged to increase creativity, out-of-the-box thinking, and thus innovation (Gardiner 2020; Moirano et al. 2020). “The multidisciplinary composition of the team is important not only in delivering outputs, but in encouraging diversity. […] Yet performance and creativity improve with greater diversity (including cognitive diversity and having a substantive range in views about how the work should be structured and executed)” (Perkin 2020, p. 69). Further, it should help teams to offer end-to-end solutions. Thus, the solution developed by the team is complete rather than partial.

The RP consists of an international research team with different disciplinary backgrounds (social science, natural sciences, and engineering). Like other interdisciplinary projects, communication between different disciplines has not always been easy. However, communication issues often led to a refinement of definitions and an awareness of jargon. Thus, when working in an interdisciplinary setting, researchers might become more aware of how to communicate something to a wider audience. For example, conceptually, the materials transition is based on the idea of reducing carbon emissions. However, it was not clear if the materials transition and, thus, the individual pathways would include indirect emissions (energy used to produce materials). Depending on the farming, certain materials would have been included or excluded within the scope of the RP. Due to the disciplinary background of involved researchers, this fundamental question of framing was held several times. In the end, it led to a much clearer delineation of the scope.

The RP also consists of researchers in different career stages. The communication strategy is one arena in which these differences became apparent. Early to mid-stage academics need to make sure to publish in scientific journals to either increase their chances of getting a tenure-track position or to fulfill their tenure-track requirements. This goal might be less relevant for more senior researchers, who focus on other career goals. Furthermore, researchers are working in different contractual situations. Therefore, some favor communicating results within the time frame of the RP, while others can exhibit more flexibility. Hence, depending on the respective situation of a researcher, certain communication pathways were preferred over others. Making the communication strategy an Agile task, a strategy could be developed that suited everyone’s needs.

Thus, if implemented well, working Agile allows to better meet the needs of involved researchers. Furthermore, the strengths of all researchers can be better integrated within the Agile process. That not only helps the project in reaching an optimal goal, it also helps researchers to further develop their skills.

Self-organization

An Agile organization is characterized by autonomy and self-organization. This means that teams organize and cooperate as needed to work on a task. The work on Stories 2.1 and 2.2 (see Table 3) described above, demonstrates self-organization. Collaboration between the industry and the methods teams was needed. The industry and the methods teams did not merge permanently but cooperated as needed. This cooperation was self-organized and ended when the task was completed. A new cooperation, including a new way of cooperating, started with a new task. After the cases were selected (which also followed an Agile process), we decided to adopt a new cooperation mode. The industry teams requested support for working on the cases. The case study work involves transdisciplinary collaboration, and the industry teams need methodological assistance to connect and collaborate with the stakeholders. To make collaboration easier, each case was paired with one researcher from the methods team to function as a linking pin between the case studies of the industry teams and the methods team. This allows to efficiently use the resources of the methods team, as not all members need to follow all cases closely. At the same time, the linking pins report back to the methods team and work on suitable methods to support the case studies. This new cooperation mode can be tested in practice and changed if needed. The members of the methods team could self-select the case they wanted to connect to. This increases engagement as researchers can follow their interests and expertise.

Another example of self-organization is that the methods team and the piloting team merged at some point. Conceptual overlaps between the two teams led, after iterative discussions, to the merging of the two teams.

Culture

A key success factor within interdisciplinary projects is the creation of the right culture (König et al. 2013). The Agile process is geared to facilitate learning, adaptation, innovation, and creativity. The installment of reflective sessions at the end of every sprint alone, will, however, not be sufficient to support creativity or learning. Additionally, the right culture needs to be created and maintained. People need to feel safe to share failures, successes, criticism, feedback as well as crazy ideas (Mayer et al. 2022). Cairns et al. (2020) indicate that, due to its risky nature, interdisciplinary work can evoke anxiety. Arguably anxiety does not support creativity. Therefore an atmosphere needs to be created that counterbalances such negative emotions. Freeth and Caniglia (2020) point out that when managed correctly, a certain degree of discomfort can be conducive to creativity. Annosi et al. (2020) highlight the importance of culture and norms for Agile teams to be creative. The behavior of teams and their members is oriented toward set norms and incentives. If learning and innovation is the goal, then a culture needs to be established that supports these behaviors.

An Agile culture supports transparency, openness, and mutual respect. The Agile culture should support a feeling of community and working towards a common vision. Accordingly, the culture should support people’s commitment to the project and individual tasks. In contrast, the Agile culture should not support overly competitive behavior that compromises team cohesion.

Incentive structures within academia do not necessarily support collaborative work (Pope-Ruark 2017, p. 59). Conducting interdisciplinary research is challenging, and it might take longer to generate publishable output (Leahey 2018). Some report that it might be more difficult to publish interdisciplinary work (de Bakker et al. 2019). Since publications are a requirement to advance on the academic career leader, some might be repelled from working in interdisciplinary teams. In more general terms, academia is a highly competitive field with only a few permanent positions (Sovacool 2023). Thus, most academics live in a state of constant competition for the next contract. This competitive culture is not necessarily supportive of collaborative endeavors. Accordingly, to make the Agile process work, depending on the environment, much effort needs to be put into creating a safe and cooperative work environment.

Pope-Ruark (2017, p. 63f.) suggests starting the Agile project with a Sprint 0 meeting, which ideally is a day-long team-building event. At this event, the first steps of getting to know the team members and building trust are made. Pope-Ruark (2017, p. 63f.) briefly introduces different activities that help to identify a) personal goals, competing interests, and intended commitments, b) understand each other's weaknesses, c) determine shared goals, d) agree on expectations and behaviors, e) agree on all logistics. She (Pope-Ruark 2017, p. 65) highlights that creating a good work foundation early on is detrimental to the success of the project.

The RP holds bi-annual community meetings to which the core RP members, as well as adjacent program members and externals, are invited. At these events, different interactive activities, such as world cafes, vision exercises, walk & talks, mini-workshops, and interactive poster presentations, are organized. At different stages of the RP, the purpose of the community meeting changed slightly. For example, at the second meeting, we used the majority of the time to reflect and talk about our personal visions, how our strengths contribute to fulfilling this vision, and which skills we would need additionally to reach our goals. At the third meeting, new adjacent short-term project members were welcomed, and methods that could be applied to facilitate stakeholder interaction were tested. Some of these activities have similarities with the activities suggested by Pope-Ruark (2017, p. 63f.).

People

Agile focuses on creativity and disruptive innovation. To facilitate innovation, not only the right processes, structures, and culture need to be in place. It also needs people who have creative skills and are creative problem-solvers (Perkin 2020, p. 182). It is “[people] who can see the change that’s needed and have the passion and enthusiasm to advocate it and bring it to life. People who are willing to go against the flow (ibid.).” Further, the Agile process builds around adaptability. Thus, people need to be able to cope with the adaptability of the project.

While Agile aims at supporting creativity, not everyone will thrive in an Agile setting. Not everyone wants to work in a self-organizing setting and continuously adapt work in progress to incoming feedback. Furthermore, Agile projects are inherently interdisciplinary, which might also not suit every person (Freeth & Caniglia 2020). It needs to be acknowledged that people’s personalities and character traits are different, and some might struggle with an Agile process. Annosi et al. (2020, p. 561) report that for some people, Agile settings create “serious difficulty in generating innovative ideas […].” Thus, it cannot be expected that every researcher excels in an Agile setting. Additionally, working in an Agile setting might be a new experience. Previously acquired knowledge and experiences impact how we perceive new knowledge and experiences. When researchers are only acquainted with linear processes, they might struggle with an Agile approach. Thus, people first need to learn to navigate this new setting which takes time.

Due to these circumstances, researchers might be confronted with an anxiety or discomfort-provoking situation (Cairns et al. 2020; Freeth & Caniglia 2020). Freeth and Caniglia (2020) suggest that researchers need to learn the necessary skills to profit from collaborative environments. Tying back to Agile culture, the right culture is vital to create a safe and invigorating environment that supports this learning process (Perkin 2020).

In the RP, people handle the Agile setting differently; some struggle more than others. Those who struggle would like more guidance and linear structures. Based on the experiences within the RP, it is vital that these people are supported and understood. Referencing back to the Agile culture, if people do not feel understood or feel that they cannot voice criticism, trust might get damaged. If there is no access or not enough time to attend training about Agile processes, it is up to the project leader to support respective people in the best possible way.

Leadership

Since interdisciplinarity is a core aspect of Agile, leadership requirements in Agile settings are similar to those in interdisciplinary projects. König et al. (2013, p. 267) state that “[i]nterdisciplinary management implies a distinct and diplomatic form of temporary leadership […].” This is similar to Agile settings. Perkin (2020) discusses leaders’ role in coordinating tasks, leading the team, and creating the right atmosphere. “Great leadership has always been about great communication, exceptional skills, and the ability to get the very best out of people and teams, but this has never been more important in this highly changeable age. […] Leaders who can create an environment that not only enables people to do their best work, but one that enables people to do their best work as part of a highly functioning team, can add disproportionate value. This requires high levels of self-awareness” (ibid.: 249).

As stated earlier, successful Agile processes require the right structures as well as a suitable organizational culture. If these factors are not in place, Agile processes might not support learning, creativity, and innovation. Leaders have a key role in creating an environment that supports learning and creativity. For example, Annosi et al. (2020), show that learning within Agile settings is not a given but needs proper support from leaders, who can actively influence the processes. Using a multi-level approach to learning, they explain how leaders in Agile processes can steer learning through, e.g. increasing or reducing certain feedbacks, creating routines of interaction among and within teams, or making sure that new knowledge is accessible to everyone. Mayer et al. (2022) also indicate that leaders need to create an atmosphere that rewards learning, by allowing failure.

Mayer et al. (2022) bridge leadership with design thinking and Agile, indicating that leadership “[…] refers to the desire to understand human needs, emotions, behaviors, and values as a basis for gaining inspiration for one’s own work “ (ibid.: 158). They highlight that leaders need to introduce an open and reflective stance and allow and support a diversity of opinions. Mayer et al. (2022) discuss four key tasks of leaders: (1) enable autonomy, (2) foster psychological safety, (3) build systems to learn and experiment and (4) create and communicate a clear vision (see also Perkin 2020, p. 187f.). Further, since Agile does not support top-down management, leaders rather have the role of facilitator (Hidalgo 2018, 2019).

Of course, the creation of the right culture is a two-way street. In expanding on creating learning spaces within interdisciplinary projects, Freeth and Caniglia (2020) propose that those who design and implement the project are responsible for creating the conditions for learning. Though at the same time, researchers within the project need to make use of the provisioned conditions.

Within agile projects, there are not only overall project leads, but also team leads. These also need to have certain skills and facilitate ongoing activities. Pope-Ruark (2017, p. 68) indicates that such leads might not be appointed but might emerge organically. It is usually “a team member who is organized and detail oriented [and who] can help everyone see both the immediate and long-term activities to be done based on the collective goals” (ibid). When the methods and the piloting team merged, the team lead changed as well. This change was also part of the Agile process and was owed to the time scarcity of the previous team lead. Ideally, team members should work full-time on one project at a time (Ochoa et al. 2021). This is rather unrealistic in academia. However, the lead position asks for more time investment from the respective researcher. If the respective researcher does not have enough time, a leadership change might be advisable. As described by Pope-Ruark (2017, p. 68), the new team lead emerged organically exhibiting leadership traits.

Discussion

While the RP intuitively applied Agile principles, a more purposeful application of these principles might still have been better. In the spirit of Agile, this article, and this section, serve as a means to reflect and learn. Therefore, some lessons learned pertaining to the principles are shared.

Table 4 summarizes the Agile principles and how they have been applied in the RP. Aspects refers to the five clusters of principles that have been introduced in the “Agile principles” section. Extracted from the “Applying Agile Principles” section, the “Implementation” column summarizes how the principles were applied in the RP. The “Challenges” column outlines shortcomings in implementing the principles. These challenges are further discussed below.

Table 4 Overview of the implementation of Agile principles in the RP
Full size table

Process

In retrospect, the vision should have been discussed in more detail and in earlier stages of the RP. The vision is the lighthouse all teams use for orientation. Thus, if the lighthouse disappears in the mist, teams can get lost easily. To address the vision question, the RP organized a workshop in which everyone had to fill in an individual reflection sheet. The reflections were then discussed in smaller groups. Results were further analyzed and used to carve out a clearer vision. However, in the end, the RPM decided to not use these results and started over again. Thus, clarifying the vision remains an ongoing issue.

Structure

Communication channels could improve. We did not use tools such as Kanban.Footnote 7 That makes it difficult to stay up to speed with what every team is working on. We are using MS Teams to organize the RP, but the MS Teams platform is only rudimentarily used for communication purposes. Due to the high workload of involved researchers, communication suffers at times. For example, it has been difficult for team representatives to attend meetings. Since these meetings are often the only means to inform everyone about the ongoing task, absence leads to a lack of information.

The RP consists of an international research team with different disciplinary backgrounds (social science, natural sciences, and engineering). Optimally, Agile teams are located in the same building and in close proximity. However, as common at universities, faculty buildings are spatially dispersed (Cairns et al. 2020). Thus, the Agile teams within the RP were not located in close proximity. Though, since some researchers are part of the same discipline (but not necessarily part of the same team), contact among those researchers is closer. This then can lead to information asymmetries where not everyone is up to date due to the usage of informal information channels. The spatial distribution of researchers might also counter endeavors to increase interdisciplinary collaboration. By spatially remaining in disciplinary silos, researchers mostly communicate with researchers of the same discipline.

Culture

Additional teambuilding events that are dedicated to the RP team could have helped to generate a group feeling. The RP organized bi-annual community meetings. Though, while these community meetings are great for sharing ongoing work, they might not be optimal for building trust among the members of the core RP. That is as not only the core of the RP participates, but researchers from adjacent projects as well as externals. Thus, these meetings are directed more toward the greater community. For team building events, it needs to be noted that these should be inclusive. For example, drinks in the afternoon might conflict with some researchers’ childcare responsibilities.

The competitive culture within academia might generally not be supportive of creating team cohesion and promoting collaboration. Thus, the needed effort to create a team spirit should not be underestimated. Competition between the two industry teams was created to increase the respective team’s effort. However, this competition might have reduced collaboration.

People

In reference to anxiety-provoking circumstances, it can also be noted, that the seeming lack of structure within the RP might have had negative impacts on some researchers. Even without having the Agile terminology, the leads of the researcher program could have communicated better about their idea of the research process.

Leadership

Communicating the project process could have been better. Some researchers were frustrated about the messy character of the research process. On the other hand, the RPM voiced a lack of understanding for team members not being able to navigate the messy set-up. With the introduction of the Agile terminology and process logic, some researchers started to better understand the iterative nature of the project (including myself). Leaders have nurtured competition (e.g., among the industry teams), by continuously drawing comparisons on their performance. Likewise, comparative languageFootnote 8 used in meetings was not inhibited. Generally, competition is facilitated by using terms such as “progress meeting.”

Based on the challenges identified the following directives are suggested.

Have a clear vision throughout the research process:

Whether the vision is co-created with the whole team or not, the vision needs to be clear early on. The vision of the program should be discussed repeatedly to ensure the coherence of planned and ongoing activities.

Keep everyone in the loop:

The use of a (electronic) dashboard that shows the current state of the process is advised. Simple means to keep everyone informed should be prioritized. Apart from a Kanban one could use a (electronic) whiteboard to visually summarize the ongoing and completed tasks. This can be linked to more detailed information about respective tasks (e.g. adding a link to a folder).

Create and nurture a supportive culture:

Competitive language should be undercut. The goal should be to go through the finish line together. Thus, collaboration and support need to be more important than progressing faster than another team. A team-building activity at the beginning of the research process is advised to introduce the Agile process and discuss expectations. More importantly, such an event should build the basis to create a community. Continuous efforts should be put into nourishing the community.

Be clear about the process flow:

Everyone involved needs to be informed about the process flow. Potential challenges of the Agile process as well as means to overcome them need to be discussed. Furthermore, since Agile is adaptable, adaptations of the process should be explored if the need arises.

Leadership that provides clarity and nurtures collaboration:

Leaders need to be role models. Thus, a collaborative, supportive stance needs to be embodied by the leadership. The leadership is also responsible for clearly communicating the vision and process flow.

Conclusion

Applying Agile principles to a RP that is Agile by accident, shows that there is more structure in an ostensible mess than it first appeared. However, the challenges the RP faces indicate that the RP would have profited from applying the Agile principles consciously at an earlier stage. On the one hand, Agile allows and aims at adaptability. Thus, in principle, it is not necessarily negative if challenges faced are corrected throughout the RP. As outlined above, structures can be changed as a response to changing priorities and needs. On the other hand, certain challenges are more difficult to correct at a later point. This especially pertains to the aspect of culture. Once a competitive culture has been established it is more difficult to nourish cooperation. The lack of structure and guidance led to frustration within the team that could have been avoided. As outlined, other interdisciplinary research projects dealing with complex matters might also have organically emergent processes and structures and might not be aware that using Agile principles could provide a useful frame and guidance. Thus, hopefully, this article contributes to informing the research community.

The lessons learned allow to give some recommendations for applying Agile consciously. Clear communication is key. The Agile process needs to be explained well, especially if researchers never worked in this manner before. Enough adaptation time needs to be granted and leaders need to be open to learning about the challenges people face when working Agile. Without a detailed plan, the vision has a vital role. The vision should be established early on, and it can be used as a means to build team cohesion. To support collaboration, competition should be minimized. While Agile does not require defining deliverables or milestones, it might nevertheless be advisable to agree on a set of deliverables.

It is difficult to evaluate whether being Agile by accident had a positive or negative impact on the RP. Since no deliverables or milestones were formulated, there is no benchmark against which the success of the RP can be evaluated. This might be one of the major drawbacks of Agile. Nevertheless, for the RP, Agile has proven to be a suitable frame that helps to explain the ostensibly messy character of the RP’s processes.

Interdisciplinary research projects that deal with complex issues, such as sustainability, might require a different way of organizing and planning project work. Interdisciplinary work might aim at increasing creativity, though rigid project structures might not enable creativity. Research on complex problems might need approaches that support learning and out-of-the-box thinking, but traditional waterfall project approaches might not provide enough learning spaces. An Agile approach to organizing such projects might offer a viable framework to facilitate interdisciplinary collaboration and create learning spaces.

This article outlined how Agile principles can be applied in an academic research project. Due to the need for more flexibility and adaptability of research dealing with complex problems (Spangenberg 2011, p. 282), project managers might intuitively apply alternative ways of organizing a project. Researchers might intuitively apply Agile ways of doing research without being aware of the Agile principles (Hidalgo 2019). Interdisciplinary work can be a mess, and the seemingly unstructured nature of interdisciplinary processes can provoke anxiety (Cairns et al. 2020). Using the Agile principles can bring some structure into this mess, while maintaining learning spaces. The application of Agile principles can create the right atmosphere (culture) to facilitate learning and creativity. Literature about Agile can also enlighten which skills and qualities researchers and leaders should have, strengthen, or acquire to create these learning spaces. By providing guidelines on how to structure the creative process, Agile might have the potential to reduce the power of the anxiety-provoking mess.