Introduction - the Educate Programme

The need for closer engagement between the educational technology industry (who create educational technologies), educators and learners (who consume educational technology) and academics (who know how to research its efficacy and effectiveness) was highlighted by Luckin (2016) in her construct of the ‘Golden Triangle’ of evidence (Fig. 1).

Fig. 1
figure 1

Adapted from “The Golden Triangle of Collaborative Evidence-based Product Design” (Luckin, 2016)

Full size image

The London-based EDUCATE programme (UCL Institute of Education, 2017–19) was designed to enable 250 small and medium-sized enterprises from the educational technology industry to forge a closer collaboration with academia through a six-month research training and mentorship programme. Simultaneously, a design-based research methodology was adopted by the UCL team (the authors and their colleagues) that aimed to evaluate the impacts on the programme on its participants (Clark-Wilson et al., 2021). This article focuses on one of the theoretical constructs, the logic model, which evolved during the programme as an important boundary object, enabling academics and entrepreneurs to both communicate and learn from each other. We extend the work of Cukurova et al. (2019) by evaluating the specific impacts of the logic model for a sample of the participants of the programme. This evaluation was designed to explore the use and effectiveness of the logic model as a boundary object within the EDUCATE programme.

A logic model, which is a visualisation of the theory of change for an intervention or initiative, maps the route to the desired impacts by detailing the intermediate steps between the intervention and its goals. Logic models have been prevalent in general programme evaluation literature since the 1980s, and they are commonly used within both private and public sector organisations to describe and evaluate their programmes (McLaughlin & Jordan, 1999). However, in these settings, the traditional interpretation of a logic model perceives the end user as a client or customer, which suggests a tension for education programmes where the customer is rarely the end-user, but rather a learner or teacher.

Zhao et al. (2008) offer a critique of the evaluation literature with respect to educational technology innovations and highlight the shortcomings such as, “researchers are often confined to take a simplistic approach to studying technology” and “the obsession with assessing outcomes in the end”. Similarly, the trend towards large-scale experimental studies of educational innovations has been extensively critiqued by Simpson (2017, 2018) and Cukurova and Luckin (2018). In an attempt to bring in more clarity to the process of impact evaluations, Zhao et al. (2008) describe the ‘classic’ logic model as consisting of three components: inputs, outputs and outcomes. The inputs are considered neutral in that they are the resources needed to implement the innovation (human, financial, organisational and community). In this model, the ‘magic’ of the intervention is considered as the resulting processes and products – the activities, events, services and processes. It is these processes and products that are hypothesised to have a causal relationship to the (desired or planned) outcomes.

A more comprehensive logic model structure is to consider a five-stage model developed by the W.K. Kellogg Foundation that comprises: inputs (or resources); activities; outputs; outcomes and impacts (1998), as shown in Fig. 2.

Fig. 2
figure 2

The Basic Logic Model (W. K. Kellogg Foundation, 1998, p. 1)

Full size image

By separating the activities of the innovation, that is the (human) engagement and interactions with the resources that have been provided, the outputs become more tangible. Furthermore, by describing an outcome as ‘a change in an input’, it allows for a temporal consideration in that outcomes can be small and immediate, or by researching the intervention over time, outcomes can be more substantial and lasting. The final element, the impact, is thus the ‘big change’ that is usually closely aligned to the overarching goals or vision for the innovation.

The EDUCATE programme challenged its participants to consider the theory of change for their product or service and offered the logic model construct as a boundary object to support this process. The research reported in this exploratory study addresses the following research questions:

RQ1 How was the logic model used as a boundary object by companies and researchers within the EDUCATE programme?

RQ2 How effective was the logic model as a boundary object to support companies in their development of a theory of change?

Overview of the UCL Educate Research Training and Mentorship Programme

The overall purpose of EDUCATE is to advance the efficacy of EdTech through an evidence-informed process by making the best research evidence and practice accessible for educators, researchers and technology developers. The project will enable developers and educators to integrate four sources of information: research evidence, local context, practitioner experience and judgement, and user values and preferences to further drive and inform EdTech design and implementations. (Cukurova et al., 2019).

The main goal of EDUCATE was to enable the educational technology (edtech) start-ups and small- and medium-sized enterprises (SMEs) to learn how to access and use existing research and to build the capacity within their own organisation to conduct research on their products. Such research practices might be used by companies at every phase of product development, to include: initial design and development of a minimum viable product (MVP); assessing findings from pilot testing and making product changes; and evaluating the impact of a product that might already be freely available or on the market.

From 2017 to 2019, the EDUCATE programme team worked with 252 London-based companies in 12 cohorts of between 7 and 23 participating companies each. The UCL-based research team members had prior professional experience of working both in educational technology research and in business and were employed to act as mentors for companies to guide their research journey. Between one and three participants from each company accessed at least 12 hours of research training and mentoring, provided by UCL researchers, over a 6-month period. The objectives of the research training were to support participants to:

  • understand the nature of evidence for educational technology: what it is; how to locate and interpret it; and why it is important;

  • learn how theories of change are useful within the design and development of educational technology and how to operationalise these using a logic model approach;

  • create and critique research questions, as appropriate to their particular technology and its educational context;

  • formulate a research proposal based on their resulting research question(s), become familiar with experimental and exploratory research methods, and gain a more in-depth understanding of the methods appropriate to their proposed research.

To demonstrate to the programme funder that a company had successfully participated in the EDUCATE programme, participants were required to develop a research proposal for a study they would like to conduct to inform the development of their product or business. As the EDUCATE programme progressed, the usefulness of the logic model was evidenced as a tangible output of mutual benefit to both the research team and the companies. Thus the logic model, rather than the research proposal, became the programme output required for reporting to the programme funder.

A more detailed account of the programme design and the key activities undertaken by participants is provided in Clark-Wilson et al. (2021). It is important to note that in addition to the research arm of the programme, EDUCATE also included training and mentoring on business and product development topics, such as product development and management methodologies, marketing and sales, social networking and pitching to potential investors.

Companies wishing to participate in EDUCATE were required to articulate how their product had an objective to improve learning, at any point in the learning process from early childhood, though formal schooling to university and beyond. Companies needed to have a concrete idea for their product and to have begun work on making that idea a reality, but they did not need to have an MVP upon entry to the programme. While some participants were still in this idea stage, others were financially profitable SMEs with products already on the market. From the 4th cohort onwards, companies were asked to self-assess their progress towards the development of their innovation as a means to build a sense of shared enterprise within each EDUCATE cohort (in the sense of Wenger, 1998). The Innovation Spiral, a tool developed by Nesta, one of the EDUCATE programme partners, was used to support this self-evaluation (Fig. 3).

Fig. 3
figure 3

The Nesta Innovation Spiral (Nesta, 2014)

Full size image

The distribution of these self-evaluations for the majority of EDUCATE companies from cohorts 4 through 12 is shown in Fig. 4, which highlights the diversity of their contexts.

Fig. 4
figure 4

Company self-evaluations of their Stage of Innovation (n = 217)

Full size image

Figure 4 shows that 70% of the enterprises were between innovation Stages 1 and 4, which indicates their early stages in the development process. This is also highlights the context of the EDUCATE programme, which was particularly designed to support this community. The six enterprises that participated in this study spanned Stages 2 to 6 (See Table 1).

Table 1 The six companies that participated in the study
Full size table

The EDUCATE mentorship process, and in particular the nature of the relationships that were formed between research mentors and their mentees (the company representative(s)) has been explored by De Ossorno Garcia and Doyle (2021). Their research advances understanding of the constructs that define effective mentorship relationships for accelerators such as EDUCATE. De Ossorno Garcia and Doyle’s analysis of the interpersonal process is of importance to further understand the definition of “good mentorship” within formal mentoring programmes and accelerator research for evaluation purposes. Constructs such as trust, decision making, personality and self-efficacy were deemed important for mentees, whereas mentors privileged career functions and aspects of the programme such as time and frequency of interaction, knowledge about research methods and programme goals. The research reported in this paper compliments these interpersonal constructs and expands on them by exploring the use of a key object that both bounded this mentor-mentee relationship and facilitated communication and knowledge-sharing across organisations to achieve programme and company objectives.

Theoretical Frame

The design and structure of the EDUCATE programme is grounded in the literature of socio-cultural learning, in which learning is considered a social endeavour (Vygotsky, 1978) that occurs through an individual’s interaction with others as part of a community of practice (Wenger, 1998). These theories also assert that knowledge is produced socially (Gunawardena et al., 2009) and that individual expertise cannot be built without social interaction, even when the learner has access to other learning resources (such as books, etc.) (Bransford & Schwartz, 2009). Communities of practice, within which this learning occurs, are defined as “groups of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis.” (Wenger et al., 2002, p. 4). Learning in a community of practice occurs through an individual’s participation and interaction with others (Lave & Wenger, 1991), and mirrors the learning that occurs naturally in society (Luckin, 2010).

In this model, learning is not limited to members within one community of practice; indeed learning often occurs when there is collaboration between people from different organisations or fields of study (Akkerman & Bakker, 2011). In such instances, which are becoming increasingly common as globalisation and specialisation of workforces increase, (Akkerman & Bakker, 2011), learning takes place on the boundary of two communities of practice (Wenger, 1998). Learning within a community of practice requires the same kind of participation and negotiation of meaning as working across boundaries. However, traversing the culture, language and norms of a different community necessitates objects or people to aid in these transitions. Hence, the boundary object concept was first offered by Star and Griesemer to describe artefacts that serve to coordinate the different perspectives of multiple communities (Star & Griesemer, 1989).

Boundary objects can be artefacts, documents, concepts or other objects that exist between or within several intersecting communities of practice (Wenger, 1998). They are robust enough to retain their original meaning in a variety of contexts, yet flexible enough to be adapted to the needs of the local user (Star & Griesemer, 1989). In their original work on the use of boundary objects to understand the collaboration of scientists from different fields, Star and Griesemer (ibid) classified four types of boundary object:

  • Repositories: Groups of objects that are accessible by members of multiple communities and labelled or organised in a manner that everyone understands.

  • Ideal type: A diagram or type of atlas that is intentionally vague. Rather than describing any one situation or item, it draws meaning from many domains.. The example ideal type provided by Star and Griesemer in their original text is that of a species which, as a concept, describes no one specimen in particular but provides a framework which members of differing communities can adapt and use to communicate with one another.

  • Coincident boundaries: Star and Griesemer describe this type of boundary object by using the example of a map. Multiple maps of a particular location may have the same external boundaries but different labels and markings internally depending on what needs to be communicated or portrayed.

  • Standardised forms: As the name indicates, these are forms that allow different groups of people to enter information in the same manner regardless of their background, expertise or experience.

However, even with access to boundary objects, learning at the boundaries cannot occur without human communication and collaboration (Akkerman & Bakker, 2011). Boundary crossers, or brokers, can navigate from one community of practice to another or, through multi membership, be members of more than one community of practice (Wenger, 1998). These individuals are needed to interpret, explain, negotiate meaning and create the practice that occurs at community boundaries. The use of boundary objects by such brokers to facilitate knowledge sharing across and between organisations is becoming more prevalent in literature from the fields of entrepreneurship and business model research (Roessler et al., 2019). Boundary objects are increasingly seen as conceptual tools to support collaboration between individuals in organisations with extensive domain-specific knowledge (Joeger & Pedersen, 2020). The EDUCATE programme made use of ideal types of boundary objects, in the form of ‘theories of change’ and ‘logic models’, to help collaboration occur between the communities of each entrepreneur’s EdTech start-up and academia.

If professional learning within and across the boundaries of academic and industry institutions is conceived as a social change within a complex community initiative, then there are clear parallels with Kurt Lewin’s seminal work, which offered theories to help understand the diversity of social change (Lewin, 1947). Writing in 1996, Schein reflected on Lewin’s contributions thus, 

power of Lewin’s theorizing lay not in a formal propositional kind of theory but in his ability to build ‘models’ of processes that drew attention to the right kinds of variables that needed to be conceptualized and observed. (Schein, 1996, p. 59).

Such ‘theories’ of change provide a structure to communicate the resources, processes and products for any initiative that is seeking to bring about change for its participants. A common tool within both the social investment and grant funding communities, the goal of a theory of change is to produce a diagram that conveys the essence of the initiative along with the underlying assumptions. The origins of the use of theories of change lie in the general programme efficacy literature (McLaughlin & Jordan, 1999). More recently they have been a key component of the design of randomised control trials in education in the United Kingdom as they enable externally-appointed evaluators to extract knowledge of the intervention from the designers such that an evaluation protocol can be developed.Footnote 1

Research Methodology

Given the unique nature of the EDUCATE project context, for which there were no existing theoretical frames, in this study, a qualitative exploratory study was deemed appropriate. Lincoln and Guba claim that the qualitative approach empowers the researcher to gain a more in-depth understanding of the phenomenon at stake by generating a range of data types that provide different perspectives (Lincoln & Guba, 2000). We adopted a grounded approach to the generation of data and subsequent data analysis, from which the theoretical ideas would emerge. The research was conducted between June and December 2020, which was six months after the EDUCATE programme had completed.

Sampling Method

Six EDUCATE companies were selected to investigate within the study (See Table 1), based on the selection criteria provided in Appendix A. Our grounded approach required us to select EDUCATE participant companies as case studies such that a contextual commentary could be developed that bore a relation to the emerging theory. This form of theoretical sampling does not stipulate a given sample size as the notion of data saturation indicates when to cease data collection (Cohen et al., 2011). In our context, the project resources (human, financial and time) do not enable us to assert complete data saturation, however the data gathered were sufficient to enable us to report valuable findings.

Data Collection

For this study, the data collection and analysis were conducted by the first two authors. This began with the analysis of data from a set of documents that had been created during the companies’ participation in the EDUCATE programme. In addition, a 30-min interview was conducted with the main programme participant(s) from each of the companies, which took place after their participation in the programme was completed (See Appendix B for the interview questions).

The following three artefacts relating to the sample of companies were collated for document analysis:

  • Research mentors’ informal notes of mentoring meetings held with the company.

  • Formal summaries of these mentoring meetings that were held centrally within the EDUCATE programme database.

  • Copies of the logic model(s) for each company, which in some cases included up to three versions or iterations.

These documents were analysed as outlined in section 4.3 and, once analyses of these artefacts were complete, an emergent set of codes were developed to inform the interview protocol. Interviews were then conducted with the representative(s) of the companies who had participated in the EDUCATE research training sessions and worked with one or both of the authors in their roles as research mentors. The purpose of the interviews was primarily to triangulate the concepts that emerged following the analysis of documents in relation to the study’s two research questions. The authors were also interested in understanding the companies’ perceptions of the longer-term impacts of their programme engagement in relation to our research objectives.

The interviews were held via video conference within the university’s online communication platform (Microsoft Teams) and the resulting interview transcripts were maintained within the secure data portal. The first two authors attended all interviews and the interviewees were the representative of the company who had participated in the EDUCATE research training sessions and worked with the one or both of the authors in their roles as Research mentors.

Data Analysis Process

The data analysis for this study involved coding the artefacts that could demonstrate the role of the logic model as a boundary object to support companies and researchers in the EDUCATE programme. The coding process involves developing and applying a series of codes to help identify themes and make connections across the various data sources (Bazeley, 2013). In preparation for coding, codes were developed from the literature on boundary objects as well as from the researchers’ own knowledge of both the individual companies in the sample and the objectives and content of the EDUCATE project as a whole. The resulting list of codes related to data that evidenced:

  • the logic model as a tool to support all aspects of communication;

  • the logic model as a tool to support all aspects of learning;

  • challenges in using and applying the logic model in the company context;

In accordance with Bazeley (2013), further coding evolved during the act of coding, and as a result the more nuanced codes that resided within these three themes changed, were added to and were deleted as coding progressed. This was particularly evident as the process of transcribing and re-reading the interviews began, in preparation for coding. In this respect, a grounded theory approach was used, by which researchers were guided by theory emerging from data analysis in order to develop any subsequent data collection and analysis (Glaser & Strauss, 1967). For example, the broad theme “Communication” evolved to include sub-themes such as: the ontology or language of the logic model; communications between the company and the EDUCATE researchers; communications between employees within the company; and communications between the company and external agencies.

A dynamic codebook was created and shared between the two researchers so that codes could be refined as the coding advanced. This codebook included the overarching theme of the codes, the code name, a definition for each code and an example of how it was applied in the texts. This detail helped make sure that no codes overlapped, which could negatively influence data analysis (Cohen et al., 2011). The final version of the codebook is shown in Table 2.

Table 2 The codebook for the study
Full size table

Similarly, it is vital that when multiple researchers are coding, each applies codes in the same manner. To ensure consistency in coding, each researcher coded one interview for the other to review. Any discrepancies were discussed and resolved and the codebook was edited accordingly. Some codes that had not been applied were deemed no longer relevant and were deleted, while others were added or modified. Once the final codebook was agreed, one researcher finished coding the remaining interviews.

Role of the Researchers and Ethical Considerations

The research was conducted under the ethical protocols of UCL Institute of Education (Ethical Approval Reference: REC1056 and Data protection registration number: Z6364106/2018/03/121). As such, all participants gave their informed consent to be involved in the study and had the right to withdraw at any time. Furthermore, as professional relationships had formed between the authors and all of the case study participants, additional care was taken to conduct the interviews very formally according to the agreed interview protocol. The researchers were highly aware of the confidential nature of the sensitive commercial information regarding the design and implementation of the different products and we recognised the urgency of not compromising the trust that participants had placed in them. All participants were sent a copy of this article with the invitation to highlight any company specific data or comments that they preferred not to be in the public domain. No companies highlighted any such issues.

Findings

We structure the findings in accordance with the themes that emerged from the coding process, which we first explain and then exemplify from the research data.

The Logic Model as a Tool to Support Multiple Aspects of Communication

All participants commented on how the logic model had supported them to communicate aspects of their educational technology product or service to others. Central to this communication was a shared understanding of the underlying ontology of the logic model construct within the context of the particular educational technology and its assumed theory of change. Whilst the terms input, activities, output, outcomes and impacts were widely understood in an English grammatical sense, each took on a unique meaning for participants. Several commented that their initial stages of logic model development involved simply trying to understand what the categories meant in terms of their product. “I spent quite a bit of time trying to really figure out exactly what each of the logic model categories really meant,” noted the co-founder of Company E. “The difference between...outcomes and impact….I think that was definitely something that I kind of struggled with initially,” said the founder of Company B. This feedback resonated with our experiences as research mentors, in which role we had been required to explain and clarify these terms on multiple occasions.

Communication within the Company

As only one or two of a company’s employees were able to attend the EDUCATE research training and mentoring sessions, the SMEs participating in EDUCATE often needed to collaborate with other team members to develop the logic model. The attendee from Company A drafted the initial version of the logic model herself but then discussed it with the company’s founder and another team member to refine it. She said her team members helped with, “refining the and pinpointing...the specific activities. I think before I had shared it with them, it was slightly blurrier and there wasn’t as much clarity for each kind of element.”

Furthermore, EDUCATE participants used the act of creating the logic model or the logic model itself as a way to discuss and agree upon key elements of product design and development or company strategy. In addition, more than one company expressed the positive aspects of the logic model as a tool for sharing company plans with other members of the team. For example, Company B had two full-time employees at the time of their participation in EDUCATE but employed contractors to help with product development. Their founder noted, “I’ve used it for some team members…. For example, our CTO or our design lead - they definitely had to kind of engage with [the logic model] to get a broader overview of what the company does.” Similarly, the co-founder of Company E also used the larger categories of the logic model in team conversations about product development, saying,

I’ve had to educate [the team] but in a very hands-on kind of way…: leading the discussions on feature design and prioritisation and product roadmap prioritisation where I … brought in the questions from the logic model during the discussion time.

Communication between the Company and EDUCATE Researchers

In order for the UCL researchers to guide companies on their research journeys, the researchers had to have a certain amount of familiarity with the companies, their product and its intended goals, and their objectives for the research they wished to conduct in EDUCATE. The logic model and discussions about its development and revisions during mentoring meetings aided in the communication between EDUCATE companies.

Some of the conversations between researcher and company involved simply understanding how to apply the logic model categories to the company’s own product and context. The co-founder of Company E noted,

A lot of those discussions on the mentoring side were to try and figure out whether we had our assumptions right about each of the steps and whether we knew when and how that step would be completed so that the user could move...from the inputs to the activities and then from the activities to the outputs. I remember the sessions being mostly about really understanding how to use the model and identifying where we were missing the crucial parts in our models.

Likewise, mentors’ notes after the meeting reflected these discussions. One of the authors wrote the following about one of the companies:

They made a great start on their logic model but we discussed some areas where things weren’t quite right. Some of their resources were actually assumptions about the resources; some of their activities were actually outcomes of the activities. Outcomes and impacts were combined. Assumptions were missing or not called out as such.

These discussions about meaning ultimately led the companies to make conclusions about their product, the development cycle, or focus that they might not have done without the mentor input. Company C noted,

I don’t think we could have had [these conversations] ourselves...for what goes into four and five [outcomes and impact]. [They] help us shape our research in a way that …hopefully will help us show in the future that we will achieve those outcomes and those impacts.

Communication outside the Company

Companies also spoke of being able to use the logic model in interactions with stakeholders and exploration of other business opportunities outside of the company. Multiple start-ups discussed being able use the content from their EDUCATE logic model in applying for grant funding. The logic model provided information for the grant applications that described the activities, outcomes and potential long-term societal impact of the product.

Companies were also able to use the assumptions sections of the logic model to inform potential partners or customers about the conditions necessary for successful product implementation. For example, when customers asked “How will [the product] work in our area?” Company A was able to point to the steps in the logic model and use the assumptions to say, “Here are the core implementation measures and here are the optional [ones].”

The Logic Model as a Tool to Support Multiple Aspects of Learning

Companies who wrote and revised more than one version of a logic model were able to look back at how their learning progressed while on the EDUCATE programme. For example, when the co-founder of Company F re-examined the multiple versions of their logic models before his interview with the authors, he noted, “When I looked at the first compared to the last version, it was clear there was a big difference...what… is objective and what is subjective and what are the hypotheses we need to unpack and test?”

Many companies also noted how the conversations with their mentors about the logic models helped them come to a realisation either about the logic model itself, or about how they should focus their product or their thinking about their product. Company D noted,

I do remember there was a particular call you and I had where I think I was still sticking with the old way and trying to...create one logic model for the whole thing and there was a moment when we realised...this needs to be two separate logic models, one for teachers and one for pupils.

Several of the companies interviewed have continued to use logic models in some form beyond their time in the EDUCATE programme. They either use the same or a slightly revised version of what they created during EDUCATE; they have created new logic models; or they have kept the principles of logic models in their regular planning even if they use databases or other software to help with planning. For some, it is the spirit of the logic model that is embedded in the company’s consciousness. For example, Company A said,

While we don’t ever refer back to the document that is the logic model, as we go forward with designing content or features within the app, it’s very much there.We make hypotheses around how the features will impact our users.

The Logic Model as a Tool for Product Development

As the following data extracts exemplify, all of the companies in the sample expressed that the act of creating the logic model helped them focus either the audience their product or feature was targeting, or the aspect of the product they needed to develop, or both.

It helped us [have] razor sharp focus on the true elements of what [the] product was meant to be. When we came to [EDUCATE] we have loads of things and...the logic model really helped us...put the focus on the ones that truly were in our hands to work with. (Company C)

[The logic model] often lead us into: (a) having a better understanding about areas of blind spots....I think also it framed our thinking when it came to product development. (Company F)

I think [the logic model] helped encourage us to not be everything to everyone, and try to actually define what value we are actually providing -- and how -- to whom. (Company D)

The mentors’ notes from meeting sessions with companies echo this, reflecting on the discussions with companies about revising their logic models so that the target audience for inputs, activities, outputs and so forth, was clear.

We reviewed the revised logic model that breaks the use of [the product] into steps, and makes some assumptions for [certain users]. The discussion centred on the methodology to find out what the use cases for different [user groups] might be, depending on where they are in the research process and the type of [output] they are looking to produce. (Mentor meeting notes, Company E)

Challenges in Using and Applying the Logic Model in the Company Context

For some start-ups, the challenges with applying the logic model categories were related to the stage of development of the company or their product in the innovation cycle. Some companies whose products were in the earlier stages, such as Companies B, C, and E struggled to understand how later components of the logic model would apply to their products,

I also think that when you're early stage, it's quite difficult to...think that far out when you're still figuring out what exactly the activities are that you're meant to be doing. (Company B)

Because we are such at the beginning of our journey..., the most beneficial part of the model for us is still stages one and two [inputs and activities]. So we're talking about the activities…. When we come to stages four and five [outcomes and impact], they feel very far from us just yet. We are looking for something very tangible. Stages one and two are very tangible. (Company C)

In our situation, I think the impact part was much less relevant, especially because we were still at the point where we're finding our product-market fit, and so the kind of research and activities we want to run and we want to test and do research on don't really factor in impact as much as the other parts of the logic model. (Company E)

Others felt that the linear nature of the logic model did not match their thinking about the product cycle. For example, Company B’s co-founder would have preferred to move from right to left (from high-level impacts to inputs) in constructing the logic model, rather than beginning with the individual inputs that his company would provide. “I struggled with the direction that you’re meant to approach it... I kind of felt like it might have been easier to model it backwards. So starting from the impacts [and] going to the outcomes.”

Companies A and D, who had well-established products but were designing new features as part of EDUCATE, were concerned about how to accurately represent their various users and stakeholders.

I suppose that a difficulty was trying to ensure that [the output] wasn't based on assumptions. So I think while I do know that's true in some cases...I almost felt like [for] some of the outputs I wanted to...go out to our practitioners and ask them to kind of validate. (Company A)

It's really tricky because there's so many stakeholders involved. So, when you're thinking about who actually are we influencing and how...where do we even start with this? Because there's inputs for pupils, inputs for parents, for teachers, you know, everybody involved. (Company B)

Discussion and Conclusion

The research questions guiding this study were as follows:

RQ1: How was the logic model used as a boundary object by companies and researchers within the EDUCATE programme?

RQ2: How effective was the logic model as a boundary object to support companies in their development of a theory of change?

In this section, we discuss each one in turn before concluding the article with some indications of future research directions in relation to this field of study.

The Role of the Logic Model as a Boundary Object

The data presented in the findings section demonstrate the different roles that the logic model played to help companies and researchers communicate across organisational boundaries during the EDUCATE programme. Companies were able to use the logic models to communicate with others on their teams who were not able to participate in the EDUCATE programme, to discuss product development, how different audiences would experience the product, and outputs, outcomes and impacts that the product would or could provide. Companies also used the logic model to communicate with stakeholders outside of their companies, for example, when applying for grants or other funding. Most mentioned by interviewees was the use of the logic model as a communication tool between research mentors and the companies themselves. These conversations not only helped the companies develop a logic model that expressed their product’s theory of change, but also helped founders question some of the assumptions they had made about their product’s functionality, users and use.

Effectiveness of the Logic Model as a Boundary Object

As the findings indicated, companies’ views of the logic model’s effectiveness varied. It helped some companies narrow the focus of their product or service to one achievable set of activities, outputs, outcomes and impacts. For others, the logic model allowed them to target their product to one audience (teachers, learners, parents), rather than trying to “be everything to everyone,” as Company E said. One of the key benefits of the logic model, as expressed by interviewees, was providing companies with the opportunity to stop and think about what they were doing and why they were doing it. The frenetic pace of technology start-up companies does not often allow founders this kind of time to consider in detail the chain of events outlined in the logic model. If the product provides Resource A, users will complete Activity B, producing Output C, which leads to Outcome D. As Company C noted, “I think one of the biggest things [the logic model] did was stop us from making some really critical mistakes.”

However, the benefits of the logic model were not immediately apparent to companies participating in the EDUCATE programme. As noted earlier, the EDUCATE programme comprised many other activities competing for participants’ attention, such as training and mentoring on business and product development topics, that some companies found more relevant at that point in their journeys. Moreover, when the EDUCATE programme started, the logic model was at first seen as a step that led to the larger objective of creating a research proposal, and as such not every mentor or company gave it as much attention as they did the research proposal. When the EDUCATE programme progressed and the logic model became a programme output that was required for reporting to the programme funder, this change in status increased its value for both parties.

As expressed in the interviews, companies also found the terminology in the logic model confusing and were often unsure how to apply it to their product or service. As the findings indicated, companies often went through several revisions of the logic model before developing a version that reflected both their product and the intention of the logic model categories. The research training delivered to EDUCATE participants was revised four times over the course of the programme in an effort to simplify the teaching of the logic model and improve participants’ understanding of its application, by providing more time for directed tasks that enabled its component parts to be constructed.

However, as the mentorship relationship required working across organisational boundaries to negotiate shared meaning, the act of researchers and entrepreneurs collaborating to understand and apply the categories of the logic model helped companies examine their own product journey and the assumptions they were making more carefully. It was perhaps due to these challenges that several interviewees noted that elements of the logic model -- if not the logic model itself -- have been embedded in how their companies make decisions about their products well after their participation in the EDUCATE programme ended. Companies expressed that they use their learnings from the logic model creation and the ensuing conversations with their mentors to ask themselves questions about how and whether their product will work as intended.

From the researchers’ perspectives, working with so many companies in the programme on the repeated, focused task of logic model creation built expertise that bridged the academia-industry gap, which was at the heart of the programme’s original goals. This expertise was founded in weekly meeting of the mentoring team at which we shared collective knowledge and practice to learn not only from our experiences but those of colleagues. This expertise included:

  • Knowing how to use the logic model as a tool to challenge pre-conceived notions.

  • Appreciating and adapting the order in which the logic model could be approached.

  • Supporting the company to situate the product more authentically within the context for which it was being designed.

The iterative design of the EDUCATE programme served to support and strengthen the growth of this expertise as the research mentors were continuously challenge to work with all 252 companies.

The implications of our work relate to the design and implementation of incubator and accelerator programmes that aim to develop (educational) research capacities within educational technology companies. First, there is a practical need to design (and be prepared to adapt) boundary objects that enable the academia-industry boundary to be crossed in ways that add value for both communities. Second, through the process of adapting and iterating the design of these boundary objects during programme implementation, both communities are required to develop increased awareness of how these adaptations support greater understanding of what makes educational technologies most effective in particular contexts.

Further Research

This research has offered evidence of the promise of the logic model as a boundary object to help technology companies communicate with a variety of stakeholders as they develop their technology product or service. Future research with a larger and more diverse sample of companies would enable understanding of the conditions under which the logic model is more or less effective as a boundary object.

Furthermore, the EDUCATE programme used a paper template of a logic model to work with start-up companies creating cutting-edge educational technology for learners. The research training explaining the logic model was delivered live and in-person and was disconnected from the company’s work in creating their own logic models. Creating a digital tool to aid in logic model development that connected users to scaffolding throughout the process and allowed online collaboration with mentors might be beneficial for educational technology developers.