Outreach Project Background
The aim of the Outreach Project was to increase the number of young people choosing to study physics, and physics-related degrees (including those in engineering and technology), with particular focus on increasing diversity through greater participation by females and people from lower socio-economic backgrounds. Previous efforts to achieve this aim have been focussed on secondary school pupils and have met with limited success. The project therefore chose to work with children and young people in primary schools, as well as secondary schools. The Outreach Project worked with around 30 partner schools covering the age range from 2 years to 19 years old and provided ongoing interactions with children and young people, as well as their teachers and families, initially for three years. The project was a partnership of 10 organisations, including local authorities, visitor attractions, STEM organisations and the university where the outreach team was situated. The majority of the schools engaged in the project were in areas of deprivation.Footnote 3
The broad age range of children and young people involved in the Outreach Project means that evaluating the impact of the project in relation to its stated aim is not possible for all participants because after three years, children in primary school will not be at a point of career decision making. Dyson and Todd (2009, p. 124) note that ToC evaluations “rely on predicting what outcomes might emerge as much as identifying outcomes that are already apparent. Outcomes in ToC evaluations are conceptualized as materializing at the end point of a change of intermediate changes which the evaluation seeks to track.”
This feature means that a ToC approach is particularly suitable for evaluating the outcomes of interventions in complex contexts, such as education, or in situations where the outcomes emerge after the completion of the intervention (Dyson and Todd 2009) as in the case for the Outreach Project.
Theory of Change
ToC approaches were initially developed in the US as a way of evaluating complex community initiatives (Sullivan and Stewart 2006) but have been used in the UK as a way to evaluate policy initiatives, such as Full Service Extended Schools (Dyson and Todd 2009).
Developing a ToC involves “a systematic and cumulative study of the links between activities, outcomes and context” (Connell and Kubisch 1998, p. 16) and provides “an overarching framework for understanding, systematically testing and refining the assumed connections (i.e., the theory) between an intervention and the anticipated impacts” (HM Treasury 2011, p. 57). They can be particularly useful in the evaluation of complex interventions where it is difficult to identify or track the endpoint outcomes of the intervention (Connell and Kubisch 1998; Dyson and Todd 2009). Once the first step of identifying the final impact or change that the programme or intervention is intended to bring about is completed, a process of backward mapping is undertaken and intermediate outcomes that are required to achieve this goal are articulated. This mapping process helps to surface the explicit or implicit theories that are held by those involved in developing the intervention. Intermediate outcomes may be short-, medium- or long-term. Together, the outcomes will create a causal pathway which supports the final goal of the programme (Taplin et al. 2013). A ToC is both a process and a product (Vogel 2012); therefore, an iterative approach can be helpful.
Process Used to Develop the ToC
The Outreach Project was a multi-year intervention, with an intended long-term evaluation of children’s qualification choices planned using the National Pupil DatabaseFootnote 4 which will take place ten years after the start of the project in primary schools. However, evaluation of the project on a short-to-medium-term timescale was also required. Rather than relying on short-term evaluation of individual activities, a ToC approach was chosen to allow the evaluation to be clearly linked to the project’s long-term aim through a chain of intermediate outcomes which are more amenable to tracking and evaluation (Dyson and Todd 2009).
At the beginning of the Outreach Project, a simple model was produced outlining the journey of a child through different activities, ages and stages, along with complementary activities for key influencers (Fig. 1). This was a first step in development of a ToC and enabled the development of a narrative understanding of the expanse of the STEM ecosystem in which children make decisions about careers. However, it did not allow for a clear elucidation of the behaviour changes and linked subjective norms required to lead to an increase in young people choosing a STEM career. Creating and using a detailed ToC allowed the development of a layered series of outcomes encompassing short-, medium- and long-term time-scales and guided the level and nature of evaluation of the intermediate stages.
The ToC was developed through an iterative series of workshops with members of the Outreach Project team and other academic staff within the university. Backward mapping was used to clarify the steps required to attain the overall aim of the project: increasing the number and diversity of students choosing a career in STEM post-18 (see bottom of Fig. 2).
The backward mapping process began with a workshop which involved the core delivery team of the outreach project and two academic staff from the university. Firstly, key stakeholders with an interest in the project aim were identified: these included children and young people, teachers and schools, parents and families, companies in different STEM sectors (both locally and nationally), further and higher education institutions, and government. However, the project team realised that aiming to work directly with all of these stakeholders was unrealistic. It was therefore important to narrow the range of stakeholders targeted. The choice of key stakeholders is supported by the Theory of Planned Behaviour (Ajzen 1985) and the importance of subjective norms on the intention to try a particular behaviour. Children and young people are the ones whose behaviour we are aiming to change, but parents and teachers are the referents (significant others) that strongly influence the subjective norms, and whose views children and young people are (usually) motivated to comply with. The more a child believes that parents and teachers think they should exhibit a behaviour, then the stronger the subjective norm towards that behaviour will be. Therefore, it was important that the key stakeholders described by the ToC would allow influence and change to be effected on those subjective norms. For this reason, in addition to children and young people, the project team identified teachers and parents/carers as the stakeholders that were key to the impact of the project in the North East. This narrowing of stakeholder focus also allowed the team to identify the level of evidence at which impact could be measured (Kazimirski and Pritchard 2014).
Having identified the key stakeholders, the group then started with the aim and worked backwards in time to consider what attitudinal, behavioural and structural changes for those stakeholders would need to occur to achieve the aim—the backward mapping .
This was an iterative process. The group first worked individually to identify the interim changes required for each group of stakeholders, writing each potential change on a post-it note. The choice of changes was informed by previous examination of the research literature, and also professional expertise. This identification process resulted in a large number of possible changes written on post-it notes. Two members of the group then worked together to group the notes by theme for each stakeholder group and to categorise them into long-, medium- and short-term changes. The validity of the groupings and categorisation were then discussed by the group as a whole, and changes were made to the organisation until consensus was reached.
The discussions identified long-term outcomes for the project as shown in Fig. 2.
For children and young people, two long-term outcomes were identified: “Increased confidence in ability to study STEM post-16”Footnote 5 and “Increased number choose to study A-level or vocational qualification in STEM subjects”. These two outcomes link to the changes in behaviour required at age 16 to enable young people to progress into STEM careers at age 18 (project aim). Improving a young person’s attitude towards success through increased confidence will increase the likelihood of them choosing to exhibit the desired behaviour (Ajzen 1985).
For schools, the long-term outcome “School environment mitigates effects of bias and stereotypes” was identified as important, and for parents and carers, the long-term outcome identified was “Parents and carers support and encourage STEM career choices for their children”.
To achieve these longer term aims, other medium-term changes were required. These changes were categorised to create a number of medium-term outcomes for each stakeholder group which were those that were expected to develop as a consequence of repeated interactions or which were time-critical to the school year or maturity of the children and young people.
A final round of mapping took place to identify short-term outcomes which fed into the medium-term outcomes, and the overall ToC diagram was created (Fig. 2). There are often many-to-many relationships between the different outcomes. This is due to the complex nature of the system within which the project is situated. The project team used the categorisation of short-, medium- and long-term outcomes to identify causal chains which linked the short-term outcomes with the long-term outcomes. Figure 3 shows an example of one such causal chain taken from the ToC.
Having created a first draft of the ToC, the outreach team then audited a number of the interventions that had already been developed and delivered in schools against the draft ToC. This audit was focussed on the following two questions: (a) does the ToC accommodate the intervention and (b) does the ToC have something to say about the value of that intervention. The audit identified a number of causal links that had not been included in the initial mapping (for example, including an explicit link between medium-term outcomes for the family and child stakeholder groups), which were then added. The audit also caused the outreach team to change the focus of some of the interventions, with the activities becoming more explicitly careers-centred as a result.
After the draft was produced, each outcome was cross-referenced to relevant research literature to ensure that the ToC was supported by prior research (see the Supplementary Material S1 for an overview of supporting research literature).
To further increase the confidence in, and trustworthiness of, the draft ToC, it was shared and discussed with the advisory and management bodies of the Outreach project. These groups included representatives from formal and informal education, industrial and charitable groups. The comments from these discussions were then used to finesse the ToC.
Developing Interventions Using the Short- and Medium-Term Outcomes
A number of case studies are presented in Table 1 which illustrate interventions that have been developed using the ToC. These case studies were chosen to represent the breadth of academic STEM subjects included in the Outreach Project overall and provide examples of collaboration between the outreach team and other research-active academics.
The ToC applies to all parts of a child’s educational journey, but the nature of the interventions developed to meet the outcomes change as the child gets older. For children aged between 2 and 5 years old, workshops focus on encouraging children to ask scientific questions and provide opportunities for role play based around different employment sectors. Between the ages of 6 and 11 years, children are introduced to a range of different STEM careers through exploratory workshops with titles, such as the Solar Physicist (solar system and light) and the Mechanical Engineer (gears and simple mechanisms).
At secondary school (age 11–16 years), support is focussed on careers ideas and information through workshops and assemblies, and on sustaining young people’s identification as a “STEM person”, together with the provision of careers-linked subject resources for use in the classroom. Once a young person has chosen to study physics and maths at 15–16 years old, the focus moves to activities aimed at supporting both attainment and self-concept as a “STEM person”, such as research experience weeks, networking events and after-school lecture series. These are designed to reduce the drop-off in STEM aspirations often seen at this age.
To support the short-term outcomes for teachers, CPD sessions about science topics, careers support and unconscious bias are provided to schools. In addition, science coordinators in primary schools are invited to attend a Primary Science Coordinators Forum six times a year where they are supported to strengthen the teaching of science and STEM in their schools.
Families are engaged through after-school workshops, holiday pop-up STEM shops and online materials. These activities challenge the gendered expectations of different careers, through careful activity design and delivery which removed gendered language, images and role models, and normalise scientific enquiry and science conversations for all participants: adults and children.