In this paper, we apply the outcomes model to empirical data collected from ethnographic research conducted over the course of one academic year at four ISL settings in England, UK, as part of the wider Youth Equity + STEM (YESTEM) project on youth equity and informal STEM participation, funded by the Wellcome Trust, National Science Foundation and the Economic and Social Research Council. Two of the settings were based in the capital, London and two were located in Bristol, a city in the South West of England. The settings were selected as illustrative of a number of different ISL offers, including designed spaces (a community zoo and a science centre) and community spaces (a digital arts centre and a social enterprise working to support young women in STEM). We liaised with practitioners to identify a focal youth programme within each setting, namely a holiday programme at the zoo, a girls’ after-school STEM club, an after-school coding club at the digital arts centre and a youth engagement programme at the science centre. Together, these programmes offered us opportunities to consider youth outcomes across a range of activities, modes of engagement, ISL settings and young people.
Data collection: youth ethnographies, observation and interviews
Multimodal youth ethnography work was conducted with 33 young people aged 11–14 who participated in programmes at the four partner ISE settings. Some were young people who were already naturalistically participating and others were recruited from under-represented communities to participate in programmes as part of the project. At three ISL settings, young people participated in distinct programmes and were recruited as part of this project: a weeklong programme at the community zoo, weekly school-based STEM club with two day trips as part of a girls-only STEM club, and bi-weekly school-based sessions with termly trips to the science centre. At the fourth ISE setting, young people were long-term participants, who had attended a range of different technology and media-focused programmes for between one and four years at the time of joining our study. Table 2 shows a summary of programmes and participants involved. Details of individual participants are presented in the Appendix. In addition, a young person’s self-identified ethnicity, gender and social class are given in the text the first time a young person is referred to in the paper. All young people chose their own pseudonyms.
Data collected included researchers’ observations of programmes; youth-constructed portfolio data (e.g. videos, photographs, written pieces of work, online posts, drawn pieces), audio-recorded interviews and discussion groups with young people and practitioners (at the start, during and after participation). In addition to the data collected during and immediately after the programme, we also met with young people c. six months after the end of each programme to ask more about the outcomes of their ISL participation. In the latter sessions, we used a set of outcome cards as group discussion prompts, based on the areas of the synthesised model, in which each card detailed potential outcomes relating to different outcome areas. For example, cards relating to the STEM trajectories included statements such as ‘The programme has made me want to do more STEM in the future’. Cards relating to STEM identity and identity work included statements such as ‘The programme has made no difference to how science-y I feel’. We worked with young people in small discussion groups of around three people, and asked them to look through the cards and discuss the extent to which they agreed or disagreed and their reasons, exploring differences and commonalities in view across the group. We also conducted individual follow up interviews, exploring participants’ experiences in greater depth.
Analysis of the qualitative data followed several steps. First, following data anonymisation, a theory-led interrogation was undertaken. We constructed a ‘case’ for each young person, combining all the available data (e.g. artefacts, researcher observations, photographic evidence, practitioner accounts and all the different data produced by young people—including interviews, written and artistic products, and online data from their portfolios). As discussed further below, in a small number of instances, where there were ‘conflicts’ between the data, where applicable, primacy was given to a young person’s account and all examples of outcomes were recorded (e.g. a young person’s claim of a positive outcome on one occasion would still stand even if they claimed on a different occasion not to have experienced any positive outcomes sources). We then undertook deductive coding, using the synthesised model outcome areas identified in Table 1. Each youth case was coded for potential outcomes, which were identified and coded by two of the research team using the NVivo software package. Emerging codes were consolidated and refined through successive waves of coding and analysis, with researchers comparing coding and discussing discrepant cases in order to reach a shared understanding. Where codes were not easily resolved, a third member of the research team read the cases in question and a shared position was reached through discussion. This produced a set of data for each participant that was coded in relation to the five outcome areas in the model. For instance, data extracts that had been coded as examples of grounded fun characteristically included accounts of youth deriving pleasure, enjoyment and entertainment from participation and instances of young people laughing, smiling, examples of joyful playfulness and a sense of ‘buzz’ in a session. They also tended to involve some degree of ‘interest’ and intellectualised engagement and/or connection between the experience/content and the self within and alongside instances of enjoyment/entertainment.
We produced a set of 165 tables—five for each 33 youth participants, summarising data pertaining to five overarching outcome areas (Grounded fun, STEM capital, etc.), which we read across for each outcome area in turn (e.g. reading across all grounded fun tables), to arrive at an interpretation about how these outcomes were being achieved, or not, within and across the different programmes and participants. From these cross-case readings, we produced a memo, detailing key themes, for each of the outcome areas, which form the basis of the discussion of findings. Our data and our conceptualisation of outcomes mean that these cannot be ‘counted’ in any straightforward way—for instance, just counting the presence of an outcome can obscure the scale and nature of it, suggesting an equivalence between, for instance, a momentary, ‘slight’ outcome and a longer-term, more consequential outcome. However, to give some provisional sense of the ‘shape’ of our coding across the areas, we note here the number of coded data extracts that we found in relation to each area: Grounded fun (n = 31), STEM capital (n = 59), Science identity/work (n = 28), STEM trajectories (n = 30) and Agency+ (n = 28).
Coding challenges: The complexity and inconsistency of outcomes. We found the process of analysing data on potential youth ‘outcomes’ underlined how outcomes were rarely coherent or discrete in the data. For instance, different data sources sometimes pointed to different outcomes for a young person. While there were often overlaps and similarities across the data relating to either a particular youth and/or a particular setting, there were also multiple instances where practitioners, youth and researchers all identified different outcomes for a given youth at a particular time in a particular setting.
Take, for instance, Ginger (white, working-class boy, Digital Arts Centre). Ginger was regularly recognised as a coding expert both the practitioners and fellow participants during the club sessions. This recognition stood in stark contrast to his struggles for recognition at school. We recorded numerous examples in the observation and practitioner data that suggested Ginger experienced increased confidence and recognition as a result of his participation. For example, when asked what she thought Ginger gets out of coming to the weekly sessions, the practitioner leading the programme reflected 'it’s confidence, more than anything', recounting various examples of changes in his behaviours and interactions that she had seen over time. On a number of occasions, Ginger also concurred and talked about the skills, recognition and confidence that he had derived from participation. However, during his final follow-up interview, Ginger told us he was 'tired' and appeared to be in a more downbeat mood than when we had previously observed him. In this interview he voiced quite different views, saying that participation in the club had made 'no difference' to him, his confidence and his identification with STEM. He insisted instead that “no, it’s mainly school that made this happen”, attributing his coding interest and expertise to his first experiences of coding at school. Similarly, while Ginger had regularly attended club sessions that introduced new technical skills and content, in this particular interview he was adamant that he had not learned anything new from the programme ('Not really. I feel like I learn more from Mr H’s old YouTube channel').
So, how to account for these inconsistencies in the data? One potential interpretation is that Ginger did not like interviews and found it hard to represent himself verbally, so perhaps we should give less ‘weight’ to his verbal data compared to other sorts of data that we collected. Indeed, he had previously told us that he did not like sharing his views verbally and much preferred using digital technologies to express himself ('it’s quite hard for me to answer these questions […] when I type it I’m just so much better at it')—a point that we expand on elsewhere in relation to the importance of materiality in Ginger’s constructions of STEM identity (Godec et al., 2020).
From our conceptual position, we trust the voices and accounts of under-served young people (Mohanty, 2003). In doing so, we also recognise that there is no singular, consistent ‘truth’ of their views and experiences (in this case regarding what outcomes young people feel they derive from ISL). Rather, we respect that they will experience and voice multiple truths, each of which will be true to the moment and context in which they are expressed by a young person. Hence, we interpreted Ginger as telling us that there were numerous moments in which he felt that he derived valuable outcomes from his ISL participation, but there were also moments when he did not. On balance, his data tells us that he felt that he derived equitable outcomes from participation, but this was not absolute—there were also moments when he felt he had not derived value from the experience.
This pattern was also found across the other young people. For instance, some young people were much more positive about their experiences during the follow-up interviews than they had recounted during the actual sessions. For instance, 007 (white, middle-class boy, Science Centre) repeatedly shared his frustrations about the science centre programme with us during interviews and observations conducted during the sessions ('I feel like although we’ve done stuff, but we haven’t really done any stuff'; 'I found it a bit repetitive cause I think we’ve done it now three times now'; 'I’d like to do something more … bigger'). Yet, when we met him several months later, 007 enthusiastically identified a range of positive outcomes that he felt he had derived from his participation. For instance, in a card sort activity, he strongly agreed that he enjoyed the programme, learnt new things and gained new confidence through taking part.
In this way, we seek to embrace the complexity of young people’s experiences, recognising that they can and will often express contradictory views about the benefits they derive (or not) from their experiences. These accounts help us see how youth may articulate and experience different outcomes from their participation at different time points, and may feel differently about these outcomes in the moment during sessions compared to later follow-up interviews conducted several months afterwards. For instance, Emerald told immediately after the programme she found computing 'more interesting than before'. Yet when we caught up with her months later, she remarked that the programme did not make much difference, because 'I was always interested in it'. We suggest that both points need to be recognised and valued.
Others, like Ginger, above, were more negative in the catch-up interviews compared to previous data collection points. Young people who were interviewed at the end of an all-day session also tended to provide less positive accounts. In each case, we suggest that multiple interactions of personal, contextual, social and institutional factors will have shaped the accounts that they gave—there is no one ‘truth’, rather each account has a truth in the moment of its social construction.
These examples also draw attention to how young people’s ISL outcomes and experiences—and indeed their role in the research process itself—are not separate from their wider lives. That is, ISL outcomes, experiences and the research process are all situated within and mediated by wider identities and inequalities. There are many reasons why Ginger may have been tired on the day of his final interview, but it felt significant enough to him to voice it. Looking across his data, we interpret his feelings of tiredness as potentially mediating the extent to which he can derive equitable outcomes from his ISL participation. This leads us to consider how understanding and respecting how youth feel in the moment (and the extent to which these feelings may reflect or be exacerbated by wider relations and experiences of inequity in youths’ lives), and the extent to which ISL can enact relations of care in this respect, may be as, if not more, important for supporting equitable youth outcomes as the STEM content of a programme.
In the same way that we seek to privilege the views and experiences of young people from under-served communities, while recognising that these will often not be simple or consistent, we also wish to accord value and respect to the views of practitioners, noting that these will likewise reflect their own situated truths which may vary with time and context and which may, or may not, align with the young people’s accounts.
A further layer of complexity is added when we consider that each of the programmes also contained different component elements, again complicating the notion that there might be a generic set of outcomes that might be derived ‘overall’ from ISL participation. For instance, the girls STEM club included a regular after school STEM club, an industry visit and a weekend coding event. Young people, practitioners and researchers all associated different aspects of the programme (and indeed, different weekly club sessions) with different outcomes for different young people. For instance, Innocent (Black working-class girl) described some parts as ‘fun’ and ‘engaging’ but found other parts 'boring'. For instance, she described the day trip as 'pretty interesting but half of it, we already knew it. So, I had to sit there whilst they repeated it', whereas she described a club session that covered codes and maths games as 'all boring …the numbers, we already do like in Maths, so what's the point?'.
Matters were further complicated when young people did not always differentiate between the research component of the study (during which they worked with the research team to make their reflective portfolios and co-research their lives and ISE experiences, both on the programme in question and more broadly) and the STEM programme. For instance, Tori attributed a range of identity outcomes to the STEM club programme, but our analysis suggested that the experiences she described as producing these outcomes actually took place during research portfolio sessions, which were often held before or after club sessions.
This complexity, across and between data sources, participants and aspects of each programme, meant that there were no simple outcomes that could be identified from the young people’s ISL participation that could be straightforwardly and reliably measured. As explained, we see these variations as normal and to be expected—identities, experiences and outcomes are socially constructed and mediated phenomena that are generated through interactions between multiple actors across time and space. In this respect, we wish to strongly bracket the outcomes findings that we discuss next, noting that these are never neat or definitive, but are meaningful in their complexity. We have also intentionally foregrounded young people’s accounts of their outcomes, as opposed to researcher or practitioner interpretations, although at some points the latter are brought in where they seem to offer an additional dimension or potential interpretation for a point made by the young people.