Introduction

STEM has been touted as the pathway to enable school students to become engaged in Science, Technology, Engineering, and Mathematics (STEM) and fill the projected needs of the twenty-first century. Discovering where STEM is being taught as a coherent entity rather than in the siloed subjects is challenging; however, one area where this has occurred is in the context of international educational robotics competitions, such as the FIRST® LEGO® League: Challenge. According to the research literature, educational robotics learning experiences and competitions provide students with authentic, challenging opportunities to learn and apply new STEM knowledge and twenty-first century skills through their collaborative development of solutions to complex, real-world problems, incorporating the use of digital or robotics technologies (Anwar et al, 2019; Brancalião et al., 2022; Eguchi, 2017; Kandlhofer & Steinbauer, 2016; Menekse et al., 2017). Informed by constructivist and constructionist learning theories, educational robotics competitions aim to promote students’ STEM interests and consideration of STEM career pathways; and support their development of twenty-first century problem-solving, creative thinking, communication, and collaboration skills (Anwar et al., 2019; Brancalião et al., 2022; Evripidou et al., 2020). The quality of educational robotics competitions may be measured through the learning outcomes they produce, and how they assess the development and demonstration of these outcomes (Anwar et al., 2019; Evripidou et al., 2020).

The FIRST® LEGO® League (FLL) Challenge was founded in the USA by the not-for-profit organisation For the Inspiration and Recognition of Science and Technology (FIRST) in partnership with the LEGO Group (LEGO Group & FIRST, 2013). Since its inception in 1998, the FLL Challenge has become a popular program to promote students’ interests in STEM subjects and careers, and their development of twenty-first century or transversal skills (Anwar et al., 2019; Eguchi, 2017; LEGO Group & FIRST, 2013). The FLL Challenge is part of a family of FIRST robotics programs for K-12 students and is one of several international STEM competitions based on the LEGO® Mindstorms™ robotics platform (LEGO Group, 2022). In 2020, the FLL Challenge involved over 318,000 students, working in 38,600 teams in approximately 110 countries, making it one of the world’s largest educational robotics competitions (FIRST, 2020; Menekse et al., 2017).

This article explores the themes in the research literature published between 2004 and 2022 regarding the FLL Challenge’s impact on participating 9–16-year-old students’ STEM attitudes and learning, and their development of twenty-first century skills. These may be strong indicators of participants’ engagement in future post-secondary STEM education and career pathways (Burack et al., 2019). Studies of STEM attitudes and learning were further refined through emerging focusses on underrepresented groups, including female participants, ethnic minority groups, and students with special needs. The influence of coaches’ skills and pedagogical expertise upon students’ learning and skill development is a potentially underrepresented area of research.

About the FLL Challenge

The FLL Challenge is typically run as an extracurricular program in classrooms, school or community makerspaces, and afterschool clubs (Melchior et al., 2018). Over the course of a typically 8–12-week season, commencing in August each year, participating teams of 2–10 students work under the guidance of 1–2 coaches (often a teacher, parent, or community volunteer) to explore real-world problems relating to an authentic STEM theme, such as human space exploration (2018/19 INTO ORBIT), urban design (2019/20 CITY SHAPER), physical activity (2020/21 RePLAY), and cargo transportation and logistics (2021/22 CARGO CONNECT). During the season, students work through activities relating to the FLL Robot Game, Innovation Project, and Core Values.

In The Robot Game, students employ real-world engineering practices to design, construct, program, and test LEGO® Mindstorms™ EV3 and Spike Prime™ robots to complete missions (Menekse et al., 2017). These missions typically require students to solve programming and engineering design challenges to enable their robot to consistently navigate obstacles and physically interact with LEGO® models on a 2.3 m × 1.1 m game field (Ma & Williams, 2013). In the FLL Innovation Project, students are tasked with researching, prototyping, and pitching an innovative solution to a real-world problem relating to the competition’s theme (Chen, 2018; FIRST, 2021). As part of their project design process, teams are required to engage with professionals, experts, and people affected by the problem, and seek their feedback upon their proposed design solution (FIRST, 2021).

Students, coaches, parents, and volunteers competing in the FLL Challenge are expected to uphold and celebrate the FIRST philosophies of ‘gracious professionalism®’ and ‘coopertition®’. Gracious professionalism® is defined as a ‘way of doing things that encourages high-quality work, emphasizes the value of others, and respects individuals and the community’ (FIRST, 2022c, para. 8). Coopertition® emphasises the importance of teams demonstrating ‘unqualified kindness and respect in the face of fierce competition’, encouraging cooperation and shared learning within and between competing teams (FIRST, 2022c, para. 9). These philosophies are expressed through the FIRST Core Values of discovery, innovation, impact, inclusion, teamwork, and fun (FIRST, 2022c); and teams’ demonstration of these values is an essential requirement for advancement to state, national, and international tournaments (Dwivedi & Dwivedi, 2017).

At the end of the season, teams attend a culminating tournament, described as high-energy ‘sporting type’ events where teams demonstrate their robots in a series of three competitive 2.5-min Robot Game matches, and present their robots and innovation projects to a panel of volunteer judges (LEGO Group & FIRST, 2013). Team performances are assessed using expert-validated official rubrics which emphasise the assessment of twenty-first century teamwork/collaboration, communication, and problem-solving skills (Usart et al., 2019). While it is a competition, the FLL Challenge strives to promote a culture of learning, teamwork, and fun by placing equal assessment weighting on robot game performance scores, robot design, core values, and the innovation project in tournament rankings and awards (FIRST, 2019).

Purpose of This Paper

This paper is important as it provides a significant semi-systematic historical review into FLL research studies. This is required as recent literature reviews of educational robotics competitions as seen below, whilst interesting, had some limitations that are subsequently addressed in this paper. The first paper, Brancalião et al. (2022), conducted a systematic review of more than 50 mobile robotics competitions, identifying their objectives, target audience, technological platform, and area of application. The review found that FIRST competitions, including the FLL Challenge, were among the oldest, most famous, and frequently cited educational robotics competitions targeted at primary and secondary school students; however, it did not examine program impacts or learning outcomes as this was planned for a subsequent paper (Brancalião et al., 2022).

The second review, Zuhrie et al. (2021), examined the learning and skill development outcomes of 11 national and international educational robotics competitions based on a range of technological platforms, targeted at primary, secondary, and university students in Indonesia. They found that educational robotics competitions provided opportunities for students to develop practical skills and master theoretical knowledge through hands-on learning experiences; and that project-based and problem-based learning pedagogies were widely used across the competitions under review (Zuhrie et al., 2021). This review had several limitations. It did not include inclusion/exclusion criteria for the selection of the competitions and studies, and it only included a single study for each competition. The article did not examine studies relating to several prominent international competitions with larger participation numbers, such as the FLL Challenge, World Robot Olympiad®, or VEX Robotics. This may be due to these competitions having a relatively small presence in the authors’ Indonesian context.

The third review, by Evripidou et al. (2020), focussed on the expected learning outcomes of K-12 students’ engagement with educational robotics. Based on a meta-analysis of the research literature and creation of a bibliographic map using study keywords, the authors proposed a learning outcome framework, and mapped this against the goals, design, and rules of 40 international and regional (e.g. country/continent based) educational robotics competitions, including the FLL Challenge. Based on their analysis, Evripidou et al. (2020) posited that the FLL Challenge would have a moderate effect on students’ problem-solving skills and creativity, and strong impacts on students’ self-efficacy, computational thinking, motivation, and collaboration. The authors stressed that their mapping of proposed outcomes was based on their analysis of the competition design, not the findings or conclusions of their studies included in the scope of their review.

This current semi-systematic review is part of a wider, ongoing qualitative study examining students’ and coaches’ lived experience of the FLL Challenge. It seeks to build upon the work of Evripidou et al. (2020) by conducting a historical review of empirical research findings relating to the impact and learning outcomes of an established and long-running international educational robotics competition designed for primary and secondary students. This review seeks to inform future research examining the design, implementation, and assessment of students’ learning and skill development in the context of robotics competitions, with the view to better supporting educators and coaches working to support their students’ engagement in these complex, multidisciplinary learning experiences.

Aim

To review the research literature published between 2004 and 2021 focussed on the learning outcomes and impact of the FLL Challenge on participating students aged 9–16.

Research Questions

  1. 1)

    What research methodologies were used to measure impacts of student participation in the FLL Challenge?

  2. 2)

    What was the geographical distribution of FLL Challenge studies?

  3. 3)

    What were the impacts of the FLL Challenge on students’ STEM attitudes, learning outcomes, and twenty-first century skill development?

  4. 4)

    What research gaps were identified in the literature on the FLL Challenge?

Methodology

This study adopts a semi-systematic historical literature review methodology, defined as the examination and synthesis of qualitative and quantitative research literature published about a topic or phenomena, starting with its emergence, and tracing its development and evolution over time (Snyder, 2019). The use of a semi-systematic approach allows for the examination of the topic from different disciplinary perspectives and methodologies, resulting in the creation of a historical timeline of research themes to identify gaps in the literature and likely directions for future research (Snyder, 2019). In conducting this review, the authors followed a pre-defined, replicable search strategy based on the inclusion and exclusion criteria identified in Table 1.

Table 1 Inclusion and exclusion criteria
Full size table

The literature in this review was found through searches of the ACM Digital Library, Engineering Village, IEEE Explore, ERIC, ProQuest, Web of Science, and Wiley databases. Searches were restricted to English-language publications, which may have influenced the geographical context of the included studies. A combination of keywords was used in conjunction with Boolean operators, including ‘FIRST LEGO League’, ‘FLL’, ‘robotics’, ‘21st Century skills’, ‘impact’, ‘learning’, and ‘students’. The initial search identified a total of 35 unique publications, which were read and reviewed for relevance to the research aim and questions. Their reference lists were used to identify frequently cited historical literature, such as the evaluation studies published by Melchior et al., (2004, 2009) and the Center for Youth and Communities (2013) and the longitudinal study reports published on the FIRST Impact website (FIRST, 2022a). The evaluation studies were commissioned by the FIRST USA not-for-profit organisation, and while they were not peer-reviewed, they are included in this literature review as they were heavily cited by subsequent peer-reviewed studies from 2011 onwards.

There are multiple publications relating to the ongoing FIRST Longitudinal Study (2013–present). Annual impact reports published between 2013 and 18 were excluded, as their findings were summarised in peer-reviewed publications (Burack et al., 2019; Melchior et al., 2018). The 2019 and 2020 reports were also excluded, as their findings were broadly similar to the most recent report by Meschede et al. (2022). A small number of studies which described the FLL competition, but did not examine student learning outcomes, were excluded. These included two theses, one study of spatial ability, three personal narratives written by FLL coaches and students, and three studies focussed on the impact of university engineering outreach programs upon participating university/college student mentors or coaches. A total of 26 publications, 4 reports, 9 peer-reviewed conference papers, and 13 peer-reviewed journal articles, were included in this review.

After the articles were selected, a chronological matrix was created to identify the authors, year of publication, methodology, sample, geographic context, key findings, and conclusions of each article. This matrix was used to identify and trace the development of broad historical themes in the FLL literature over time. The primary analysis was conducted by the first author, and the themes were extensively discussed and refined by all authors. These are summarised in Fig. 2 in the “Findings” section below. As noted in the Appendix (Table 2), some studies inform multiple research themes.

Findings

Research Methodologies Used to Assess FLL Impacts

As elaborated in the Appendix (Table 2), the 26 research publications in the FLL literature included in this review used a variety of methodologies, with the majority (22) using post-season surveys, typically conducted at championship level tournaments. Seven studies included pre- and post-season surveys for comparative analysis, although only the longitudinal study mentioned accounting for baseline differences. Thirteen studies incorporated interviews—most commonly with students, but also with parents and coaches. Only 5 studies included field observations of FLL teams during their competition season, and several of these involved participant researchers coaching the students in the study. There are 13 mixed method papers, 7 qualitative, and 5 quantitative papers explored here.

Geographical Distribution of FLL Studies

As shown in Fig. 1, FLL research has been conducted in 11 countries; however, 57% of the reviewed publications focus on the experience of students in the USA and Canada. This geographic bias may be influenced by the exclusion of non-English language studies in this review. It is also possible that while the FLL Challenge runs in approximately 110 countries, it may not be as popular compared to other national or international robotics competitions in countries outside of North America.

Fig. 1
figure 1

Number of FLL studies by geographic region

Full size image

Impact of FLL Participation

This literature review identified three broad themes pertaining to the impact of the FLL Challenge on participating students’ STEM attitudes, learning, and twenty-first century skill development. The first theme was further refined through three subcategories focussed on the experience of underrepresented student participant groups. These themes are outlined in Fig. 2.

Fig. 2
figure 2

Historical themes in FLL research

Full size image

Theme 1.1: Impacts on Students’ Motivation, STEM Learning, and Attitudes (2004–2019)

The first theme relates to the research evaluation of the impact of the FLL competition on participating students’ motivation, STEM learning, and attitudes. This theme emerged with three mixed-methods program evaluation studies by Melchior et al., (2004, 2009) and the Center for Youth and Communities (2013). The original study (2003–2004) surveyed 919 students (aged 9–14), 162 coaches, and 699 parents in the USA and Canada, supplemented by field observations and telephone interviews with 32 coaches (Melchior et al., 2004). The follow-up studies broadly replicated this methodology, with some changes to ensure the selection of a random, nationally representative (USA only) sample (Melchior et al., 2009). Despite the methodological and sampling differences, team members, parents, and coaches across all three studies reported that the FLL Challenge positively impacted upon students’ STEM interests, problem-solving, academic motivation, teamwork, and life skills (Center for Youth and Communities, 2013; Melchior et al., 2004, 2009). These studies were not peer-reviewed and did not include pre- and post- comparisons or statistical analysis of the program impacts.

Later peer-reviewed studies, with a range of sample sizes, participant ages, methodologies, and geographic contexts also report positive, although not always statistically significant, impacts on students’ motivation, STEM interests, and attitudes. Nugent et al. (2011) noted promising, although not significant, impacts on FLL students’ STEM attitudes in pre- and post-season multiple choice attitudinal surveys. Tai et al. (2015) found that Taiwanese high school students’ engagement in hands-on FLL robotics activities had a significant impact on their science attitudes and academic motivation, as measured in pre- and post- tests. Andic et al. (2015)’s survey of FIRST participants in Montenegro found that students valued their robotics learning experience and were motivated to pursue further robotics learning opportunities. Kaloti-Hallak et al. (2015b)’s mixed method study found no significant changes in students’ motivation and self-confidence pre- and post- competition; however, the authors argued that this was likely due to highly motivated and interested students choosing to participate in FLL. Finally, Arís and Orcos (2019) found that that Spanish FLL coaches perceived positive, statistically significant impacts on participating students’ STEM interests, motivation, and creativity. Interestingly, this study also found that students reported their enjoyment of the Innovation Project (22.2%) and Core Values (20.9%) activities, meaning that they were not solely motivated by the opportunity to build and program LEGO® robots (Arís & Orcos, 2019).

Research findings regarding the FLL competition’s impact on students’ learning of STEM technical skills are mixed. Using a validated pre- and post- multiple choice assessment instrument, Nugent et al. (2011) found that FLL participation significantly impacted on students’ learning of programming skills, but that the competition had non-significant impacts on their learning of engineering knowledge and the and engineering design process. The authors suggest that this result may be influenced by their data collection instrument design, or the size of teams restricting students’ opportunity to be involved in robot design activities. Interestingly, there was a significant positive correlation between students’ scores on the technical assessment and their team’s competition performance scores (Nugent et al., 2011).

A later qualitative study, by Kaloti-Hallak et al. (2019), found that while FLL teams demonstrated growth in their robotics engineering design, construction, testing, and debugging skills and processes, the quality and depth of their learning, assessed against Bloom’s Revised Taxonomy, varied dramatically. This study, which incorporated team observations and student interviews during the season, and videotaped observations of teams’ tournament robot design judging, found that FLL students’ learning of engineering design processes was influenced by their coaches’ pedagogical approach, the lack of robotics curriculum resources, extracurricular time pressures, and robot reliability (Kaloti-Hallak et al., 2019).

The FIRST Longitudinal Study tracking the long-term impact of three of the four robotics competitions organised by FIRST is included here as it examines impact and attitudes. A representative sample of 1273 FIRST participant students in the USA was recruited in 2012 and 2013 (including 206 involved in FLL). A comparison representative sample of 451 students involved in maths and science programs was recruited from the same schools and organisations as the FIRST participants (Melchior et al., 2018). Participants completed pre- and post- surveys on entry and were invited to complete annual follow-up surveys. Survey response rates were broadly consistent over the subsequent 8 years, and the most recent (2021) survey had a response rate of 74% (Meschede et al., 2022). These surveys were supplemented by focus group and telephone interviews in various years of the study (Burack et al., 2019). Controlled for baseline differences, the longitudinal study findings consistently report statistically significant long-term impacts of FIRST program participation on students’ STEM attitudes and interests, especially for female participants (Burack et al., 2019; Meschede et al., 2022). These impacts were greater for students who competed in multiple competitions, and hold true across all demographics, socioeconomic, and community contexts (Melchior et al., 2018; Meschede et al., 2022). FIRST alumni are statistically twice as likely to have higher-level STEM attitudes and interests than students in the comparison group, and these effects persist into their post-secondary education pathways (Meschede et al., 2022).

Many studies examining impacts on FLL students’ STEM learning, engagement, and skills share similar limitations. They typically rely on participating students, parents, and coaches self-reporting at the end of the competition season, when their enjoyment of the experience and team performances may influence their responses (Burack et al., 2019; Nugent et al., 2011). Additionally, these studies all involve students who either self-selected or were nominated for FLL by parents or teachers due to their prior interests in STEM or LEGO® (Center for Youth and Communities, 2013; Kaloti-Hallak et al., 2019; Melchior et al., 2009). Only three studies in this theme examined impacts pre- and post-competition learning experience, and none tracked changes during the season. Finally, as noted by Melchior et al. (2018), the longitudinal study has two important limitations, namely, the use of a comparison rather than random population sample and the existence of significant differences in the groups’ 2013 baseline attitudes, which had to be controlled for when conducting statistical analyses.

Theme 1.2: Impacts on Female Students (2004–2020)

A key factor in encouraging female participation in STEM and computer science careers may be the provision of informal educational opportunities for girls to develop their interests and self-confidence through authentic, hands-on programming experiences (Witherspoon et al., 2016). While there is strong longitudinal evidence that engagement in FIRST robotics competitions has a powerful long-term impact on girls’ STEM attitudes, learning, identity, and career choices, female students are historically and consistently underrepresented in the FLL Challenge in the USA (Burack et al., 2019; Center for Youth and Communities, 2013; Melchior et al., 2009; Melchior et al., 2018). Female students are proportionally underrepresented in most of the studies in this review, and unfortunately, there are no recent statistics from either the USA or internationally regarding female students’ representation in FLL.

Gender differences in FLL participants’ learning experience, motivation, and outcomes have been explored from various international and methodological perspectives. Melchior et al., (2004, 2009)’s early (non-peer-reviewed) studies found that while both male and female participants reported high levels of satisfaction and enjoyment of the FLL competition, male students were more likely to engage in technical programming and engineering activities, and female students were more likely to focus on the innovation project and creative activities which improved their collaboration and communication skills. These differences were less obvious in the Center for Youth and Communities (2013) evaluation study. While these findings were not subjected to statistical analysis, they have been broadly supported by later peer-reviewed studies in international contexts, as we will explore next.

In the UK, Ball et al. (2012) found that both boys and girls enjoyed using LEGO® Mindstorms™ and learned new computer science, teamwork, and problem-solving skills through FLL; however, it is important to note that their non-representative sample included 65% male participants. Kaloti-Hallak et al., (2015b, p. 110) observed more positive, although not statistically significant, impacts of FLL participation on female students’ STEM attitudes, motivation, and self-efficacy, which ‘increased to the point they felt they were better than male students’. The authors found that students’ attitudes and motivation were positively influenced by their peers, teachers, and parents, and noted that students entered the study with high levels and motivation, which remained high at the end of the season (Kaloti-Hallak et al., 2015b). This study included a disproportionately high number of female participants. More recently, Schina et al. (2020a, p. 322) conducted a post-season survey study with a relatively gender-balanced sample of Greek FLL students, finding that ‘based on statistically significant results’ female students seemed ‘to be more engaged, enthusiastic, creative, and … more likely to adopt collaborative strategies than male participants’.

Witherspoon et al. (2016) conducted a large-scale quantitative survey study examining how students’ gender, interests, motivation, and prior programming experience impacted on their opportunities to learn programming in five FIRST® and VEX robotics® competitions in the USA. Their sample included 155 FLL students, and while 33% of the overall sample (483 students) were female, data collection issues prevented the authors’ breakdown of gender participation by competition. Two-thirds of students had prior competition experience, and 55% reported prior programming experience. The study found that students’ active engagement in competition programming activities correlated with higher levels of motivation to pursue further programming experiences (Witherspoon et al., 2016).

A significant finding of Witherspoon et al. (2016)’s study was the identification of a striking and widening gap in the level of female elementary and high school students’ involvement in programming, particularly as they moved into more technically advanced competitions beyond FLL. Female FLL students were more likely than boys to be interested and involved in programming in elementary school; however, this trend reversed as they transitioned into other secondary school competitions (Witherspoon et al., 2016). This decline in female students’ programming interest and involvement may be influenced by societal expectations, and the more specialised and technical nature of team roles in more advanced competitions, which may ‘emphasise team success over equitable participation’ (Dwivedi et al., 2021; Witherspoon et al., 2016, p. 10). The authors suggest that coaches could help to address this decline through their use of inquiry-based pedagogical strategies and by explicitly recruiting girls into technical roles. This study had several limitations, including the lack of a control group, and the use of single items to measure key constructs (e.g., motivation and experience) on the survey instrument (Witherspoon et al., 2016).

Theme 1.3: Impacts on Ethnic Minority Students in the USA (2013–2018)

Encouraging minority (non-Caucasian) student representation in FLL competitions is a strategic priority of the FIRST organisation in the USA; and research focussing on these students’ participation, experience, and learning outcomes is US-specific (FIRST, 2022b). The term ‘minority’ reflects the terminology used in these studies. According to the Center for Youth and Communities (2013), minority representation in the US FLL Challenge increased to approximately 33% between 2004 and 2013. Rosen et al. (2013) conducted pre- and post-season surveys with 21 African American and Hispanic FLL students, observing small positive, but not statistically significant, changes in their STEM interests and career aspirations. The ongoing FIRST Longitudinal Study, which involves a larger sample of Asian, Hispanic, and African American students (approximately 32% of the baseline population), reported medium effect sizes on measures of both minority and non-minority students’ STEM interests, knowledge, and involvement in STEM activities, and small impacts on minority students’ STEM identity and interest in pursuing STEM careers (Melchior et al., 2018).

Theme 1.4: Impacts on Students with Special Needs (2017–2019)

Several recent studies have examined the experience and learning outcomes of students with special needs in FLL, albeit with small sample sizes and only in non-mainstream educational environments. Fisher et al. (2019) conducted exploratory qualitative research with a small sample of three male students with autism spectrum disorder (ASD) in a special needs school in the USA. The study found that with significant pedagogical and social support from parents and coaches, FLL participation had a positive impact on ASD students’ social skills, helping them grow in confidence, make friends, and experience being part of a team. As noted by the lead researcher, their participant-observer role and existing relationships with the students and parents may have influenced the findings. Additionally, the study involved a very small non-representative sample in a non-mainstream educational environment.

Lindsay and Hounsell (2017) and Lindsay (2019) conducted mixed methods studies piloting and implementing a heavily modified FLL robotics program in a paediatric children’s hospital in Canada. Youth with a range of physical and neurodivergent special needs were involved in an adapted FLL experience, with the support of technical and medical experts. Lindsay (2019) reported that participating students perceived improvements in their communication and collaboration skills, such as listening to others, making decisions, managing time, and teamwork. The children enjoyed the program and felt a sense of belonging. As noted by the author, the study had an unintentional gender imbalance and an overrepresentation of ASD students, who may have different STEM-related interests and experiences to students with other disabilities (Lindsay, 2019).

To date, these are the only studies examining the learning experience and outcomes of students with special needs in the FLL competition, and they did not include students included in mainstream educational environments. It is unclear whether this is a sampling issue—i.e. studies did not collect statistics about students’ special needs, or if there are other factors which may discourage the participation of students with special needs in school-based FLL teams.

Theme 2: Development of Twenty-First Century Skills (2011–2020)

Interest in the impact of the FLL Challenge on students’ teamwork, leadership, and project management skills dates to 2004; however, research aimed at identifying the development and assessment of specific twenty-first century skills in FLL emerged in 2011 and developed over time (Melchior et al., 2004, 2009; Nugent et al., 2011). Multiple studies, conducted in different international contexts with both qualitative and quantitative methodologies, have found positive perceived and statistically significant measured impacts on FLL students’ development of twenty-first century problem-solving, communication, and collaboration (teamwork) skills (Chalmers, 2013; Ma & Williams, 2013; Nugent et al., 2011; Schina et al., 2020b). Several quantitative studies with larger sample sizes found a correlation between students’ improved interpersonal skills and the effectiveness of their team performance both during the season, and at their culminating tournaments (Arís & Orcos, 2019; Menekse et al., 2017). Ma and Williams (2013) cautioned that children’s development and retention of 21st Century skills was dependent on how team coaches supported their students’ articulation and reflection upon their learning. This finding was partly supported by Chen (2018), who noted the importance of coaches’ guided questioning to support their students’ development of collaborative problem-solving skills. These two qualitative studies share similar limitations, particularly their use of small, non-representative population samples. Additionally, Chen (2018) relied on semi-structured interviews as the only data collection method.

A significant study in this theme was Menekse et al.’s (2017) study of the relationship between the quality of FLL teams’ collaboration skills and their tournament performance. The researchers analysed robot game performance scores, judging rubrics, and observed teamwork task assessments for 61 teams (366 students) at a 2015 state championship level event in the USA. They determined that established FLL teams with multiple years of competition experience, regardless of whether they included new team members, demonstrated higher level twenty-first century collaboration and communication skills compared to newer teams, enabling them to build superior robots, and achieve higher performance scores across all aspects of the competition. They recommended further research examination of how experienced teams may support their more inexperienced team members by passing down rules, collaboration norms, and written documentation or routines. A potential limitation of this study was its reliance on judging performance assessments conducted by volunteers working with different teams; however, the authors argue that the statistical significance of the effects was strong enough to reduce the possible impact of observer bias and varied judging quality (Dwivedi et al., 2021; Menekse et al., 2017).

While studies are broadly consistent in their findings regarding the FLL Challenge’s short-term positive perceived impacts on twenty-first century skills, there may be gaps between students’ perceived and actual skill development (Kaloti-Hallak et al., 2019; Schina et al., 2020a). For example, Kaloti-Hallak et al., (2019, p. 127) found that most FLL ‘students did not demonstrate sophisticated problem-solving strategies’ when applying engineering design processes. Time and workload pressures imposed by the design of the competition may encourage teams to prioritise the use of inefficient trial and error strategies to solve technical and research problems, limiting their opportunity to engage in higher-level peer discussions and collaborative problem solving (Dwivedi et al., 2021; Kaloti-Hallak et al., 2015a, 2019; Stewart & Jordan, 2017). This finding contrasts with students reporting their use of high-level problem-solving strategies in an earlier study by Nugent et al. (2011). As Schina et al. (2020a) suggest, there is a need for future qualitative research using observation to examine teams’ FLL learning process, and to compare their perceived and actual twenty-first century skill development during the season.

It is not entirely clear whether perceived twenty-first century skill impacts persist beyond the immediate competition experience, and whether students transfer and apply these skills in other contexts. The FIRST Longitudinal Study found statistically inconclusive evidence for long-term impacts of FIRST participation on students’ teamwork, problem-solving, and communication skills (Burack et al., 2019; Melchior et al., 2018). Survey responses, focus group interviews, and telephone interviews suggest that while FIRST participants reported improvements in twenty-first century skills, comparison group students were more likely to engage in other school and community-based extracurricular activities which also promoted the development of these skills (Burack et al., 2019; Melchior et al., 2018).

Theme 3: Coaching/Pedagogical Strategies to Support Student Learning and Skill Development (2013–2019)

The influence of coaches’ skills and pedagogical expertise upon students’ learning and skill development in the FLL Challenge is a potentially underrepresented area of research. Multiple studies suggest that the positive impacts of FLL participation on students’ STEM attitudes, learning, engagement (Theme 1.2), and twenty-first century skills (Theme 2) are highly dependent on the supporting adult coaches’ robotics knowledge, skills, and pedagogical approach (Chen, 2018; Eguchi, 2017; Kaloti-Hallak et al., 2015a; Ma & Williams, 2013; Stewart & Jordan, 2017). Yet, many coaches lack relevant technical knowledge and experience, and initially struggle to facilitate highly complex, multi-disciplinary robotics projects requiring a high level of collaboration (Andic et al., 2015; Dwivedi et al., 2021; Eguchi, 2017; Kaloti-Hallak et al., 2019). Dwivedi et al. (2021) suggest that this problem is not unique to the FLL Challenge; noting that more experienced and knowledgeable coaches, especially those with access to relevant support materials and training, are able to guide their teams more effectively, resulting in more successful performances across a range of educational robotics competitions. While official FIRST professional development programs exist in the USA, supplemented by online community resources, research-based coaching and team resources are not easily accessible by the international coaching community, especially for those for whom English is an additional language (Andic et al., 2015).

To date, relatively few studies have explicitly examined effective coaching pedagogical approaches and instructional strategies to support student learning and skill development in the FLL Challenge. Ma and Williams (2013) identified a need for coaches to engage in explicit teaching and modelling of technical, collaboration, and self-management skills, as well as a need to guide teams’ engagement with unstructured, non-routine problem-solving strategies. This finding was supported by Kaloti-Hallak et al. (2015a, 2019, p. 127), who found that FLL coaches who implemented more student-centred pedagogical approaches ‘integrated with explicit teaching of various concepts’ facilitated more meaningful learning measured against Bloom’s taxonomy. Similarly, Stewart and Jordan’s (2017) ethnographic case study of a frustrated fifth grade FLL club participant reinforced the need to scaffold students’ learning and peer communication within the complex informal learning environment of robotics competitions. Stewart and Jordan (2017, p. 147) found that club coaches’ perception of their role as ‘facilitators rather than teachers’ resulted in their avoiding the use of guided questioning, explicit teaching, and providing constructive feedback. This unintentionally limited the learning opportunities afforded to the more inexperienced students on the team, and negatively impacted on the team’s communication and collaborative knowledge construction.

Conclusions

The findings of FLL studies published between 2004 and 2022 conducted with a range of methodologies and sample sizes in different international contexts are broadly consistent in reporting generally positive, although not always statistically significant, impacts of FLL Challenge participation on students’ STEM attitudes, interests, and motivation (RQ1, RQ2, & RQ3). These impacts were observed for underrepresented student groups, including female students, ethnic minority students in the USA, and students with special needs. These impact findings are similar to those of Kandlhofer and Steinbauer’s (2016) empirical evaluation of RoboCup Junior, another prominent international robotics competition; and they are also consistent with the learning outcomes proposed by Evripidou et al. (2020). Differences in study designs, methods, and sample populations, including demographics, educational level, and prior competition experience, may have influenced the differences in the statistical results (Kandlhofer & Steinbauer, 2016).

Research evidence of the long-term, statistically significant impacts of engagement in FIRST robotics competitions, including FLL, on students’ STEM attitudes, learning, and post-secondary education and career pathways is compelling, especially for female students. There is potential value in conducting a similar longitudinal study on FLL or another well-established robotics competition outside of a North American context.

Findings regarding impacts on students’ STEM technical learning outcomes are mixed, and these may be influenced by coaches’ pedagogy, team sizes, and the design of the competition. Interestingly, while some studies suggest students are motivated by the non-technical aspects of the FLL Challenge, such as the Innovation Project and Core Values, there has been very limited research examination of student learning and skill development in these areas.

Multiple qualitative and quantitative studies consistently report positive short-term impacts of FLL participation on students’ development of twenty-first century problem-solving, communication, and collaboration skills. There is evidence that FLL teams’ development and application of these skills impact upon their competition performance, and that these skills are assessed as part of the official judging procedure. There is inconclusive research evidence regarding the competition’s longer-term impacts on twenty-first century skills. The FLL Challenge’s emphasis on students’ development and application of twenty-first century skills is consistent with the findings of Kandlhofer and Steinbauer (2016) and Zuhrie et al. (2021), and supports Evripidou et al.’s (2020) finding regarding the competition’s expected impact on problem-solving and collaboration skills. Such results illustrate the educational value of robotics competition learning experiences beyond building and programming robots.

Recommendations

There is a gap in the research literature regarding effective coaching strategies to support students’ technical learning and twenty-first century skill development in FLL (RQ4). Like many of the competitions reviewed by Zuhrie et al. (2021), the FLL Challenge emphasises the use of a project-based learning approach; however, there is limited qualitative research exploring the quality and impact of specific coaching practices on student learning outcomes in informal extracurricular robotics competition learning environments (Kandlhofer & Steinbauer, 2016; Ma & Williams, 2013). Future research in this area could inform the development of professional learning resources for robotics coaches, and possibly support teachers interested in integrating project-based-learning robotics activities in their curriculum programs.

Further to the above recommendation, future studies could examine instructional strategies and competition designs to address the gender imbalance in female students’ participation in FLL and more advanced educational robotics competitions. They could also collect more detailed, nationally representative demographic and gender statistics of student participation in FLL and similar robotics competitions in different international contexts. The promising results of studies examining impacts of students with special needs suggest a need for further empirical investigation of strategies and adaptations to support their engagement in mainstream school and community robotics teams (Lindsay, 2019).

From a methodological perspective, future studies of FLL and similar robotics competitions could explore the use of alternative data collection methods before and during the competition season to help address the limitations of solely relying on post-season surveys. Post-season surveys, typically conducted at state and national championship level events, rely on students’, coaches’, and parents’ self-reporting perceived impacts of competition participation, and their responses may be influenced by teams’ enjoyment or performance in the event. Future studies could consider the use of pre- and post-surveys or interviews, and semi-structured observations of team processes, learning activities, and coach pedagogy during the competition season. Studies could also include measures of teams’ tournament performances, such as judging rubrics, robot game scores, or observations of team judging interviews. Data collection at smaller regional competition events could improve the representation of newer, more inexperienced FLL teams who rarely qualify for advancement to championship level events.

Further qualitative studies could investigate factors and pedagogical interventions impacting on students’ participation, learning, and twenty-first century skill development in FLL and other robotics competitions, including specific impacts on underrepresented student groups.

Limitations

This review focuses on the impact of a specific educational robotics competition run by one international organisation. Similar to other competitions, FLL students are more likely to be nominated or choose to be involved in the competition based on their prior interests in STEM and robotics. This makes it very difficult to generalise impact findings beyond the extracurricular context of this competition.