Introduction

Many science curricula emphasise the need to contextualise science in real-world problems to help make science learning more relevant and meaningful for students (Goodrum & Rennie, 2007). The ability to critically engage with and address these real-world problems has been increasingly important as students face new and existing socio-scientific issues (SSIs) such as climate change, air pollution, sustainability, and genetic engineering. This emphasis has shifted into the forefront as science education places greater priority on promoting citizenship as an educational goal, rather than just focusing on student engagement, science knowledge, and skills development (Sadler, 2011). To address this need, researchers and educators are turning towards implementing alternative educational approaches like science, technology, engineering, arts and mathematics (STEAM). STEAM education integrates arts-related practices and perspectives (e.g., humanities and intercultural learning) to provide a different lens for students to interpret and understand science and comprehend how things in the real world relate to each other (Herro et al., 2017; Yakman, 2008). The inclusion of the arts brings in creative modes of learning and reframes the learning through the social and humanistic dimensions inherent in science (French, 2020; Abed, 2016). However, as STEAM-based education has continued to evolve to meet the demand that students need to understand systems and connections more deeply in relation to real-world issues, a new conceptualisation of the STEAM approach has emerged (Mang et al., 2021; Yakman & Lee, 2012).

SSI-based STEAM is a relatively recent interdisciplinary approach which integrates SSI into STEAM practices (Mang et al., 2021). By merging SSI perspectives with STEAM, teachers not only create exciting and creative learning opportunities, but also connect science learning more authentically and meaningfully to students’ everyday lives (Baek et al., 2022). This approach is more likely to allow students to see the purpose of science in their lives and thus encourage them to develop a stronger sense of responsibility and willingness to enact changes and new behaviours to address real-world issues (Alcaraz-Dominguez & Barajas, 2021; Choi et al., 2021). Studies have shown that SSI-based STEAM fosters the development of students’ conceptual knowledge, skills, and practices, as well as positive values and attitudes (Baek et al., 2022; Chu et al., 2019). By focusing equally on the development of all these components, teachers can help to enhance students’ scientific literacy (Mang et al., 2021). According to Lin et al. (2021), the use of STEAM could also reduce the gender-based interest and participation gap and differential participation in industrial development seen between girls and boys in the sciences.

SSI-based STEAM evolved from the awareness that STEAM as a standalone approach had shortcomings that impacted on its capacity to be implemented deeply and effectively in the classroom (Mang et al., 2021). For example, interdisciplinary connections are often superficially addressed in classroom activities, which makes it difficult for students to authentically engage with real-world problems (Herro et al., 2019). The lack of engagement with real-world problems becomes more apparent when the issues selected by teachers tend to lack complexity and social, cultural, personal, and historical relevance (Zeidler, 2016). These issues can be addressed by integrating SSI into STEAM practices and content. This is made possible because both approaches have overlapping elements such as the emphasis on inquiry-based learning, affective learning, and interdisciplinary learning (Fig. 1; Mang et al., 2021; Klosterman & Sadler, 2010). These intersecting elements found in SSI and STEAM could complement and help strengthen learners’ experiences with more connected science learning.

Fig. 1
figure 1

The SSI-based STEAM approach model

Full size image

Although there are overlapping elements, it is important to note that the pedagogical application of SSI-based STEAM is still distinct from SSI and STEAM. Table 1 provides a comparison of the key foundational characteristics of all three approaches and how they are adopted within the classroom. Generally, the SSI approach focuses on incorporating the intersection of science and society so that students can consider socio-political, cultural, moral, and ethical views that may enhance the real-world application of their scientific knowledge (Ekborg et al., 2009). In doing so, SSI teaching often adopts open-inquiry strategies to enable students to deal with science content framed within a problem/case that is both controversial and open-ended in nature (Klosterman & Sadler, 2010). The aim is to allow students to develop their own position on the SSI topic based on their investigations, and thus empower them to become activists on social and environmental issues (Klosterman & Sadler, 2010; Reis, 2014). The STEAM approach focuses on using the inherent connections between science and other disciplines as access points for eliciting students’ inquiry, interest, and curiosity to solve real-world problems (Hong, 2021). Therefore, inquiry-based learning strategies typically focus on positioning students in inquiry processes that provoke creativity, encourage exploration, and emphasise affective learning (Park & Park, 2020).

Table 1 Comparison of the key foundational characteristics of the SSI, STEAM, and SSI-based STEAM approaches
Full size table

Some common features such as contextualising science concepts in real-world problems can be seen at varying degrees across SSI, STEAM, and SSI-based STEAM. However, the way the common features are executed in the instructional process in each approach is different. SSI-based STEAM distinguishes itself from the other approaches by utilising a structured inquiry-based learning approach to guide SSI-mediated STEAM lessons. Through this instructional approach in SSI-based STEAM, students apply science concepts to study real-life social scientific issues and engage in action to address the issue. This is different from STEAM, where although science learning is situated in real-world problems, the problems need not be social scientific issues. SSI-based STEAM is distinct from SSI in that SSI focuses on social scientific issues, and while students do learn science in the process, the primary aim is to raise awareness of the issue and help students develop a position regarding it. Furthermore, the emphasis in this approach is on providing concrete, practical opportunities that teachers can plan within the classroom context to allow students to enact the knowledge, values, and practices they develop (Mang et al., 2021), enabling students to continually practice using and understanding the interconnected roles of knowledge, values, and practices when decision making and problem solving, which will better support their capacity to become global citizens who can address science-related social problems (Clément, 2012; Lee et al., 2013).

Many teachers support and value the importance of SSI-based STEAM; however, they often struggle to conceptualise how to interpret and translate STEAM practice into their classroom (Quigley & Herro, 2016; Won et al., 2021). This tends to lead to teachers designing and implementing activities that address content superficially and in a disjointed manner (Quigley & Herro, 2016; Rodrigues-Silva & Alsina, 2022). Furthermore, teachers struggle to engage in assessment strategies that evaluate the overall learning process rather than just the outcome or product (Cook et al., 2020). The challenge arises because teachers typically lack knowledge, experience, and understanding of SSI-based STEAM and therefore often are reluctant to implement SSI-STEAM practices (Won et al., 2021). This mismatch between teachers’ perceptions and their STEAM practices is often rooted in teachers’ misconceptions and biases about the approach and the science topics that may be targeted in an SSI and STEAM-based curriculum, as well as the perceived lack of professional development (PD) opportunities for teachers applying SSI-STEAM (Boice et al., 2021; Kind, 2019). Currently, there are some theoretical models such as Baek et al.’s (2022) SSIBL-STEAM and SSI-based STEAM (Mang et al., 2021) that are available to help teachers understand the theory behind the proposal to integrate SSI perspectives into STEAM. With the addition of PD opportunities, teachers could develop a stronger understanding of the benefits of adopting STEAM as a teaching approach (Herro & Quigley, 2016). However, studies such as Boice et al. (2021) and Cook et al., (2020) found that while STEAM PD improved teachers’ knowledge and practices, they still required ongoing support to identify and plan STEAM lessons and encourage the actual practices. For example, Herro et al., (2017) developed a rubric called ‘co-measure’ to help teachers and educators assess student collaboration in STEAM activities. The development of similar programs or assessment tools in SSI-based STEAM would provide stronger support and incentivise teachers to implement the approach. At present, it appears that there are no quantifiable standards available to help teachers identify the overall components of SSI-based STEAM such as task-relevant assessment strategies, selecting meaningful SSI contexts, or ways to meaningfully include interdisciplinary and integrated practices (Cook et al., 2020; Won et al., 2021). Furthermore, there are only limited examples of such SSI-based STEAM programs available; as a result, teachers would need to rely on their own intuition to determine whether the programs already adopted are effective, and how they could be revised for future use. The study presented here is part of a joint project between researchers in Australia and Korea, who have collaboratively developed a program evaluation rubric to help teachers plan and assess SSI-based STEAM programs used for science learning.

The SSI-Based STEAM Approach Model and the 6E Inquiry Model

In the context of this study, the authors’ SSI-based STEAM approach will be adopted. According to Mang et al., (2021), teachers implementing the approach need to understand three dimensions (Fig. 1): (1) learners as active participants, (2) the four key principles (enacted values and practices, affective learning, authentic context and activities, and interdisciplinary thinking and integrated practices), and (3) three educational pedagogical approaches (social constructivist-, situated-, and inquiry-based learning). To translate this theory into practice and identify both the STEAM and SSI components, teachers need to adopt the 6E inquiry instructional model and engage students with six stages of inquiry: engage, explore, explain, elaborate, evaluate, and enact (Appendix, Table 5). The 6E inquiry model adapts Bybee et al.’s (2006) 5E inquiry model by including an additional inquiry stage known as ‘enact’. In the ‘enact’ stage, there is a focus on giving students the space to have meaningful discourse and dialogue about different socio-cultural and political values and views, and how they affect social practices. Students are then given opportunities to put into action the values and practices they have learned. This inquiry model helps guide teachers in relinking the science concepts and activities back to the SSI topic, emphasises the interdisciplinary dimensions, and allows students to use their own thoughts, values, and emotional experiences to mediate their learning and their exploration and negotiation of different identities that they may adopt in their lives (Gao et al., 2019; Mang et al., 2021). This fosters stronger contexts for students to understand how values, identities, and emotions affect the way knowledge is used in practice and decision-making (Lee & Brown, 2018; Sjöström et al., 2017).

With the lack of available content and pedagogical knowledge or experience of SSI-based STEAM teaching, it is crucial that instructional support such as program evaluation tools be accessible (Won et al., 2021). Such evaluation tools can help teachers to gauge the extent to which a program reflects the approach and targets related skills and content. This in turn ensures teachers are designing appropriate learning opportunities that explicitly reflect the core dimensions and principles of SSI-based STEAM and can make adjustment to the programs, materials, and instructions (Herro et al., 2017). Not only can this enhance students’ learning outcomes, but it can also support teachers’ successful implementation of SSI-based STEAM practices. Currently, there are no program evaluation tools available to assess the quality and scope of SSI-based STEAM programs.

The Need for a Program Evaluation Rubric

When developing and implementing programs in the science classroom, teachers need to carefully consider whether their program constructively aligns with relevant outcomes, goals, curricula, educational theories, and instructional strategies (Centre for Education Statistics and Evaluation, 2015). To do this, teachers need to have adequate knowledge about the program and curricular and pedagogical practices (Kind, 2019). SSI-based STEAM PD or teacher training is one way to help teachers improve in areas such as using inquiry-based instruction and planning appropriate lessons (Rodrigues-Silva & Alsina, 2022). As teachers continue to develop their capacity to plan lessons using unfamiliar approaches like SSI-based STEAM, it is vital that ongoing support is provided after the training is finished, to facilitate their further learning and practice. In Boice et al.’s study (2021), teachers were provided additional support after their STEAM PD, such as access to materials, lesson modelling, and weekly newsletters with resources, and meetings. However, to continue to support teachers to determine and evaluate the quality of their SSI-based STEAM programs or materials, program evaluation tools like rubrics are vital.

A rubric is an evaluation tool that sets out the criteria that reflect dimensions of interest and are used to make a judgement about the performance or quality of work done (Martens, 2018a, b). The use of rubrics will ensure greater transparency during the evaluation process and allow teachers or researchers to determine whether the program meets targeted outcomes, and thus clarify what SSI-based STEAM program quality and success look like (Dickinson & Adams, 2017; Wilkerson & Haden, 2014). The inclusion of a program evaluation rubric during program planning provides opportunities to revise throughout the process and helps teachers to justify how well their own developing or existing programs reflect the approach (Martens, 2018a, b; Wilkerson & Haden, 2014). Furthermore, the evaluation rubric could be used to improve teachers’ teaching and learning practices by helping them to document program and activity development during program planning and to ensure that they consistently apply the approach for student learning (Kind, 2019).

Climate Change as an SSI Topic

In this study, climate change was selected as the SSI topic of interest for program evaluation. It is a current global issue, with catastrophic impacts being observed on aspects such as food security, biodiversity, wellbeing, weather patterns, and sea levels (Intergovernmental Panel on Climate Change [IPCC], 2018). The work presented in this paper was adapted from a Master’s thesis study conducted in 2019–2020. At that time, Australian students experienced extreme environmental events including droughts, water restrictions, bushfires, intense hailstorms, severe thunderstorms, and cyclones (Australian Bureau of Statistics [ABS], 2020). In the state of New South Wales, students experienced the ‘Black Summer' fires, the worst bushfire season in state history, with more than 19 million hectares of land burnt, over 3000 houses destroyed, and 3 billion animals killed or displaced (Filkov et al., 2020; NSW State of Environment, 2021). Therefore, climate change was an appropriate topic to select for the purposes of this study due to it being a compelling and emotional event that students were experiencing and have since continued to experience.

Although students tend to be aware of climate change, research has indicated that many students have poor knowledge of and attitudes towards learning about the topic, often disengaging during learning (Corner et al., 2015; Kuthe et al., 2020). It has also been noted that this type of student tends to be more likely to feel psychologically distant from the problem: such disengaged students possess a low sense of agency and are less likely to participate in any social or environmental practices aimed at addressing the issue (Corner et al., 2015; Yilmaz & Can, 2019). Due to these factors, there has been a greater focus on incorporating stronger climate change education into science curricula and classroom learning (Hestness et al., 2019; Oliver & Adkins, 2020). Additionally, science researchers and educators have continued to explore ways of identifying effective climate change education strategies and programs (Monroe et al., 2019). The aim has been to improve student climate change literacy, attitudes, and awareness, so that they can understand the interdisciplinary connections and personal connection that the climate change issue has with their daily lives (Bunten & Dawson, 2014). This, in turn, helps to invoke students’ sense of agency and empowerment to make better-informed decisions and engage in social and environmental adaptation and mitigation practices (Trott, 2020).

Significance of This Study

This study seeks to fill the present gap in SSI-based STEAM research by designing and providing a program evaluation rubric that informs teachers about the key features for program development. As highlighted by Won et al. (2021), teachers face challenges in using SSI-based STEAM because they lack both the pedagogical knowledge and the understanding of the approach, as well as experience with planning and using the approach in their classrooms. To better guide teachers to develop quality SSI-based STEAM programs, it is necessary to provide them with program evaluation tools such as rubrics.

The rubric will allow teachers of any background and experience level to make evidence-based judgements about SSI-based STEAM programs rather than resorting to personal assumptions (Dickinson & Adams, 2017). It can also help them to articulate which parts of their programs align more easily and clearly with the criteria of the approach and with the 6E inquiry instructional model (Dickinson & Adams, 2017). Teachers will thus be able to collect evaluative information about the relevance, quality, and success of the programs in helping students to reach the short- and long-term learning goals targeted (Boyce, 2017; Martens, 2018a, b). The rubric also has the potential to be used in teacher professional development settings by providing well-defined criteria that can help teachers identify the weaknesses and strengths in their own practices and in the scope across which they teach certain science topics (Kind, 2019).

Purpose of This Study

The purpose of this study is to investigate how the ‘SSI-based STEAM Teacher Evaluation and Program Development’ (SSTEPD) rubric identifies the SSI-based STEAM approach in programs. Through this investigation, we aim to provide teachers with a program evaluation tool that can be used to help them plan, select, and assess the effectiveness of SSI-based STEAM programs and resources. This study will address the following research questions:

RQ1. How well does the SSTEPD rubric measure the SSI-based STEAM dimensions and principles in climate change programs?

RQ2. To what extent can the SSTEPD rubric be applied across different types of climate change programs?

Methodology

The SSTEPD rubric was developed following a multi-phase process adapted from design-based research (DBR) methodology (Brown, 1992). Drawing from the interconnected phases of the DBR method (Plomp, 2007), this study followed three phases: (1) preliminary research, (2) prototyping, and (3) assessment (Fig. 2). In Phase 1, preliminary research was conducted to identify the elements that needed to be incorporated into a defined rubric that can evaluate SSI-based STEAM programs. In Phase 2, the program evaluation rubric was developed based on the findings from the preliminary research and subjected to feedback and validation from an expert panel. Feedback from the expert panel was then used to revise the rubric. In Phase 3, the finalised program evaluation rubric was applied to actual programs to assess whether it could evaluate the specific criteria targeted. The information gathered was used to make further revisions and recommendations for future practice.

Fig. 2
figure 2

The three phases of the multi-phase development process: preliminary research, prototyping, and assessment

Full size image

Preliminary Research Phase

The existing literature relevant to SSI-STEAM, scientific literacy, and inquiry-based learning was reviewed to identify key elements and goals consistent with the approach. From the literature review, several elements and key goals related to three main themes were identified: the SSI-based STEAM approach, the 6E instructional model, and the three visions of scientific literacy (Fig. 3). For example, it has been emphasised that SSI-based STEAM lessons need to encourage students to act on their own sense of social and scientific responsibility when addressing real-world issues (Mang et al., 2021). The use of inquiry-based instructions included the use of collaboration, hands-on activities, and posing questions (Mang et al., 2021). These elements identified from the literature aligned well with the three visions for scientific literacy. Vision I focuses on developing scientific knowledge, skills, and processes for understanding (Roberts, 2007). Vision II focuses on contextualising science so that a learner can understand how science is applied in real-world situations and human endeavours (Roberts & Bybee, 2014; Tan, 2016). Vision III focuses on integrating various perspectives of science from a socio-cultural and political point of view (Tan, 2016). The list of educational elements and goals helped define the key teaching and learning criteria and the evaluation categories.

Fig. 3
figure 3

Goals, dimensions, and principles reflected in the SSTEPD rubric

Full size image

Prototyping Phase

This phase comprises of two main components: (1) designing the rubric and (2) validating the items with a panel of experts.

Designing a Rubric

After the educational elements and goals were identified, they were used to generate measurable evaluation criteria. Each criterion had to define a key component of SSI-based STEAM and as such needed to reflect one or more of the key dimensions outlined in the SSI-based STEAM model (Appendix, Table 6). However, teachers need to plan and select activities that not only reflect the approach but also target certain learning opportunities embedded in the 6E inquiry model (Fig. 4). To assist teachers in establishing how to address the activities in the different inquiry stages in the 6E model and the general outcomes of SSI-based STEAM learning, we determined that the rubric needed seven evaluation categories: (1) learning outcomes, (2) engage, (3) explore, (4) explain, (5) elaborate, (6) evaluate, and (7) enact.

Fig. 4
figure 4

Learning opportunities targeted at different stages of the 6E inquiry model (Mang et al., 2021)

Full size image

Once the evaluation criteria and categories were identified, various rubric guidelines were explored to determine the rubric format. To address concerns about time constraints and ease of use, a rubric checklist style was selected. This format allows teachers to determine the level of the performance without the complexity of detailed descriptors for each criterion found in traditional rubrics, which can be a lengthy process to complete (Enszer & Buckley, 2020). Teachers and educators rated the extent to which the program reflected the criteria on a 4-point Likert scale: great extent, (3) somewhat (2), very little (1), and not evident (0). This scale provided teachers with a score that could be tallied up at the end of the evaluation to provide an overall score total. For example, if the program facilitated opportunities for students to actively participate in developing local and global solutions to ‘a great extent’, then a score of 3 was given. The final score obtained gave teachers an indication of the overall quality of the SSI-based STEAM program.

Validation Process

After the first draft of the SSTEPD rubric had been created, it was validated by a panel of six experts from the fields of STEAM education, SSI education, and science education research. An online survey was administered to the panel to rate whether the rubric items targeted the constructs and were specific, measurable, attainable, realistic, and timely (SMART; Wilkerson & Haden, 2014). The ratings were used to establish the face validity and calculate the content validity ratio (CVR). Any item that did not achieve an CVR of 1 was deemed invalid and modified or removed. The ratings and additional written feedback guided three consultation sessions and new iterations of the rubric. The final rubric consisted of 37 items, each of which reflected one or more principles of the SSI-based STEAM model (Appendix, Table 6).

Assessment Phase

To determine that the rubric accurately represented and could evaluate the approach, two science educators, one from a school and one from a university, used the developed rubric to assess a newly developed SSI-based STEAM climate change program and five other existing climate change programs. The science educators independently rated and scored the programs using the SSTEPD rubric. Any differences in how scores were given or how criteria were interpreted were discussed until both educators reached an agreement. This process provided insights into how criteria were interpreted, and scores were allocated, and hence helped identify differences between the programs and how each program was addressing one or more SSI-based STEAM dimensions or principles. Afterwards, the researchers held joint sessions with all the science educators to discuss their ratings and determine how each expert had interpreted the rubric when providing their score. The climate change programs evaluated were developed by various parties including university researchers, government, and council initiatives, external education course providers, and schoolteachers (Table 2). Applying the SSTEPD rubric across a range of programs helped to establish whether it could be universally used to assess programs from other contexts. This evaluation allowed us to highlight the strengths and weaknesses of the rubric and make judgements on its suitability and practicality for implementation by teachers and researchers. This allowed for further recommendations to be made for future studies.

Table 2 Programs evaluated using the SSI-based STEAM program evaluation rubric
Full size table

Results

The SSTEPD rubric was used to score the programs and the results were analysed to help answer two key questions regarding whether the rubric could (1) measure the SSI-based STEAM dimensions and principles and (2) be applied across different types of climate change programs. Two science educators independently evaluated and scored the program before engaging in discussions to compare the scores provided. The scoring was highly consistent between both the science educators and between the different programs. The kappa coefficient calculated showed the level of agreement was within the range 0.70–0.85, supporting a high level of inter-rater agreement (McHugh, 2012).

Measuring SSI-Based STEAM Dimensions and Principles

The final iteration of the SSTEPD rubric received strong agreement and positive feedback from the expert panel, who agreed that most of the evaluation criteria and categories would be able to measure SSI-based STEAM dimensions and principles clearly. To explore how well the SSTEPD rubric could achieve this goal in practice, the rubric was first applied to a newly developed SSI-based STEAM program on climate change. The educators found that the format of the SSTEPD rubric was user-friendly and allowed easy identification of evident or missing SSI-based STEAM components. However, there were a few issues that had an impact on each user’s interpretation of whether the program addressed a certain criterion. Firstly, it was unanimously agreed that certain terminology such as ‘social and scientific responsibility’ can be interpreted and defined differently depending on the user. It was recommended that a clear definition needed to be provided to users to ensure that there is a consensus on the meaning of the terms used. Secondly, there was a lack of explicit examples of what some activities may look like in practice. For example, it was unclear what type of activities reflected the ‘use of arts knowledge and practice to make connections to science and SSI’. Overall, the science educators still strongly agreed that the rubric was highly capable of measuring the SSI-based STEAM dimensions and principles in climate change programs.

Universality of the SSTEPD Rubric

To test the universality of the SSTEPD rubric, six different climate change programs developed by different program developers were evaluated using the rubric. These program developers included external education course providers Cura and Stile Education, Western Washington University, the Great Barrier Reef Marine Park Authority, and schoolteachers. It should be noted that while the programs were all on the same topic of climate change, they adopted different educational and inquiry approaches. As such, the study’s goal here was to determine what SSI-based STEAM components might already be present in these existing programs or where such components might be missing, thus offering a great opportunity for SSI-based STEAM inclusion in those existing programs. This process, in turn, would give a clear indication as to whether the SSTEPD rubric could consistently assess and score programs across various contexts. As shown in Table 3, the overall score was calculated and converted into percentage terms for scoring comparisons.

Table 3 Summary of the scoring results for each climate change program (total full score: 111)
Full size table

The SSI-Based STEAM Climate Change Project Group Program

This program demonstrated a high capacity to successfully address the criteria in the rubric and therefore received a score of 101 out of 111 (91%). This program was specifically designed to reflect the SSI-based STEAM approach, and as a result had a strong focus on using SSI to contextualise the science concepts and on creating wide ‘enact’ opportunities, for example, for students to critically evaluate the roles of values and identities in social practices. Thus, the program was able to score higher in many evaluation categories such as ‘engage’ and ‘explore’. However, the raters noticed that there was a lack of explicit inclusion of activities that facilitated the use of arts-related knowledge and practices to reinforce or make connections with science. This shortcoming had an impact on how well the program was able to address the principle of ‘interdisciplinary thinking and integrated practices’. As a result, this program scored poorly (M = 2.50) in the categories ‘explain’ and ‘elaborate’ (Table 4), where many of these types of activities were embedded.

Table 4 Item mean, item total, and rubric total score in six selected programs
Full size table

The Abandon or Adapt? Program by Cura

This problem-based learning (PBL) STEM program scored 73 out of 111 (66%) and strongly demonstrated its ability to engage students in the inquiry process and thus develop their scientific knowledge and skills. The program includes many hands-on activities and collaborative learning devices to help students engage with their learning and offers several opportunities for students to reflect on and assess their own learning. However, it does not take advantage of several access points for integrating socio-scientific components such as exploring values, ethics, and morals. As a result, it received the lowest mean score (M = 1.33) in the ‘enact’ category. Furthermore, the program’s superficial application of the ‘interdisciplinary thinking and integrated practices’ concept, which is limited in this program to opportunities such as drawing, raised some concerns.

Earth Systems: Climate Change by Stile Education

This online module scored 62 out of 111 (56%) and focuses strongly on developing students’ scientific knowledge and understanding. It consists of many self-directed learning activities and promotes the use of student and teacher assessment to monitor student progress. This allowed the module to receive full scores in the ‘evaluation’ category (M = 3). Hands-on investigations and research activities were presented to engage students with the SSI topic and apply their scientific skills. However, at times, there were very limited opportunities present for students to pose questions or create hypotheses or predictions. The technology used was well integrated to help facilitate more blended learning and reinforce learning. Interdisciplinary thinking and integrated practices such as the use of the arts were adequately addressed in the module. However, there were missed opportunities to integrate socio-cultural components and perspectives or activities to facilitate the enactment of values and practices (M = 1). This really removed the affective learning experiences that would connect the learning more to students’ emotions. Overall, the program did not address many of the other SSI-based STEAM principles deeply enough and thus scored poorly across all categories (Table 4).

Climate Change: Connections and Solutions Program by Facing the Future

This interdisciplinary program developed by Western Washington University researchers received a high score of 109 out of 111 (98%). The program makes clear and authentic connections between climate change and students’ daily lives, and thus provides optimal opportunities for them to engage deeply with the SSI raised. For example, under this program, students engage in tasks that encourage them to consider the impact of their consumer behaviour on climate change. Furthermore, many of the program’s activities challenge students to use interdisciplinary perspectives to understand the program and develop solutions. Hence, the program received high mean scores (M = 3) in most of the evaluation categories. However, the program could offer greater benefits to students if it provided more opportunities for more diverse technologies, designs, and practices to help them understand and explain concepts. Additionally, this program does not promote more in-depth discussion about different cultural and personal values. As shown in Table 4, these factors lowered the program’s mean scores for the evaluation categories ‘explain’ (M = 2.80) and ‘enact’ (M = 2.83).

The Reef Guardians: Climate Change Program by the Great Barrier Reef Marine Park Authority

This council program scored 85 out of 111 (77%). The program follows the 5E inquiry model and provides many hands-on activities such as investigations and group tasks. This facilitates quality opportunities for students to collaborate with their peers. However, the program follows a teacher-guided inquiry approach, which limits opportunities for learners to engage in more student-driven activities to understand the problem more openly. This made it challenging to see how the program could effectively address SSI-based STEAM principles, and thus lowered the program’s scores across all evaluation categories in our study (Table 3). The program did put forward perspectives from various other dimensions, such as politics and economics. However, it neglected to introduce students to perspectives from dimensions such as ethics, culture, and values. As a result, it appeared that this program could not facilitate opportunities for students to apply values and social practices; hence, it scored very poorly in the ‘enact’ category (M = 0.83).

Program Developed by Schoolteachers

This teacher-developed program performed the worst of all the programs, with a score of 47 out of 111 (42%). It was difficult to determine what inquiry model had been adopted in the program as many of its activities were teacher directed. There was an over-reliance on the use of research tasks, lectures, and worksheets, which hindered opportunities for students to engage actively as learners or in student-driven activities. The content focused strongly on the science content, such as greenhouse gases and the carbon cycle, reinforced by some opportunities for hands-on investigations. However, it was noted that the tasks only superficially linked the science concepts to climate change as an SSI. Limited opportunities were provided to students to engage in strong discourse about the real-world problem of climate change or to develop and apply wider perspectives from areas such as culture and ethics. This did not allow the program to adequately address any of the important principles of SSI-based STEAM—‘enacted values and practices’, ‘interdisciplinary learning and integrated practices’, ‘authentic context and activities’, and ‘affective learning’, for example. This resulted in low mean scores (M = 1.80) for this program across all evaluation categories.

Discussion

The intent of this SSTEPD rubric was to help guide teachers and educators in identifying features of SSI-based STEAM, measuring the quality of their programs, and developing their own programs. Moreover, teachers could use the tool to ensure that there was alignment between the curriculum and goals of SSI-based STEAM learning. By consulting a panel of experts from relevant fields, the study was able to produce a rubric that reflected (1) the four key principles of SSI-based STEAM, (2) the 6E inquiry model, and (3) the three visions of scientific literacy. To ensure the SSTEPD rubric can assess the activities and outputs targeted, the tool was applied to various climate change programs. The SSTEPD rubric demonstrated the ability to rate the principles and dimensions reliably and consistently in various climate change programs. In doing so, the rubric was also able to highlight gaps and identify areas in the programs where there were opportunities to include SSI-based STEAM learning.

However, we noted that in some items, terms, such as ‘social action’, were ambiguous, so their interpretation is dependent on the rubric user. This leaves room for discrepancies to arise and thus has an impact on how certain programs may be scored between different rubric users. These discrepancies may become more apparent between different stakeholders who may assign their own values and goals to the programs (Dickinson & Adams, 2017). Another area of concern about the SSTEPD rubric relates to some items’ focus on assessing whether a lesson could facilitate opportunities for student experiences in areas such as ‘social and scientific responsibility’. Based solely on the wording of the criteria and with no example of how a concept like this can be represented through a practical activity, it can be difficult for teachers to know what to look for in the programs and how to assign a score (Dickinson & Adams, 2017). It is important to clearly define how these principles or dimensions can be manifested as practical activities (Allen & Tanner, 2006).

The checklist format of the SSTEPD rubric engages teachers in a process of rating and scoring the evaluation criteria, and thus captures qualitative information about the quality of the program being evaluated. Additionally, the checklist style helps teachers to avoid the potential problem of describing and justifying the grade given to work that may fall between performance descriptors (Enszer & Buckley, 2020). Teachers can then easily and quickly gather evidence about how the program reflects SSI-based STEAM and how well it meets the criterion, and they can use it to guide self-assessment (Bharuthram & Patel, 2017). However, the format does not allow the option for the teacher to note written or verbal feedback during their evaluation. Supplementing the SSTEPD rubric with an additional column for annotations or notes about evidence for scoring or suggestions for modification could further promote teacher reflection and a narrative for discussion between colleagues (Dawson, 2017).

Despite the differences in the activities, frameworks, and pedagogical approaches adopted in the programs, the rubric demonstrated its ability to consistently give scores across the categories and to highlight certain trends. This is because the dimensions and principles of SSI-based STEAM (e.g., authentic context and tasks) are deployed in many educational approaches including STEM. Therefore, we believe that the differences between the programs do not affect or detract from the capabilities of the SSTEPD rubric. In fact, the SSTEPD rubric was designed to be specific in targeting the evaluation contexts, which allows the tool to be applied across different contexts and types of programs (Martens, 2018a, b). For example, five out of six of the programs showed a weakness in addressing the ‘enact’ category, failing to identify sufficient connections with socio-scientific dimensions such as values and social practices. As a result, students were not provided with adequate opportunities to put into practice their understanding of the roles of different values and practices in decisions made to combat climate change. Another trend observed across the programs was the strong focus on assessment. This is likely due to the importance that is placed on student performance and gathering evidence of students’ achievements relevant to desired outcomes, as well as other concerns such as teacher accountability (Bell & Cowie, 2001).

Overall, the SSTEPD rubric was able to provide an insight into the quality of the programs examined in relation to SSI-based STEAM learning. The Climate Change: Connections and Solutions (Facing the Future) program from Western Washington University was explicitly developed as an interdisciplinary program that effectively made strong connections between different dimensions and thus scored highly on the rubric. The other programs developed by external education course providers scored relatively well but were very weak in terms of reflecting the dimensions of ‘enacting values and practices’, ‘affective learning’, and ‘interdisciplinary thinking and integrated practices’. The program developed by schoolteachers showed the lowest scores across all dimensions. The low scores can be attributed to the lack of cohesion between program activities, the failure to consider how various dimensions interconnect with each other, and how this could facilitate a more connected learning experience for students.

Benefits of the Rubric

Most programs are created with the intention of assisting teachers to educate students effectively and achieve targeted educational objectives (Centre for Education Statistics and Evaluation, 2015). Rubrics are a great tool for setting teachers’ expectations when they assess the delivery and quality of the activities they plan in their programs so that they can achieve their targeted outcomes (Allen & Tanner, 2006). The SSTEPD rubric breaks down the specific features or instructional strategies that are being assessed and draws teachers’ focus to the strengths, weaknesses, and gaps in their own programs, thus guiding program development. Not only can teachers evaluate their own program using a rubric, but also the rubric can aid them in establishing the merit of other existing programs based on standards and evidence that allow inter-program comparisons and conclusions to be made (Dickinson & Adams, 2017). Furthermore, the use of a rubric has merit for outcome evaluation, in that it can be used to determine the impact the program has on students’ learning outcomes and hence determine whether it is suitable for the target audience (Centre for Education Statistics and Evaluation, 2015). From this process, teachers can more effectively assess the quality of a program and can judge whether it has value for teaching and learning (Allen & Tanner, 2006).

The SSTEPD rubric could also provide consistent and explicit expectations between teachers within the same faculty or across multiple disciplines (Timmerman, 2011). This, in turn, could help faculties to plan more holistic assessments that will engage students in both the outcomes and the processes of learning (Timmerman, 2011). In these cases, where more than one rubric user is involved, the SSTEPD rubric would facilitate clearer communications about the expectations and standards that constitute quality programs and how these expectations are reflected in classroom learning (Dickinson & Adams, 2017). Through a communication process, teachers can then start to form a better idea about what constitutes quality SSI-based STEAM programs. Moreover, the evaluation data collected will streamline the cycle of evaluating, analysing, receiving feedback, and revising the programs and related materials (Dickinson & Adams, 2017; Wilkerson & Haden, 2014). This facilitates teachers’ deeper involvement and engagement with their teaching tools, enabling them to understand and reflect on their own practices (Allen & Tanner, 2006; Kind, 2019). Overall, the program evaluation tool is a valuable resource for teachers currently implementing or wishing to implement SSI-based STEAM, who, due to a lack of guidance, self-efficacy, or knowledge, find it difficult to make science more meaningful to their students by engaging them in developing a greater sense of responsibility about social scientific issues (Martens, 2018a, b; Won et al., 2021).

Conclusions and Future Studies

The SSTEPD rubric is a valuable tool that provides a tangible way for teachers to assess, plan, and implement SSI-based STEAM and that has several implications for teaching and learning. Firstly, the rubric highlights for teachers the criteria that they need to look for when developing and evaluating their own and existing programs. Secondly, it provides a benchmark for teachers to communicate standards and expectations among and across faculty members, and to discuss the strengths and weaknesses of activities and programs implemented (Dickinson & Adams, 2017). The rubric can also be used to promote negotiations among teachers about what elements in their program need to be revised (Martens, 2018a, b). This can encourage teachers to reflect on and improve their own teaching practices (Allen & Tanner, 2006; Martens, 2018a, b). Lastly, the rubric has the potential to guide the planning and design of new programs and activities (Cornman et al., 2013).

Although the SSTEPD rubric was developed based on the expertise and feedback of experts and researchers in the field, it still lacks critical input from teachers. As a result, it is difficult to determine definitively whether it has practical applications in the classroom and is user-friendly and whether teachers have the skills and experience to utilise the rubric effectively (Martens, 2018a, b). Future studies need to involve teachers in any rubric revisions and explore how the SSTEPD rubric works in the classroom setting. This will provide valuable insights into what factors may have an impact on the use of the tool, such as teaching experience with using rubrics, evaluation practices, and time constraints. Additionally, it may reveal gaps in how professional teacher workshops may use the rubric to expand teachers’ understanding and acceptance of the SSI-based STEAM approach, and their willingness to implement it in their classrooms.