Collaborative analytics-supported reflective Assessment for Scaffolding Pre-service Teachers’ collaborative Inquiry and Knowledge Building

Abstract

Helping pre-service teachers (PSTs) develop competencies in collaborative inquiry and knowledge building is crucial, but this subject remains largely unexplored in CSCL. This study examines the design and process of collaborative analytics-supported reflective assessment and its effects on promoting PSTs to develop their competencies in collaborative inquiry and knowledge building. We used a quasi-experimental design that lasted 18 weeks. The experimental group was a class of 40 PSTs who took a liberal studies course with a knowledge building design enhanced by collaborative analytics-supported reflective assessment. The comparison group was a class of 28 PSTs taught by the same instructor who studied the same inquiry topics but experienced a regular knowledge building environment using portfolios. The analysis of the PSTs’ Knowledge Forum discourse showed that collaborative analytics-supported reflective assessment helps PSTs develop collaborative inquiry competencies for community knowledge advancement. The analysis of the PSTs’ reflection using collaborative analytics and prompt questions showed that the design using KBDeX visualization and knowledge building rubrics helped them engage in productive collaborative knowledge building inquiry by involving them in continuous monitoring, analysis, negotiation, synthesis of inquiry, identification of promising routes for inquiry, and actions to guide further collective inquiry. Implications for designing CSCL collaborative-analytics enriched with reflective assessment and student agency, and broadening CSCL and knowledge building approaches to pre-service teacher education are discussed.

Introduction

Within this complex world, changes in society and educational contexts have generated new goals and expectations among students and teachers. It has become increasingly important for students to learn how to use digital technologies to engage in inquiry, create knowledge, and direct these processes. At the same time, it has become an increasingly valuable goal for students to be able to inquire and learn with others, engaging in productive dialogue that leads to new understanding (Wegerif, 2013). Over the past two decades, research into computer-supported collaborative learning (CSCL) has progressed substantially in terms of understanding students’ collaborative interactions and designing CSCL technologies and environments that help scaffold collaborative inquiry and higher-order competencies (Cress et al., 2021; Stahl, 2015). Although research in the field has primarily focused on helping students as collaborative learners, there is now an increased interest in examining how teachers can develop the competence to support student collaboration, and how they can continue to grow as teachers and learners in CSCL classrooms. In charting the future trends of CSCL, Wise and Schwarz (2017) referred to the “growing awareness that teachers are indispensable for the effectiveness of collaborative learning technologies” (p. 453) but also lamented “the lack of attention in teacher preparation programs to both collaborative learning and technology” (p. 454).

CSCL research has examined teacher practice in developing productive collaboration in classrooms (Furberg, 2016; Olsen et al., 2021) and teacher communities (Chan, 2011; Jesús Rodríguez-Triana et al., 2020; Yoon et al., 2020), but less research attention has been given to pre-service teachers’ (PSTs) learning about collaborative inquiry and knowledge building. It is important to cultivate PSTs’ competencies in collaborative inquiry and knowledge building because they are also learners and because as future teachers, their teaching design and practice will greatly influence the development of students’ knowledge and competencies (Uiterwijk-Luijk et al., 2019). If they are well prepared for these endeavors, their future contributions may help transform schools and classrooms into knowledge creation organizations and inquiry communities that contribute to students’ development of higher-order competencies (Bereiter & Scardamalia, 2014; Wells, 1999). Amid the dearth of CSCL research on PSTs using a collaborative inquiry approach, this study aimed to investigate how they could develop their competencies in collaborative inquiry and knowledge building supported by collaborative analytics and reflective assessment.

The literature on collaborative inquiry can be found in two domains of research: CSCL and the examination of inquiry in science education (Bell et al., 2010). CSCL researchers have distinguished between collaborative learning and cooperative learning (Dillenbourg, 1999) and pioneered research into the topic by examining how student dyads collaborate with the support of computers to solve physics problems (Roschelle, 1992). In science education, collaborative inquiry has been used to help learners co-construct knowledge and practice science as a common endeavor undertaken with peers, as do real scientists; this has allowed them not only to learn scientific content but also the nature of science, especially how scientific knowledge is generated (Bell et al., 2010).

As alluded to above, collaborative inquiry has the potential to cultivate learners’ higher-order competencies such as inquiry, collaboration, metacognition, and knowledge building, all of which have been increasingly emphasized in the ever-changing knowledge society (Braund & Driver, 2005; De Backer et al., 2011; Rocard et al., 2007). However, helping students develop collaborative inquiry and knowledge building competencies and cultivating their autonomy are highly complex processes (Hong, Lin, Chai, et al., 2019). Teachers often find it difficult to develop collaborative inquiry practices, which require shifting to a different epistemology, and thus they often resort to shallow inquiry methods (Chinn & Malhorta, 2002; Scardamalia & Bereiter, 2014). Given their own prior experience of schooling, PSTs often lack the necessary metacognitive awareness and skills; it is thus important to help them develop appropriate scaffolding strategies and accompanying tools. Unfortunately, conventional teacher education approaches focus on helping PSTs master subject-related knowledge, pedagogical content knowledge, direct teaching skills, and individual characteristics (Jurow et al., 2012) rather than developing knowledge and competencies related to collaborative inquiry and knowledge building (Yang, Chen et al., 2020; Hong, Lin, Chai, et al., 2019).

For PSTs to be able to lead students in collaborative inquiry in the future, we propose that it is important for them to proactively engage in metacognitive monitoring, reflection, and regulation of their own collaborative inquiry (Järvelä & Hadwin, 2013; Yang, Chen et al., 2020) and that it is vital for them to experience collaborative inquiry themselves. Research has suggested that one productive way to address this challenge is to engage teachers—we use the term to refer to both in-service teachers and pre-service teachers (PSTs) unless otherwise specified—in the culture and practices of collaborative inquiry (Dobber & Van Oers, 2015; Hong, Lin, Chai, et al., 2019; Wells, 2011). Teachers practicing collaborative learning and knowledge building provide the opportunities to systemically experience, practice, and reflect on their collaborative inquiry, through which they can improve their understanding and competence (Ellis & Castle, 2010; Van der Linden et al., 2012). In addition, such opportunities may help them design innovative and engaging learning experiences for their own students (Snell & Lefstein, 2018).

This study used the Knowledge Building framework and approach, which is an influential approach in which learners work in a community to advance their collective knowledge and develop collaborative inquiry and knowledge building competencies (Scardamalia & Bereiter, 2006). To support PSTs’ collaborative inquiry and knowledge building, this study included design of an environment for them that was enriched by collaborative analytics-supported reflective assessment. Research has suggested that reflective assessment effectively promotes learners’ engagement in their reflection and regulation of inquiry, collaboration, and knowledge building (Yang et al., 2016; Yang, van Aalst et al., 2020; White & Fredericksen, 1998), which are critical for successful collaborative inquiry and knowledge building (De Backer et al., 2015, 2021; Lei & Chan, 2018). Furthermore, recent advances in CSCL and learning analytics in supporting student online interactions (Wise et al., 2021) indicate its rich potential to promote student reflection and collaborative interactions. Relevant CSCL analytics technologies make it possible to track learners’ collaborations for continuous analysis (Chen & Zhang, 2016) and provide them with learner-generated feedback to help them reflect on and improve their inquiry and knowledge building (Wise et al., 2021). In relation to the current study, the feedback provided can help PSTs better understand and monitor their collaborative inquiry status and processes, and thus regulate their collaborative inquiry and knowledge building.

In this study, the PSTs’ reflective assessment of their knowledge building inquiry was aided by visualization and data generated by the Knowledge Building Discourse Explorer (KBDeX), a social network analysis tool designed to visualize and represent collaborative ideas and discourse (Oshima et al., 2012). Consistent with knowledge building theory’s emphasis on epistemic agency, this approach allowed the PSTs to assume active roles and use the learner-generated data to reflect metacognitively and revise their collaborative inquiry. The study was designed to have both an experimental class and a comparison class. Knowledge building in the experimental class was enriched by collaborative analytics-supported reflective assessment, whereas knowledge building in the comparison class was supported by portfolio-based reflective assessment. The study was designed this way to examine the role that collaborative analytics-supported design may play in advancing PSTs’ competencies in collaborative inquiry and community knowledge.

Conceptual Framework and Literature Review

Collaborative inquiry and CSCL

Since the 1990s, the field of CSCL has grown quickly. It has aimed primarily at examining how students, supported by computer technologies, can work together for joint meaning making and engagement in collaborative inquiry and problem solving (Stahl, 2006). The seminal paper in CSCL by Roschelle (1992) examined how student dyads engaged in processes of collaboration, supported by computer tools, to solve physics problems. Research in science education has also closely examined collaborative inquiry and how it can be scaffolded. In this domain, collaborative inquiry has been found to be an approach that values students’ interests and curiosity, stimulates their active learning, and helps them to conduct scientific investigations (Braund & Driver, 2005; Rocard et al., 2007). Collaborative inquiry knowledge involves a “body of knowledge that represents current understanding of natural systems” (National Research Council, 2007, p. 26). There are no commonly accepted definitions of collaborative inquiry learning or its relevant procedures. However, by comparing the processes of inquiry learning as characterized by science education experts, Bell et al. (2010) synthesized nine “main inquiry processes” common to these approaches. The nine processes are as follows: orientation/question, hypothesis generation, planning, investigation, analysis/interpretation, model, conclusion/evaluation, communication, and prediction.

Forms of collaborative inquiry learning differ in terms of their degree of concreteness, their degree of specificity, and the ways that student activities are structured. Structured collaborative inquiry can be seen to be at one end of the spectrum, because it involves highly scripted procedures, whereas open inquiry and principle-based practice can be seen to be at the other end (Zhang et al., 2011). In structured collaborative inquiry, students pass through specific, pre-defined pathways as they undertake activities. For instance, WISE (web-based inquiry science environment) and knowledge integration provide structured scaffolds (Slotta & Linn, 2009) and suggested pathways for different learning activities to closely guide students’ scientific inquiry. A comparatively less structured inquiry model is that developed by White and Frederiksen (1998), who defined an inquiry cycle composed of the steps “question—predict—experiment—model—apply” for students’ reflective assessment activities. At the other end of the spectrum, open inquiry models give learners more freedom, allowing them to define their inquiry processes based on their emerging needs, for example, the need to identify gaps in understanding or the need to adjust plans. One such model is knowledge building, which is a principle-based collaborative inquiry approach in CSCL that only defines core inquiry principles and values, allowing teachers to engage in reflective interpretation and make adaptive classroom-based decisions to accommodate students’ emerging needs and learning contexts (Scardamalia, 2002; Zhang et al., 2011). This study used the knowledge building approach to support the development of PSTs’ collaboration and knowledge building competencies.

Collaborative inquiry competencies generally involve three dimensions: epistemic, social (Van Uum et al., 2016), and metacognitive aspects (Hong, Lin, Chai, et al., 2019; Yang, Chen et al., 2020). The epistemic dimension refers to people’s beliefs about what scientific knowledge is and their understanding of how scientific knowledge is generated, including inquiry procedures such as generating research questions, conducting investigations, and drawing conclusions (Furtak et al., 2012). The social dimension is associated with collaboration processes, involving sharing and communicating inquiry findings, receiving constructive feedback (Van der Rijst, 2009), constructing and maintaining shared understanding, negotiation/coordination, and maintaining team function (Andrews-Todd & Forsyth, 2020; Sun et al., 2020). The metacognitive dimension involves monitoring, reflecting on, and self-directing the progress and outcomes of the inquiry (Jarvela & Hadwin, 2013; Yang, Chen et al., 2020). To examine PSTs’ competencies in collaborative inquiry such as collaboration, inquiry, and knowledge building, this study examined the epistemic engagement, collaborative interactions, and metacognition/metadiscourse manifested in PSTs’ knowledge building discourse on the Knowledge Forum.

Knowledge building as a collaborative inquiry model

Knowledge building, an educational model for triggering processes of innovation, exhibits many similar characteristics with collaborative inquiry but with a distinctive focus on members creating and adding value to the community (Scardamalia & Bereiter, 2014). Knowledge building represents a way to reassess education and “initiate students into a knowledge creation culture” (Scardamalia & Bereiter, 2006, p. 97). It places students’ authentic questions and real ideas at the center of their collaborative inquiry for knowledge advances (Scardamalia, 2002). Knowledge building theory emphasizes members contributing and extending the frontiers of their community’s knowledge through collective cognitive responsibility. Two key ideas include community members taking agency to advance their community knowledge and using design-mode thinking to continually and collectively pursue idea improvement.

In knowledge building classrooms, students engage in online and classroom discourse inquiring into authentic problems collectively—they generate questions, put forth ideas and theories, construct explanations, build on others’ ideas, and continually revise their theories (Zhang et al., 2007). Knowledge building involves students in collectively taking on social, epistemic and metacognitive responsibilities to progressively improve their community knowledge. Students not only engage in collaborative problem-solving and ideation, they work together interacting and advancing ideas together; they take up metacognitive roles collectively in setting goals and monitoring and regulating the community’s inquiries and knowledge building; and they view knowledge as improvable ideas for knowledge advancement (Yang, Chen et al., 2020; Yang, van Aalst et al., 2020; Zhang et al., 2018).

To support these high-level processes and discourse, Knowledge Forum, a networked software environment designed according to knowledge building theory and principles, is used to support high-level knowledge advancement (Scardamalia & Bereiter 2014). Supported by the Knowledge Forum, learners work as a community posting a diversity of ideas and questions in multiple Knowledge Forum views as they collectively advance their understanding. By engaging in metacognitive actions using meta-discourse, students can create a high-level overview of what they have or have not investigated and the direction they should take. This conceptualization of collaborative inquiry and ideas ensures that learners’ discussions are not superficial or fragmented and can be improved upon, and that conceptual progress can be made. Knowledge building is complex and learners require appropriate pedagogical and technological support to engage in these metacognitive actions. Recent research in knowledge-building examines meta-discourse (Resendes et al., 2015) and meta-cognitive conversations (Zhang et al., 2018; Tao & Zhang, 2021) and involves the use of Knowledge Forum analytics to facilitate students’ collaborative reflections and discourse on their knowledge-building inquiry for rise-above ideas.

Knowledge building has been implemented in different subjects from K-12 to higher education around the world (Chen & Hong, 2016). Substantial progress has been made over the past 30 years in terms of learning designs and scaffolding strategies for sustained collaborative inquiry (e.g., Chen & Hong, 2016; Scardamalia & Bereiter, 2014; van Aalst, 2013). The effectiveness of knowledge building in supporting students’ open collaborative inquiry rather than structured scientific inquiry (e.g., teachers still decide the research questions) has been well documented (e.g., Zhang et al., 2011; Zhang et al., 2018). Knowledge building is a principle-based approach and consists of 12 interconnected principles (Scardamalia, 2002). Among these principles, reflective and transformative assessment, epistemic agency, community knowledge, rise-above, and improvable ideas (Chen & Hong, 2016; Scardamalia, 2002) inform the current designs for PSTs development of collaborative understanding and competence. In particular, we examine reflective and transformative assessment enriched with the use of collaborative learning analytics.

Designing collaborative learning analytics and reflective assessment

It has been a vision of CSCL to dynamically support productive collaboration through automated analyses (Wise et al., 2021). As such, the field of collaborative learning analytics has emerged. In this field, learning analytics generates results from the analysis of learners’ collaborative learning, which are then treated as inputs and feedback to improve the quality of collaboration (Wise et al., 2021). Collaborative learning analytics thus provides concurrent feedback to students based on the analysis of student-generated data so they can adapt their learning and collaboration strategies (Clow, 2012; Wise et al., 2021). One tension within collaborative learning analytics is the need to balance technology and human agency. As Scardamalia et al. (1989) put it, “the computer environment should not be providing the knowledge and intelligence to guide learning, it should be providing the facilitating structure and tools that enable students to make maximum use of their own intelligence and knowledge” (p. 54). Learning analytics, including collaborative learning analytics, should be used to support the agency of learners rather than supplanting it (Chen & Zhang, 2016; Kitto et al., 2018).

With the advent of computational technology, increased research attention has been paid to the design of learning analytics for CSCL classrooms, such as real-time dashboards that help illuminate and orchestrate students’ inquiry (e.g., Martinez-Maldonado, 2019; van Leeuwen et al., 2019). Although these studies have demonstrated many benefits of learning analytics, they have focused on the teachers’ use of analytics tools to support student inquiry. In line with the knowledge building theory of using technologies to enable students to maximize their agency (Scardamalia, 2002), this study focused on students’ own use of collaborative learning analytics, facilitated by reflective assessment embedded in the collaborative inquiry and knowledge building processes.

This study designed collaborative analytics-supported reflective assessment in a knowledge building environment, combining the advantages of both using learning analytics tools and supporting learners’ agency. Specifically, the design included reflective assessment and the provision of visualization and feedback generated by analytics, which students could use to analyze and reflect on their own online collaborative writing and inquiry. Reflective assessment was first developed in the context of scientific inquiry, for the purpose of metacognition (White & Fredericksen, 1998), and it was later adapted by researchers studying knowledge building. Through reflective assessment, students use their collective agency to set inquiry goals, monitor personal and community progress, use feedback to identify knowledge gaps, and examine how to improve their ongoing inquiry and address broader problems (Lee et al., 2006; Scardamalia, 2002; Yang, van Aalst et al., 2020; Yang et al., 2016). Reflective assessment aligns with the knowledge building principle of “transformative assessment,” an extension of formative assessment, which has three features: (1) it is embedded, as assessment is an integral part of learning and itself is learning; (2) it entails a concurrent and transparent inquiry process, and the process and outcome data made explicit by analytical tools improve students’ awareness of their current state and understanding of where should they go; and (3) it is transformative, as it guides students to monitor and reflect on their inquiry and learning, and transforms their inquiry and learning processes (Scardamalia, 2002; van Aalst, 2013; Yang, van Aalst et al., 2020). Reflective and transformative assessment supported by analytics is important in CSCL given the complexity of understanding learners’ collaborative learning process from cognitive, socially interactive, and metacognitive perspectives.

Research in knowledge building analytics has shown that even school-aged children can use some forms of analytics data to advance their knowledge building (Chen, 2017; Resendes et al., 2015; Zhang et al., 2018). Our prior research in knowledge building included the use of portfolio-supported reflective assessment guided by principles as standards to help students reflect on, assess, and improve their Knowledge Forum writing (Lee et al., 2006; Lei & Chan, 2018). Our further work examined reflective assessment using analytics, which encouraged students to collectively increase their metacognitive awareness, apply and develop their metacognitive competencies, and further direct their ongoing inquiry and learning by participating in spiral metacognitive cycles of task analysis supported by a learning analytics design (Yang, Chen, et al., 2020; Yang et al., 2016). Secondary school students were provided with an analytics and visualization tool called the Knowledge Connection Analyzer to analyze and reflect on their Knowledge Forum inquiry, and they made progressive improvements in subsequent Knowledge Forum writing (Yang, van Aalst et al., 2020; Yang et al., 2016). These studies have shown that reflective assessment aided by analytics can help students develop higher-order competencies, such as agency, knowledge creation, and inquiry, and can improve their academic performance.

In this study, we extended our earlier work and designed a collaborative-analytics approach that also focused on developing PSTs’ agency, by having them use the learning analytics data from the Knowledge Forum to foster their collaborative knowledge building inquiry. The collaborative analytics-supported reflective assessment was designed using (a) the visualization of students’ Knowledge Forum collaborative work based on KBDeX (Oshima et al., 2012), (b) knowledge building thread rubrics for measuring the quality of collective knowledge building, knowledge advancement and collaboration, and (c) scaffolds and prompt questions depicting a metacognition cycle to support students’ reflection. Over the course of the study, the PSTs were shown the KBDeX visualization, knowledge building interaction thread rubrics, and prompt questions, all with the aim of guiding their collective reflection to help them adapt their ongoing knowledge building.

KBDeX was developed by Oshima and colleagues (2012). Its primary purpose is to analyze online discourse data to depict community advances, and it is used mostly by researchers. We developed the KBDeX output as visualization diagrams, which students could use as feedback on their ongoing Knowledge Forum discussions. The knowledge building interaction/rubrics were developed based on knowledge building principles and adapted from the work of Borge et al. (2018) and van Aalst (2009). The scaffold questions on the prompt sheets depicted a metacognitive cycle to facilitate reflection and analysis. The purpose of the analytics visualization, rubric, and reflective prompt questions was to help PSTs monitor the state of and problems related to their inquiry threads and interactions and to reflect on how to better engage in productive interactions. They were to be used to support students’ collaborative analysis of and reflection on community ideas and their identification of promising inquiry directions.

CSCL and knowledge building for PSTs’ development

It is important to support students to become collaborative learners. In relation to this, over the past two decades, substantial advances have been made in CSCL pedagogical approaches and technologies; these have included the use of learning analytics to facilitate student collaboration (Stahl, 2015; Wise et al., 2021). Teachers play a key role in CSCL, as they bear the main responsibility for orchestrating the successful use of CSCL pedagogies and technologies in classrooms; however, some gaps remain in CSCL research on teacher learning (Wise & Schwarz, 2017). Thus far, CSCL research has focused on the professional development of in-service teachers and on teacher communities (Chan, 2011; Yoon et al., 2020); limited attention has been paid to teacher learning for PSTs. In addition, there has been little effort in teacher education focusing on scaffolding PSTs’ metacognition, collaborative inquiry, and knowledge building using a CSCL lens.

Apart from collaborative learning, inquiry-based learning is another set of approaches that teachers find tricky. Primarily, teachers find collaborative inquiry and inquiry-based pedagogies “complex and multifaceted, difficult to sum up in any process, practice, or structure” (Jennings & Mills, 2009, p. 1612). A review of the role of the teacher in inquiry-based teaching in K–12 classrooms has suggested that in-service teachers find it difficult to implement inquiry-based education; it has also been found that in-service teachers are mainly viewed as the executors of interventions designed by researchers rather than agents who themselves develop inquiry-based approaches in the classroom (Dobber et al., 2017). Teachers find that implementing open inquiry, in which learners are allowed to formulate authentic questions and conduct independent investigations (Windschitl, 2003), even more difficult. For instance, Leonard et al. (2011) found that even after implementing interventions to enhance PSTs’ inquiry-based instruction (e.g., courses, workshops, practicums), some PSTs still led inquiry that was relatively structured rather than open. Open inquiry requires teachers to help students engage in student-centered and contextually authentic scientific inquiry (Buxton, 2006; Seiler, 2001).

The premise of this study is that teacher education can be more productive if PSTs themselves experience the pedagogical approaches that they are encouraged to use in their future teaching. It is important to develop CSCL designs in which PSTs can engage productively in collaboration, inquiry, and knowledge creation. Similarly, given the importance of inquiry-based education in preparing students to become agentive inquirers, Dobber et al. (2017) proposed that teacher education programs should provide opportunities for PSTs to experience inquiry-based education to prepare them to foster inquiry-based classrooms. We thus propose that it is important to help PSTs develop their understanding of and competencies in collaborative inquiry by involving them in the very process of collaborative inquiry. Such is conducive to their own professional development, as they may become key players in their students’ collaborative learning.

Knowledge building is an open inquiry approach that places students’ real ideas/authentic questions at the center of collaborative inquiry (Scardamalia, 2002; Scardamalia & Bereiter, 2014). Whereas CSCL approaches have generally investigated collaborative learning among tertiary students, knowledge building researchers have conducted investigations into PSTs’ collaborative learning. For example, Laferrière and colleagues examined the promotion of knowledge building discourse among PSTs using Knowledge Forum and a hybrid learning environment (Allaire et al., 2011). Hong et al. (2016) asked PSTs to solve technology problems and found that the extent to which the PSTs engaged in productive knowledge building was positively related to their development of epistemic views. The PSTs who engaged in more productive knowledge building tended to develop more sophisticated epistemic views, believing that ideas have a social and public presence and can be created, exchanged, investigated, and improved by humans (Hong et al., 2016; Popper, 1972). Hong et al. (2019) concluded that principle-based knowledge building design was effective in supporting PSTs to work as a cohesive community, with positive impacts on their design knowledge of STEM projects. Although these studies focused on examining the knowledge building approach generally, they provide a useful background for designing, more specifically, CSCL environments for PSTs. This study builds on previous research that examined how engaging PSTs in collaborative analytics-supported reflective assessment in knowledge building, with an emphasis on student agency, would influence their competencies in collaborative inquiry for advancing community knowledge.

Informed by the literature discussed above, we propose that knowledge building enhanced by collaborative analytics-supported reflective assessment has the potential to help PSTs develop their competencies in collaborative inquiry and knowledge building (see Fig. 1 for the conceptual model). This conceptual and design framework is premised on knowledge building principles emphasizing epistemic agency, idea improvement, community knowledge and transformative assessment. PSTs are meant to take epistemic agency to foster idea improvement within the community using transformative assessment focusing on metacognitive reflection. Based on these principles, two key design components include knowledge building inquiry and collaborative-analytics reflective assessment. First, PSTs will engage in sustained idea-driven inquiry for knowledge advancement situated in writing contributed within the Knowledge Forum. This writing takes the form of posing questions, co-constructing explanations, building on peers’ ideas and revising theories. Second, PSTs will be provided with collaborative analytics using KBDeX visualizations generated from Knowledge Forum writing, together with knowledge building interaction rubrics and prompt questions to support their metacognitive reflection. PSTs would use the analytics to reflect and to assess their ongoing Knowledge forum work for deepening inquiry and further revision.

Fig. 1

 A conceptual design framework enriched by collaborative analytics-supported reflective assessment for PSTs’ collaborative competence and knowledge building

Collaborative learning analytics have the potential to facilitate learners’ reflective assessment concerning their collaborative inquiry. Specifically, we examined PSTs’ collaborative competence and knowledge advancement as reflected through Knowledge Forum discourse moves. PSTs’ collaborative competencies in collaborative inquiry and knowledge building include the epistemic, social, and metacognitive aspects; coherence in community knowledge; and connection of KB actions and KB thread quality. As part of a systematic program that examines the design, impacts, processes, and dynamics of analytics-supported reflective assessment, this study examined the impacts of reflective assessment facilitated by collaborative learning analytics on promoting PSTs to develop their competencies in collaborative inquiry and knowledge building. The research questions were as follows:

• 1) RQ1: What characterizes PSTs’ Knowledge Forum discourse moves reflecting competencies in collaborative knowledge building inquiry? And do PSTs who participated in the knowledge building design enhanced by collaborative analytics-supported reflective assessment (the experimental class) perform better than those in the regular knowledge building design (the comparison class)?

• 2) RQ2: Do PSTs’ discourse moves reflecting collaborative inquiry competencies and knowledge building change differently in the experimental and comparison classes? If so, how does using collaborative analytics-supported reflective assessment influence PSTs’ ongoing collaborative knowledge-building inquiry in the experimental class?

Method

Research contexts and participants

This study was conducted at a teacher training university located in central China. The participants enrolled in a compulsory core liberal studies course entitled Scientific Inquiry and Knowledge Creation. The course lasted 18 weeks, with 1.5 h of classroom time (two consecutive lessons) each week. The course aimed to help PSTs experience the journey of scientific research, co-direct their scientific inquiry, and progressively create knowledge as they develop higher-order competencies through collaborative inquiry. This study adopted a quasi-experimental design. The experimental group was a class of 40 PSTs who used a knowledge building design enhanced by collaborative analytics-supported reflective assessment. The comparison group was another class of 28 PSTs who experienced a conventional knowledge building environment that used portfolio assessment.

Table 1 presents the demographic information of the PSTs. The two classes were similar in terms of gender composition. The majority of the PSTs in the two classes were in their third year. The PSTs came from 10 faculties. The experimental class had more students majoring in Arts and P.E. than those in the comparison class. The PSTs had learned a lot of subject-based and pedagogical knowledge; however, they did not have any teaching experience. Generally, the PSTs were going to be required to carry out educational internships for half a year in real K–12 schools in their fourth year. In both classes, the PSTs worked in groups of five to seven (Mean = 6.18, SD = 1.08). In both classes, each small group worked on an inquiry topic concerned with the question of how to collectively engage in scientific inquiry and knowledge creation; overall, the groups in the two classes covered similar topics. Based on our classroom observation, at the initial stage of the course, most of the PSTs had limited knowledge of and ability in collaborative inquiry and knowledge building. The course instructor, who had a doctoral degree in learning sciences, had been using the knowledge building pedagogical model and reflective assessment in her teaching for two years prior to the study.

Table 1 Details of demographic information of participating PSTs

Pedagogical and technological design

Designing the knowledge building environment augmented by collaborative analytics-supported reflective assessment (intervention)

Five knowledge building principles, namely reflective and transformative assessment, rise-above, epistemic agency, improvable ideas, and community knowledge, were highlighted in the experimental class. The instructor applied a three-component design with principle-based activities (Table 2) to encourage the PSTs to engage in productive collaborative knowledge-building inquiry. The design was modified based on our previous study (Yang, Du et al., 2020).

Component 1—Developing a collaborative and co-directed learning and inquiry culture for productive collaborative knowledge-building inquiry (Week 1–5). The students were involved in a series of principle-based activities to gradually develop their metacognitive and collaborative awareness and the competencies required to engage in productive knowledge building inquiry. For example, the student groups (five to seven students) were required to create a poster about how to stay motivated and persistent in pursuing learning and scientific inquiry and were encouraged to conduct small-group and whole-class discussions and activities. They were also asked to sign a learning contract, consisting of agenda items to undertake in the course, to monitor their own learning, and to write weekly reflections. Scaffolds such as “what I have and have not learned about the theme” were provided to support their reflective writing. Additionally, each student group was required to generate a research plan for the inquiry topics they chose in Week 3 before working on Knowledge Forum, to help them direct their inquiry.

Component 2—Initiating problem-centered inquiry and fostering productive knowledge building inquiry on Knowledge Forum (Week 4–8). As the discussion continued on Knowledge Forum (Fig. 2), the PSTs contributed various inquiry questions and ideas. However, several discussion threads were fragmented, superficial, and incoherent. Thus, the instructor introduced the knowledge building interaction rubric (Table S1) in Week 7 to help the PSTs monitor and reflect on their knowledge-building inquiry and engage in productive interactions. The instructor first organized a structured discussion to help the PSTs understand the rubric and how they could use it to reflectively assess their discussion threads. The discussion focused on “What characterizes productive knowledge building interactions?” and “How to differentiate knowledge building interaction patterns with varying levels?” This was followed by a demonstration of how to use the interaction rubric. Next, the groups reflectively analyzed and reviewed their discussion threads on their inquiry themes of interest, identified the two best discussion threads for each knowledge building interaction pattern, and rated the threads with evidence and justifications.

Fig. 2

Screenshots of a view (top) and a note (bottom left) in Knowledge Forum

Component 3—Advancing competencies in collaborative knowledge-building inquiry through collaborative analytics-supported reflective assessment (Week 9–18). From Week 9 to Week18, PSTs’ quantitative participation and collaboration log data from the Knowledge Forum (e.g., number of notes written, read, scaffolds used) were provided to them each week to help them reflect. Specifically, collaborative-analytics were used to support reflective assessment to deepen collaborative inquiry.

In Week 9, the instructor introduced the KBDeX visualizations (Fig. 3) and data regarding frequencies of identified ideas (keywords) to encourage the PSTs to engage in productive collaborative inquiry and knowledge building. Figure 4 shows the design and scaffolding process of reflective assessment using KBDeX. The instructor first ran KBDeX and showed the PSTs how visualizations and data on knowledge building inquiry were generated. Next, the instructor briefly explained some key terms in social network analysis (e.g., density, degree centrality), the identification of ideas, and the key parameters in KBDeX visualizations (e.g., the cumulative degree centrality of ideas). Finally, the instructor explained how to read the KBDeX visualizations and data. She facilitated a whole-class discussion on how to interpret the growth curve of the cumulative degree centrality of ideas and an idea network map and how to use them for their reflection and assessment of ongoing discourse and further inquiry. For example, the students could first analyze the ideas at the center of the idea network map and, through this analysis, identify the focus of their discussions, and then analyze the ideas far from the center to assess how and further discussion of these ideas are needed. Cumulative degree centrality derived from KBDeX has been used to represent collective knowledge advancement (Oshima et al., 2017; Oshima et al., 2018). A higher cumulative degree centrality means a denser social network. For example, an increased cumulative degree centrality for note network over time denotes how students collectively worked on key ideas (Oshima et al., 2012). The student groups then reflectively analyzed both the KBDeX visualizations and data and their online knowledge building inquiry discourse. Guided by the KBDeX metacognitive prompt questions (see Table S2 in the supplementary file), they reflected on what they have done and also identified issues and gaps, and generated specific plans for further inquiry. They then created “rise-above” notes on Knowledge Forum, guided by the Knowledge Forum metacognitive scaffolds (e.g., Our problems in the discussion, Our action plan for deepening inquiry). In Week 13, the PSTs conducted the second round of KBDeX-supported reflective assessment on their knowledge building inquiry created between Week 9 and Week 13.

Fig. 3

Sample KBDeX visualizations; growth curve of the cumulative degree centrality of ideas (left) and the idea network map (right) generated by KBDeX

Fig. 4

How PSTs were scaffold to engage in reflective assessment process supported by collaborative analytics of KBDeX

Instruction in the comparison class

The PSTs in the comparison class investigated the same inquiry topics and were facilitated by the same instructor as in the experimental class. In component 1, the PSTs in the comparison and experimental classes were involved in the same tasks and activities at the same pace. In components 2 and 3, the comparison class was required to perform portfolio-supported reflective assessment using the knowledge building principles (van Aalst & Chan, 2007) instead of reflective assessment using the knowledge building interaction rubric and KBDeX visualizations and data. The portfolio assessments also involve reflective assessment but it is not supported with analytics—students reviewed their Knowledge Forum writing and identified their good discussion guided by principles as they wrote portfolio notes. The two classes were otherwise involved in the same activities and tasks.

Table 2 Knowledge building design enhanced by self-directed reflective assessment for pre-service teachers’ collaborative inquiry

Data collection and analysis

Table 3 Focus of research, specific questions, data source and analyses, and expected outcomes

Table 3 shows focus of research, specific questions, data source and analyses, and expected outcomes of the study. We elaborated the data sources and analysis for the three research questions addressed in this study.

PSTs’ competencies in collaborative knowledge-building inquiry: manifested by Knowledge Forum discourse moves

The primary data source for examining PSTs’ competencies in collaborative knowledge-building inquiry (RQ1) was 1,406 Knowledge Forum notes from the experimental class and 707 Knowledge Forum notes from the comparison class. We argue that PSTs’ development of competencies in collaborative inquiry and knowledge building could be manifested by their progressive knowledge building discourse through which they collaboratively improve community knowledge. Their discourse can demonstrate PSTs’ epistemic engagement, collaboration competencies for productive knowledge advances, and metacognitive competencies (e.g., identification of gaps, task analysis, regulation of collective inquiry), matching to the epistemic, social, and metacognitive dimensions of the Knowledge Building principles.

First, we parsed the Knowledge Forum notes into inquiry threads for subsequent content analysis and to comprehensively understand the PSTs’ knowledge building inquiry. An inquiry thread is a sequence of notes that address the same problem (Zhang et al., 2007). We identified 30 inquiry threads in the experimental class and 19 in the comparison class. A second researcher independently analyzed the inquiry threads in 527 notes (> 30%) from the experimental class. The two researchers obtained inter-coder reliability of 0.83 (Cohen’s kappa).

Next, with each inquiry thread as the unit of analysis, we conducted content analysis to qualitatively trace the characteristics of PSTs’ collaborative inquiry competencies using a coding framework (see Table S3 in the supplementary file). This coding framework was developed and refined based on our previous studies of discourse characteristics, including the types of epistemic engagement, collaborative interactions, and metacognition and metadiscourse (Yang, 2019; Yang, van Aalst et al., 2020). The first author of this paper and another researcher with expertise in coding online discourse independently coded the above 527 notes, and obtained the following inter-rater reliability (Cohen’s kappa) scores: 0.83 for epistemic dispositions, 0.85 for collaborative interactions, and 0.84 for metadiscourse. The rest of the notes were coded by the first author.

Finally, we conducted three comparisons between the experiment and comparison classes to understand the differences in PSTs’ collaborative inquiry competencies in the two classes; and the first two comparisons were based on the above content analysis results. First, we compared the proportion of Knowledge Forum notes with higher-level discourse moves (e.g., engaging in problem-centered uptake of ideas, reflecting on and deepening inquiry) between the experimental and comparison classes. Second, we compared the characteristics of the epistemic networks of knowledge building actions between the experimental and comparison classes. Third, we examined the collective knowledge improvement of the two classes using the cumulative degree centrality of their note networks.

In examining the differences in epistemic network characteristics, we conducted epistemic network analysis (ENA; Shaffer, 2017). ENA is a set of techniques to identify and calculate the connections among coded elements and visualize them in dynamic network models. The models illustrate the structure and strength of the connections by quantifying the co-occurrence of codes within a defined segment of data. This study analyzed four epistemic actions and three metacognitive actions. The four epistemic actions were: alleviating the lack of knowledge (AleLK), negotiating a match between diverse ideas (NegAF), engaging in problem-centered uptake of ideas (EngPU), and synthesis and rise-above of collective ideas (SynRI). The three metacognitive actions were: creating meta-cognitive awareness (CreMA), conducting a major review of collective ideas (ConMR), and reflecting on and deepening collective ideas (RefDI). We compared the characteristics of the epistemic networks of the experimental and comparison groups.

To characterize the advancement in collective knowledge in knowledge building, cumulative degree centrality (TDC) data derived from KBDeX (Oshima et al., 2012) were used. The state of collective knowledge can be reflected by a network of ideas created by students (Oshima et al., 2012). The increasing number of meaningful links between ideas can then be used as a measure of community knowledge advancement. Degree centrality in social network analysis indicates the cumulative lengths of the paths by which each node is linked to other nodes in the network; hence, a higher cumulative degree centrality represents a denser social network. For instance, an increase in cumulative degree centrality for a note network over time indicates how the PSTs collectively worked on key ideas. Therefore, cumulative degree centrality from KBDeX has become a common tool to assess collective knowledge improvement (Oshima et al., 2017; Oshima et al., 2018). Based on this assumption, we compared the increase in the cumulative degree centrality of both classes to assess their differences in community knowledge advancement.

PSTs’ development of competencies in collaborative knowledge building inquiry, and how reflective assessment support such development.

We further examined the influence of collaborative analytics-supported reflective assessment by exploring the differences in change in the experimental and comparison PSTs’ collaborative inquiry competencies and knowledge building, and how such development was supported by collaborative analytics-supported reflective assessment (RQ2).

To examine the differences in the development of the PSTs’ collaborative inquiry competencies over time, we first conducted a comparative analysis of the characteristics of the experimental and comparison PSTs’ higher-level discourse moves before (stage 1) and during (stage 2) reflective assessment based on the above content analysis results. Then we compared the characteristics of the epistemic networks between the two stages of the two classes.

Furthermore, to illustrate how reflective assessment using KBDeX helped the PSTs engage in productive collaborative knowledge building inquiry, PSTs’ responses in reflective prompt sheets (see Appendix) were collected and analyzed. The prompt sheet included a metacognitive cycle consisting of ‘Our analysis’, ‘Our Reflection’ and ‘Our regulation’, and a set of specific question prompts to guide students to analyze the KBDeX data and their knowledge building inquiries, identify gaps and promising inquiries, and regulate their knowledge building inquiries by making further action plan. The metacognitive prompt sheet responses documented the PSTs’ interpretations of the KBDeX data, analysis and reflections of their knowledge building inquiries, and new research plan and measure for deepening their further inquiry. We distributed the prompts in class to help the PSTs engage in productive collaborative inquiry and knowledge building, and collected student reflective responses from these prompt sheets after class.

To reveal the enactment process of productive collaborative inquiry and knowledge building by engaging collaborative analytics-supported reflective assessment, we used qualitative analysis method of critical events that had been validated in learning-sciences research (e.g., Yang, van Aalst et al., 2020; Yang et al., 2016; Sawyer, 2013). In identifying events of reflective assessment that were critical for PSTs’ engagement in collaborative inquiry and KB, we first identified both productive and unproductive uses of KBDeX data, followed by selection of a limited number of events according to knowledge building goals such as rise-above of collective ideas and progressive inquiry. We then analyzed the potential of collaborative analytics-supported reflective assessment to enhance PSTs’ engagement in collaborative inquiry and knowledge building.

Results

PSTs’ competencies in collaborative inquiry in the experimental and comparison classes

To answer RQ1 concerning the characteristics and differences of the two classes’ competencies in collaborative knowledge building inquiry, we first presented the results of characterization of the experimental PSTs’ discourse moves reflecting competencies in collaborative knowledge building inquiry, followed by results of classroom differences in collaborative inquiry competencies.

Characterization of PSTs’ discourse moves reflecting competencies in collaborative inquiry and knowledge building

The PSTs’ Knowledge Forum discourse was analyzed to examine their collaborative inquiry competencies. Table 4 presents the categories of epistemic engagement, collaborative interactions, and metacognitive reflection/metadiscourse with subcategories that reflect collaborative inquiry competencies. The PSTs in the experimental class generated more explanation-seeking questions (82 questions) than fact-seeking questions (5 questions) and contributed more explanations (899 notes) and rise-above notes (102 notes, with a high-level conceptualization of collective ideas) than simple claims (89 notes). These results indicate the PSTs’ involvement in epistemic engagement.

Table 4 also shows that the experimental PSTs appeared to make sustained efforts to collectively deepen collaborative inquiry and improve community ideas, as evidenced by their contribution of 484 notes to address conflicting ideas and to develop a shared understanding by negotiating diverse ideas, 658 notes to adopt collective problem-centered ideas, and 101 notes to synthesize and rise above community ideas. The results suggest that the PSTs in the experimental class engaged in productive collaborative interactions in the knowledge building process.

Additionally, Table 4 shows that the experimental PSTs in the experimental class made sustained efforts to monitor, reflect, and regulate their collaborative inquiry. For example, they asked metacognitive questions and challenged claims made by community members (172 notes) to help their peers regulate their inquiry, conducted major reviews of community ideas and inquiry processes (37 notes), and were involved in productive reflections on the advancement of community ideas (63 notes). These results suggest that the PSTs in the experimental class engaged in higher-level metacognition and metadiscourse.

Overall, the results suggest the experimental PSTs were able to use sophisticated epistemic, collaborative, and metacognitive discourse moves as they engaged in productive KB. They were engaged in deploying higher-level competencies in collaborative inquiry and knowledge building.

Table 4 Characterization of PSTs’ Knowledge Forum discourse moves reflecting competencies in collaborative inquiry and knowledge building in the experimental and comparison class: Frequency of epistemic engagement, collaborative interactions, and metacognition and metadiscourse

Classroom differences in competencies in collaborative inquiry

Classroom differences in PSTs’ higher-level discourse moves (Comparison 1). Table 5, based on results presented in Table 4, compares the higher-level discourse moves of the experimental and comparison classes. The experimental class contributed significantly more higher-level discourse moves than the comparison class, χ² (df = 1, N = 4,582) = 189.73, p < .001. The experimental class generated more explanatory questions that require explanations and more notes in the categories of explanations and rise-above. They engaged more in the uptake of problem-centered ideas and synthesis and rise-above of community ideas. The experimental class was also more likely to collectively reflect on, monitor, and regulate their collective inquiry, suggested by greater percentages of meta-cognitive awareness and reflecting on and deepening collective ideas. The results indicate the positive impacts of analytics-supported reflective assessment design on PSTs’ knowledge building discourse—which indicates their level of collaborative inquiry competency.

Table 5 Class differences in the characteristics of higher-level discourse moves

Classroom differences in the characteristics of epistemic networks (Comparison 2).

Figure 5 displays the plotted points and ENA means of the experimental class and the comparison class, the respective mean networks for PSTs in the experimental and comparison classes, and the subtracted epistemic network of the two classes. The red and blue dots represented the centroids of each PST in the experimental class and the comparison class, respectively. The mean centroids of all dots in each class are shown as squares with a 95% confidence interval for each dimension represented by the rectangular outline. Table 6 shows the connection coefficients of the seven collaborative inquiry actions of the two classes.

Figure 5(a) shows that the mean centroid values of the experimental class fell far from those of the comparison class, suggesting differences between the mean along the x- and y-axis for the two classes. To examine whether the differences were significant, we conducted an independent-samples t-test (assuming unequal variances) of the mean centroid values of the two classes. The results revealed significant differences between the experimental class (M = 0.24, SD = 0.53) and the comparison class (M = -0.34, SD = 1.07) along the x-axis, t (36.42) = -2.65, p < .05, d = 0.73; and significant differences between the experimental class (M = 0.43, SD = 0.45) and the comparison class (M = -0.62, SD = 0.48) along the y-axis, t (56.04) = 9.04, p < .05, d = 2.25. We further explored what knowledge building action (codes) connections contributed to these differences by examining the mean networks of the experimental and comparison classes. As shown in Fig. 5(b), Fig. 5(c) and Table 6, we found that compared with the PSTs in the comparison class, the PSTs in the experimental class made more connections within and between higher-level epistemic actions and metacognitive actions. For example, the connections between engaging in problem-centered uptake of ideas (EngPU) and synthesizing and rising above notes (SynRI), and between EngPU and reflecting on and deepening collective inquiry (RefDI) were stronger in the experimental class than in the comparison class.

Finally, we constructed a subtracted network (see Fig. 5(d)) to further examine salient connections that contributed to the differences between the experimental and comparison classes. The subtracted network was obtained by subtracting the mean connection strengths for the PSTs in the comparison class from the mean connection strengths for PSTs in the experimental class. Darker and thicker lines indicate greater differences in connection strength. The red lines and blue lines indicate connections that were stronger in the experimental class’s and comparison class’s network, respectively. Figure 5(d) shows a higher number of connections made by the experimental PSTs to the nodes RefDI, CreMA (creating metacognitive awareness), and EngPU, suggesting that links between the high-level metacognitive actions RefDI and CreMA and other epistemic actions, and links between the higher-level epistemic action EngPU and SynRI are prominent features of the experimental class. In contrast, the comparison PSTs made more connections to the node AleLK (Alleviating lack of knowledge, the lower-level epistemic action) and ConMR (conducting major review of collective ideas, the higher-level metacognitive action). Overall, these results suggested that that collaborative analytics-supported reflective assessment helped the experimental PSTs establish stronger connections between higher-level epistemic actions and metacognitive actions and among different higher-level epistemic actions, whcih were conducive to PSTs’ collaborative inquiry and knowledge building.

Fig. 5

ENA means, mean networks of collaborative inquiry actions for the experimental class (read) and comparison class (blue), and subtracted network for the experimental class (red) and comparison class (blue)

Table 6 Connection coefficients of the experimental class and the comparison class

Classroom differences in community knowledge advancement (Comparison 3). Figure 6 shows the increasing cumulative degree centrality of the note network of the experimental and comparison classes over time, indicating how the PSTs advanced their community knowledge. Both classes showed improvement, but the experimental class changed more rapidly in phases 2 and 3 after introducing the analytics-supported reflective assessment tools. In phase 1, a one-way analysis of variance (ANOVA) showed no significant difference in community knowledge advancement between the two classes, F(1, 8) = 0.03, p = .874, η2 = 0.00. In contrast, significant differences were observed after the reflective assessment intervention was implemented in the experimental class in phase 2, F(1, 12) = 14.40, p = .003, η2 = 0.55, and in phase 3, F(1, 4) = 367.13, p < .001, η2 = 0.99. These results indicate that compared with the PSTs in the comparison class, with the help of collaborative analytics-supported reflective assessment, those in the experimental class showed a greater level of community knowledge advancement.

Fig. 6

Cumulative degree centrality of the note network for both classes over time

Note

Figure 6 was created based on data derived from KBDeX, and this comparison data was only used in the research evaluation context. W = week; Week 1 started from the beginning of note writing on Knowledge Forum.

PSTs’ changes in collaborative inquiry competencies and engagement in collaborative analytics-supported reflective assessment

To answer RQ2, we first presented two comparison results to reveal differences between the experimental and comparison classes in terms of PSTs’ level of competence in collaborative inquiry and knowledge building. Then we described qualitative analysis results to illustrate how collaborative analytics-supported reflective assessment contributed to the differences.

Changes in competencies in collaborative inquiry and knowledge building in the experimental and comparison classes over time.

Comparison of higher-level discourse moves over phases. Table 7 compares PSTs’ discourse moves in stage 1 and stage 2 in the experimental and comparison classes. Overall, there was an increase in the occurrence of notes coded as higher-level discourse moves over the two stages, except that the PSTs asked fewer explanation-seeking questions in the two classes. However, the net increase of higher-level discourse moves of the experimental class is greater by an exceptionally high margine than that of the comparison class. These results suggest that the experimental PSTs’ development in competencies in collaborative inquiry and knowledge building shows steeper improvement over time than the comparison PSTs, thus supporting the positive effect of collaborative analytics-supported reflective assessment.

We further conducted a chi-square test and found significant differences in the frequencies of higher-level discourse moves between the two stages in the experimental class, χ² (df = 1, N = 4,218) = 108.38, p < .001. Specifically, the experimental class wrote more notes with explanations and synthesis and rise-above ideas that reflected the PSTs’ community knowledge advancement. The class engaged more in collaborative interactions for the uptake of collective ideas and the synthesis and rise-above of collective inquiry and ideas. Furthermore, the PSTs contributed more notes in the metadiscourse category, which suggests their actions to monitor, reflect on, and regulate collective inquiry in the later stage. These results indicate that within the experimental class, knowledge building design enriched by analytics-supported reflective assessment gradually improved the PSTs’ collective inquiry competencies.

We then conducted another chi-square test to examine the possible difference in frequency distribution of higher-level discourse moves between Stage 1 and Stage 2 in the comparison class, and found that significant differences existed between the two stages, χ² (df = 1, N = 1,450) = 15.20, p < .001. These results suggest that the PSTs in the comparison class also demonstrated higher-level competencies in collaborative inquiry and knowledge building in the latter stage than in the early phase, indicating the impacts of portfolio-supported reflective assessment exist as well.

Table 7 Changes of high-level discourse moves between stage 1 and stage 2 in the experimental and comparison classes

Table 8 Connection coefficients of Stage 1 and Stage 2 in the experimental and comparison classes

Comparison of epistemic network characteristics over phases. Figure 7; Table 8 show the comparisons of the epistemic network characteristics between stage 1 and stage 2 of the experimental and comparison classes. Figure 7(a) suggests salient differences between the mean centroid values for the two stages of the experimental class. T-test results (assuming unequal variances) indicated a significant difference between the mean centroid values for Stage 1 (M = 0.70, SD = 0.53) and Stage 2 (M = -0.70, SD = 0.66) along the x-axis, t (74.94) = 10.48, p < .05, d = 2.34, and a significant difference between Stage 1(M = 0.30, SD = 0.91) and Stage 2 (M = -0.30, SD = 0.60) along the y-axis, t (67.64) = -3.46, p < .05, d = 0.77. These results suggest that Stage 1 and Stage 2 had different connection patterns. Figure 7(b) further shows the most salient connections that contributed to the differences between the two stages by presenting a subtracted network of Stage 1 and Stage 2. The subtracted network was constructed by subtracting the mean connection strengths for Stage 1 from the mean connection strengths for Stage 2. Results from Fig. 7(b) and Table 8 show that in Stage 2, the PSTs in the experimental class made more connections to the nodes of RefDI and SynRI, suggesting that links exist between higher-level metacognitive actions and epistemic actions. In contrast, in Stage 1, the PSTs made a higher number of connections to the nodes of EngPU (engaging in problem-centered uptake of ideas) and CreMA (creating metacognitive awareness), suggesting salient links between higher-level and lower-level epistemic actions, and between metacognitive actions and different levels of epistemic actions. Overall, these results suggest that the PSTs in the experimental condition made stronger and more complex connections between and within higher-level epistemic and metacognitive actions in the later stage where the intervention of collaborative analytics-supported reflective assessment was introduced. The differences between the two stages demonstrates the positive effect of collaborative analytics-supported reflective assessment on PSTs’ collaborative inquiry actions.

As Fig. 7(c), 7(d) and Table 8 show, similar to the experimental class, PTSs in the comparison class contributed differently in Stage 1 and Stage 2. Figure 7(c) shows salient differences between the mean centroid values for the two stages of the comparison class. The t-test (assuming unequal variances) indicated a significant difference between Stage 1 (M = -0.30, SD = 1.09) and Stage 2 (M = 0.31, SD = 0.93) along the x-axis, t (52.14) = 2.25, p < .05, d = 0.61, and a significant difference between Stage 1 (M = -0.35, SD = 0.59) and Stage 2 (M = 0.36, SD = 0.90) along the y-axis, t (44.61) = -3.42, p < .05, d = 0.93. Figure 7(d) further shows salient connections contributing to the significant difference between the two stages. Stage 1 had more connections between epistemic actions, such as AleLK- EngPU and AleLK- NegAF (negotiating a match between diverse ideas), and a relatively higher number of connections between epistemic and metacognitive actions and within higher-level epistemic actions. These results suggest that portfolio-supported reflective assessment positively affected the comparison PSTs’ collaborative actions. However, the effect of portfolio-supported reflective assessment on comparisons PSTs’ collaborative inquiry actions is not as large as that of collaborative analytics-supported reflective assessment on the experimental PSTs’ collaborative inquiry actions.

Fig. 7

ENA means and subtracted networks of stage 1 and stage 2 of the experimental and comparison classes

PSTs’ engagement in collaborative analytics-supported reflective assessment

The following two examples illustrate PSTs’ analysis and reflection on their Knowledge Forum writing with the help of the KBDeX visualizations and their action plans for deepening knowledge building inquiry (and creating knowledge). These excerpts provide additional support illuminating how KBDeX supports reflective assessment and collaborative inquiry.

Collaborative analytics-supported reflective assessment fostered PSTs’ idea synthesis. The following excerpt illustrates how PSTs used analytics-generated data to drive their collaboration and knowledge building–the enactment process included the synthesizing of ideas through shared monitoring and regulation of their collective inquiry supported by reflective assessment using KBDeX visualization:

We want to get an overview of what we are inquiring into at different phases, what we have discussed, our inquiry process, and where should we head, to further refine our inquiry framework and identify promising inquiry directions …[note: setting collaborative inquiry goals].

With the help of the idea network map [after analysis of and discussion on the idea network map from KBDeX], we find that our discussion on critical reasoning and thinking focuses on four dimensions: the definition, characteristics, and components of critical thinking; factors influencing the development of critical thinking; methods for developing critical thinking; and educational practices for supporting students’ development of critical thinking. We have extensively discussed the first three dimensions: the degree centrality of these keywords (e.g., judgment, analysis, questioning, strategies) are high, and the connections between them are strong. [note: PSTs using analytics data from idea network map and degree centrality to reflect on collaborative inquiry and knowledge building]

We have also contributed some innovative ideas, such as the connections between personalized learning and critical thinking, and the relationship between metacognition and critical thinking …[note: identify what they have accomplished].

Our discussion on educational practice is relatively weak, with only a few keywords on the periphery, but this part is very important, and we will focus on it in the further inquiry…Additionally, we will investigate theories and practice regarding the assessment of critical thinking and how to improve PSTs’ critical thinking in China. There are a few keywords, but these inquiry issues are quite promising. [note: use KBDeX keywords on periphery, to help identify gaps for future inquiry]

In this example, one student group focusing on critical thinking engaged in the analysis of and reflection on their Knowledge Forum discussion with the idea network map generated by KBDeX to “get an overview of what they are inquiring into at different phases…and identify promising inquiry directions.” This intention indicates that the PSTs realized the importance of reviewing and synthesizing collective ideas and identifying promising directions to deepen inquiry. Using the idea network map, the PSTs constructed a high-level picture of the dimensions discussed, the present state of each dimension, and the highlights of innovation ideas (“our discussion on critical thinking focuses on four dimensions”; “we have extensively discussed the first three dimensions. . the connections between them are strong”; and “we have also contributed some innovative ideas. . metacognition and critical thinking”). Through thorough analysis, the -PSTs also identified areas for further inquiry: “educational practice” and “theories and practice regarding the assessment of critical thinking and how to improve PSTs’ critical thinking in China”. These results suggest that the -PSTs engaged in synthesizing and rising above collective ideas by conducting a KBDeX-supported reflective assessment.

Collaborative analytics-supported reflective assessment helps PSTs engage in higher-level conceptualization of collective ideas. The following excerpt demonstrates how the PSTs engaged in higher-level conceptualization of collective ideas through collaborative analysis, reflection and regulation of their collective inquiry by performing KBDeX-supported reflective assessment:

[After analyzing the idea network map from KBDeX produced by their own group] Our discussion focused on four aspects (Fig. 8), but it seems superficial, particularly concerning reasoning and logical fallacy. [note: reflecting on current state using KBDeX]

We found eight ideas (e.g., imagination, fallacy, confidence, character cultivation) on the periphery [KBdeX], and we think that the two topics of logical reasoning and character cultivation and fallacy really need further inquiry, because… [note: Identify specific areas that need inquiry].

[After analyzing the idea network map produced by the whole class, consisting of six groups] ideas such as critical thinking, scientific thinking, evidence, and conjecture are closely related to our inquiry regarding critical reasoning, and really inspire us. Thus, we will borrow some ideas from their discussion to extend and deepen our inquiry; for example, procedures for developing scientific thinking and how to train and develop critical thinking. We plan to use the following framework [developed based on the integration of ideas from the whole class] to deepen our inquiry (Fig. 8)…[note: synthesizes ideas from other groups for rise-above].

Fig. 8

The process of generating higher-level conceptualization of collective inquiry facilitated by KBDeX-supported reflective assessment

As outlined in this excerpt, the PSTs collectively reflected on their inquiry process with the help of the KBDeX visualizations, summarized what they had discussed (“focused on four aspects”), and identified problems (“[our discussion] seems superficial, particularly concerning critical reasoning and logical fallacy”). It was encouraging to see that the PSTs were able to develop an overview of their inquiry and identify the challenges they faced, which was critical to triggering a collective regulation process.

With the help of the idea network map, they also identified potentially promising routes for further inquiry (“we think that the two topics of critical reasoning and character cultivation and fallacy really need further inquiry”). They further productively regulated their inquiry by generating a research framework for deepening their inquiry (“We plan to use the following framework to deepen our inquiry”). This research framework was generated with the help of the idea network map generated by the whole class (“Ideas such as critical thinking…are closely related to our inquiry…and really inspire us”). These results suggest that KBDeX-supported reflective assessment can help PSTs to engage in contributing higher-level conceptualization of collective inquiry.

Discussion and conclusions

In light of global changes and changing educational demands, helping teachers and students to develop their higher-order competencies in collaborative inquiry for creative knowledge work is an important goal but also a challenging task. CSCL designs and technologies and recent advances in learning analytics have much potential, but their use has been limited in teacher education. In this study, we designed a knowledge building environment augmented by collaborative analytics-supported reflective assessment to help PSTs engage in collaborative inquiry and knowledge building. We examined the impact of collaborative analytics-supported reflective assessment by analyzing PSTs’ competency related to collaborative inquiry and knowledge building, including an analysis of the processes by which PSTs engaged in productive collaborative inquiry and knowledge building. To investigate how they used collaborative analytics-supported reflective assessment to set goals, reflect, synthesize, and refine their collaborative inquiry and knowledge building, we also examined their reflections recorded on reflective prompt sheets and discussions. In the following sections, we first summarize the evidence supporting the role of the designed environment for promoting PSTs’ collaboration and knowledge building. The implications of this study for the CSCL field are then discussed; they include enriching collaborative learning analytics through maximizing student agency and reflective assessment and broadening CSCL and knowledge building approaches to different contexts including teacher education.

Engagement in and impacts of collaborative analytics-supported reflective assessment on competencies in collaborative inquiry and knowledge building.

This study examined PSTs’ competence in collaborative inquiry and knowledge building reflected in Knowledge Forum discourse moves. The analysis showed that PSTs demonstrated collaborative inquiry competencies involving different aspects of epistemic engagement, social interaction, and metacognitive reflection. Analysis of knowledge forum discourse moves showed that PSTs who exhibited high-level collaborative inquiry competence contributed thoughtful questions, ideas with explanations, and rise-above ideas to elevate their inquiry and ideas to a higher level of conceptualization. They engaged in productive and collaborative interactions by alleviating the community’s lack of knowledge, addressing conflicting ideas among peers, negotiating a match between different ideas, engaging in problem-centered uptake of ideas, and synthesizing and formulating an in-depth conceptualization of community ideas. They engaged in collective metacognition and regulation by creating and supporting social meta-cognitive awareness among peers, monitoring and reflecting on the inquiry process and ideas, and generating action plans to deepen their inquiry. These high-level collaborative competencies are associated with students’ knowledge building advances.

The comparative analysis of the frequencies of Knowledge Forum discourse moves showed that the PSTs in the experimental class exhibited a significantly higher level of collaborative inquiry competencies than those in the comparison class. As well, the epistemic networks of knowledge building actions revealed that the experimental class exhibited significantly stronger connections between higher-level epistemic actions and metacognitive actions, and within higher-level epistemic actions. These findings suggest that the knowledge-building design enhanced by collaborative analytics-supported reflective assessment has the potential to scaffold PSTs to develop higher-order collaborative inquiry competencies. These results provide further empirical evidence of the positive effects of open inquiry on the development of PSTs’ higher-order collaborative inquiry competencies (Dobber et al., 2017; Hong, Lin, Chai, et al., 2019; Hong, Lin, Chen, et al., 2019; Leonard et al., 2011).

Social network analysis illustrates network connectivity and coherence relating to community knowledge advance, and has been considered as an index of knowledge building (Oshima et al., 2020). SNA analysis of the PSTs’ Knowledge Forum discourse showed that the PSTs in the experimental and comparison groups gradually advanced their collective knowledge over time, but the experimental class exhibited a significantly higher level of collective knowledge advancement than the comparison class. These results indicate that the knowledge building design enhanced by analytics-supported reflective assessment has positive impacts on PSTs’ collective knowledge advancement. These results are consistent with those of previous research, suggesting the positive impacts of reflective assessment on PSTs’ advancement of collective knowledge in knowledge building (Yang, Chen et al., 2020).

The comparative analysis results of the frequency distributions of Knowledge Forum discourse characterizing higher-level collaborative inquiry competence and epistemic network characteristics showed that the PSTs in the experimental class demonstrated a higher level of collaborative inquiry competence and had a strong and more complex epistemic network of higher-level knowledge building actions at the later stage than at the early stage. The analysis of the PSTs’ responses in prompt sheets revealed that conducting analytics-supported reflective assessment promoted their engagement in productive collaborative inquiry by engaging them in ongoing analysis, negotiation, synthesis of knowledge building inquiry, identification of promising directions, and development of action plans. These results suggest that conducting collaborative analytics-supported reflective assessment enables PSTs to apply their metacognitive competencies, including gap analysis, continuous reflection on collective knowledge building inquiry, and action planning for deepening inquiry. All of these competencies are essential and conducive to engaging PSTs in productive collaborative inquiry.

These findings, taken together, suggest that collaborative analytics-supported reflective assessment has great potential to scaffold PSTs to develop higher-order collaborative inquiry competencies in knowledge building. These findings extend previous studies that used portfolio assessment to scaffold productive knowledge building discourse (Lee et al., 2006; Lei & Chan, 2018; van Aalst & Chan, 2007) and content-related and metacognitive principle-based prompts to promote structured guidance (Yang, Du et al., 2020). The findings are consistent with studies indicating the positive effects of reflective restructuring (Zhang et al., 2018) and reflective assessment (White, & Frederiksen, 1998). These findings also provide insights into the relationships between assessment, metacognition, collaborative inquiry, and instructional practices.

CSCL collaborative learning analytics and reflective assessment

This study was situated in the knowledge building research tradition, and within that context, we enriched knowledge building designs with collaborative learning analytics. CSCL learning analytics has emerged as a key theme in the expectations both at present and into the future for CSCL research, in accordance with previously noted rapid advancements in this area (Wise & Schwartz, 2017; Wise et al., 2021). Learning analytics in CSCL has two promising purposes. First, it may aid in computational/automatic analysis by revealing diverse and hidden aspects of students’ online collaboration patterns and processes. Second, it may help to provide ongoing feedback based on the automatic analysis of students’ online activities, thereby helping to scaffold reflection for improved collaboration. Despite increasing positive evidence of the role of learning analytics (Martinez-Maldonado, 2019; van Leeuwen et al., 2019), much of the past research focuses on teachers using learning analytics tools to monitor student learning rather than having students use analytics visualizations to reflect on and to chart their ongoing inquiry. Specifying best practices for the use of collaborative learning analytics, including such issues as providing the appropriate levels of representation and feedback for students, continues to be an area of challenge and development (Wise & Schwarz, 2017).

The design of this study was based on the Knowledge Building framework, which emphasizes students’ collective epistemic agency, idea improvement, and reflective-transformative assessment. This study contributes to research on CSCL in engaging students directly in using collaborative learning analytics, rather than just having teachers use learning analytics to guide/monitor student learning. Building on our earlier research on reflective assessment using knowledge building principles and portfolio notes (Lee et al., 2006; Lei & Chan, 2018;) and more recent research using a knowledge building analytics tool called Knowledge Connection Analyzer (Yang, van Aalst et al., 2020; Yang et al., 2016), we harnessed recent advances in CSCL learning analytics and designed an enriched knowledge building environment using collaborative learning analytics, emphasizing student agency and collective inquiry.

Specifically, this study provided the PSTs with visualizations of their knowledge building inquiry using KBDeX output to support their reflective assessment. We also provided them with reflective prompt questions and knowledge building rubrics to facilitate their productive reflective assessment. The prompt sheets consisted of a set of metacognitive prompt questions including observation, analysis, reflection, problem analysis, and planning for further inquiry (see Appendix). As the qualitative excerpts show, when the PSTs were provided with the KBDeX diagrams visualizing the semantic networks of keywords and the links between them, and when they were guided by metacognitive prompt questions, they engaged in goal setting, identifying what they hoped to find out about their Knowledge Forum inquiry, traced what they had accomplished, synthesized ongoing collective work, and identified areas (i.e., educational practices) that needed further discussion in subsequent Knowledge Forum online inquiry. Based on knowledge building principles, the PSTs engaged in open inquiry and dialogue aimed at improving collective ideas. Using the analytics data as an object of collective inquiry to support their reflection, the PSTs may have progressively internalized the metacognitive cycle, including the reflective task analysis and synthesis of ideas, the identification of strengths and promising inquiry directions, and actions to deepen their collective inquiry.

Reflective assessment for promoting collaboration and knowledge building is effective as it helps students to metacognitively reflect on the status of their ongoing work in relation to scientific and epistemic standards and principles (van Aaslt & Chan, 2007; White & Frederickson, 1998). The comparative findings in this study showed that the two classes, which both used knowledge building and some form of reflective assessment, improved in terms of their collaborative competence and advanced their knowledge. These findings echo earlier research (Lei & Chan, 2018; van Aaslt & Chan, 2007), but the use of collaborative analytics has additional advantages. With the addition of CSCL analytics and student-generated data, reflective assessment can be further enhanced by learner-generated data as evidence of collaborative inquiry processes and outcomes. These learner-generated data and the analytics output provide feedback to help learners become aware of the status of their work and inform their ongoing collaboration. They also become shared objects of collaborative inquiry as students ponder over the meanings and implications of these data for their continuing pursuit of inquiry.

This study highlights and enriches the conceptual basis of collaborative analytics premised on knowledge building theory and focused on student collective agency and transformative assessment. Knowledge building theory advocates designing technology to maximize students’ agency rather than merely supporting their online collaborative activity (Scardamalia, 2002). This study is consistent with that principle: it highlights the need to explore and develop collaborative analytics embedded within pedagogical approaches and tools, which encourage PSTs/learners to use analytics data and representations and to be responsible for their own ongoing collaborative inquiry. The study demonstrates how it is possible to design collaborative analytics in CSCL classrooms combining the use of analytics-based visualization tools (KBDeX) and assessment rubrics of knowledge building and metacognitive questions to support student collaborative inquiry and knowledge building. This study contributes to the theme of CSCL collaborative learning analytics and highlights students’ collective agency in using learning analytics grounded in the theoretical basis of knowledge building and reflective assessment.

CSCL and knowledge building for PSTs’ collaborative inquiry and teacher learning

The establishment of connections between CSCL and educational practice has been identified as a goal of CSCL research. CSCL researchers increasingly recognize -attention needs to be paid to teacher learning as they are responsible for enacting CSCL pedagogies and technologies in classrooms (Wise & Schwarz, 2017). This study contributes to recent CSCL research examining teacher learning in the context of PSTs’ engagement in collaborative inquiry and knowledge building. Although university students have been frequent subjects of study in CSCL research, few investigations have focused specifically on PSTs, or students preparing to become teachers; where CSCL research has focused on teacher learning, it has focused mostly on ISTs’ professional development (e.g., Chan, 2011; Yoon et al., 2020) This study is one of the few studies to examine PSTs’ learning in CSCL contexts and demonstrating how collaborative analytics supported reflective assessment could foster student agency, collaborative inquiry and knowledge building.

It is now recognized that students need to develop new competencies such as collaboration, inquiry, and problem solving using technology. However, teachers often find it difficult to allow students the agency to direct their own learning and inquiry (Modrek & Sandoval, 2020). It is known that teachers have difficulty giving students the amount of autonomy they desire over their own learning (Nakata, 2011), and so it is important that providing students with opportunities for autonomy similarly be a focus of teacher training (Kuhn, Modrek, Sandoval, 2020). PSTs in particular are influenced by their prior schooling experience (often transmission-based), which contrasts with contemporary views of learning and collaboration. To meet changing educational needs, in addition to mastering disciplinary knowledge and pedagogical skills, PSTs need to grapple with tensions between theory and practice as well as epistemological shifts (Hong et al., 2016). They also need to develop deeper views about learning and collaboration. Nevertheless, how new teachers can be prepared in these areas has not been systematically examined using CSCL perspectives. This study was thus motivated by the need to examine and support PSTs’ learning, incorporating CSCL and knowledge building perspectives. We held the position that it would be beneficial for PSTs to experience the same pedagogical approaches that they are encouraged to use in their future teaching. By experiencing open collaborative inquiry themselves, PSTs would be more likely to develop the knowledge of and competence in collaborative inquiry and knowledge building that would lead to epistemological change and the continued pursuit of inquiry (Hong et al., 2016).

The design of this study took Knowledge Building as the theoretical CSCL foundation and framework, to enhance PSTs’ learning. Knowledge Building is one of the most open inquiry models in CSCL; it emphasizes idea improvement, epistemic agency, and collective cognitive responsibility (Scardamalia & Bereiter, 2014). The culture and norms of knowledge building value learners’ collective agency. They also emphasize the collective improvement of ideas. In addition, student agency and metacognition are central to the knowledge building model. Based on these ideas, this study focused on the scaffolding of reflective and transformative assessment and collective agency, supported by collaborative analytics. The PSTs were provided with the opportunity to work together, taking on collective cognitive, social, and metacognitive responsibilities to advance community knowledge. The PSTs worked together as a community, pursuing deeper understanding and experiencing collaborative inquiry and knowledge advancement. The knowledge building design enhanced by collaborative analytics-supported reflective assessment made explicit PSTs’ collective decision-making, agency, metacognition, and self-directedness in a collective and agentic knowledge building process. Consistent with previous research on learning designs for empowering PSTs (Hong Lin, & Chai, 2019; Hong, Lin, Chen, et al., 2019), these design features were enriched using collaborative analytics. The design has important implications for creating technology-enhanced designs to develop PSTs’ metacognition, agency, and collaborative inquiry competencies.

This study also has implications for incorporating CSCL and knowledge building in teacher education. In view of changes in society, technological development, and emerging educational goals, there is an increased need for CSCL to help people work together, dialogue with each other, and pursue collaborative inquiry using technology (Wegerif, 2013). Unfortunately, CSCL perspectives have not been incorporated much into teacher education (Wise & Schwarz, 2017). PSTs’ courses focus primarily on disciplinary knowledge and pedagogical skills; psychology courses likewise often focus on individual mental processes, neglecting the social construction of knowledge and the mediating role of technology (Jurow et al., 2010). This study was conducted in the context of a liberal studies course for undergraduate PSTs on scientific thinking and knowledge creation, and corresponding collaborative inquiry approaches were used. With the increased use of technology-enhanced collaborative and group learning in schools, it is important for PSTs to experience collaborative inquiry as one focus of their learning for their growth as learners and teachers.

Teacher education programs can be a useful venue for developing new kinds of understanding and competencies among PSTs who are both learners and teachers. CSCL is suitable as a theoretical foundation that is aligned with emerging educational goals. Teacher education courses for PSTs need to go beyond subject knowledge and pedagogical skills; broader views of learning involving the social construction of knowledge need to be developed (Jurow et al., 2010), and teaching and learning need to be viewed as knowledge building endeavors (Scardamalia & Bereiter, 2014; Zhang et al., 2011). The findings of this study demonstrate that it is beneficial to help PSTs enrich their collaborative competencies and community knowledge work by engaging them in the actual process of collaborative inquiry and knowledge building, enriched with reflective assessment and analytics. Although PSTs’ future teaching practice is not a focus of this study, it is important to examine how PSTs develop their understanding pertaining to broader views of learning and collaboration, and enrich their competencies of collaboration and knowledge building. Although this was a small-scale study involving PSTs, it contributes to CSCL research and practice by providing initial models and examples on which future studies incorporating CSCL in teacher education can be developed.

The study design can be useful for teachers and researchers aiming to create technology-enhanced designs to develop learners’ metacognition, agency, and collaborative inquiry competencies supported by analytics for knowledge building. It offers several insights. First, it shows that the development of epistemic engagement, productive collaborative interactions, and metacognition requires the development of a collaborative and self-directed culture, as well as practices that emphasize collaborative effort and agency for knowledge advancement. By engaging in the culture and practices of collaborative inquiry, learners can gradually develop both self-directedness and collective agency to co-construct knowledge, actualize and develop collaborative inquiry and metacognitive skills, and collectively set goals, monitor, reflect on, and direct their collaborative inquiry. More specifically, the knowledge building rubrics developed for coding Knowledge Forum writing—including different dimensions of epistemic engagement, collaborative interactions, metacognition and metadiscourse—can be adapted as scaffolds to help learners become more productively involved in collaborative inquiry and knowledge building.

Second, it is necessary to engage learners in collaborative dialogues and meta-talk about their inquiry, which will help them explicitly understand good discourse and inquiry, engage in productive reflection, and undertake action plans, all of which contribute to self-directed collective inquiry. These dialogues and meta-talk (i.e., metacognitive conversations that allow them to reflect on their progress) are critical for PSTs’ development of collaborative inquiry competencies, and this talk can be focused on collaborative analytics as objects of inquiry. Helping students to articulate their ideas and engage in dialogue with them will help them make their ideas public and facilitate metacognition.

Third, collaborative learning analytics and technology-assisted assessment have developed rapidly and can provide opportunities for concurrent and embedded feedback for learners. We advocate the use of collaborative analytics enriched with reflective assessment. Rather than having analytics used merely by researchers and teachers, we propose that learners themselves use analytics and visualizations as shared objects of inquiry and reflection for further action. Learners can also be provided with collaborative inquiry/knowledge building rubrics and metacognitive questions to help them assess their current state of work and refine their collaborative inquiry. These technologies can be used to design engaging and meaningful activities that help them use this feedback in a self-directed and collaborative ways and afford them more collective agency in assessing their work.

Limitations and implications for future research

This study has some limitations. First, this study provides empirical evidence of the effect of collaborative analytics-supported reflective assessment in helping PSTs develop their competencies, but it is unclear whether the changes are transferable to their understanding of collaborative inquiry and future teaching. The competencies PSTs developed individually and collectively supported in the knowledge building environment could be further examined as to how they can be transferred to different contexts. Spefically, it would have been helpful to examine PST practice but in teacher education research with PSTs, due to the contextual constraints, many focus on students’ understanding of certain phenomena and strategies (for example, studies on improving students’ conceptions of nature of science) but not necessarily assessing the extent to which PSTs can do better in classroom teaching (e.g., on nature of science) (see e.g., Abd-El-Khalick & Akerson, 2009; Mesci & Schwartz, 2017). Nevertheless, future research could examine whether individuals can transfer the knowledge and competencies (e.g., knowledge creation, metacognition, and self-regulation) obtained through collaborative inquiry practice augmented by analytics- reflective assessment to different contexts and even to their own specific teaching practice.

Second, although relatively rich data sources (e.g., Knowledge Forum notes, reflective prompt sheets) were used, the study did not include students’ artifacts or video recordings of - collaborative-analytics-supported reflective assessment sessions. It is critical to understand why and how analytics-supported reflective assessment works for productive collaborative inquiry; for example, why some groups are more engaged in self-directed reflective assessment and use it for productive collaborative inquiry, whereas others are not. Future research could analyze the social practices that support PSTs’ collaborative inquiry development around analytics-supported reflective assessment and unpack the mechanisms and dynamics by leveraging multiple data sources. The answers to these questions will help us develop adaptive scaffolding to facilitate learners’ productive inquiry and higher-level competencies.

Conclusions

This study examined preservice teachers’ collaborative inquiry and identified the positive effects of the knowledge building design enhanced by analytics-supported reflective assessment on PSTs’ competencies in collaborative inquiry and knowledge building. In the collective inquiry process, the PSTs were encouraged to gradually develop their competencies in collaborative inquiry and knowledge building by engaging in continuous reflective assessment and monitoring and regulating their collaborative inquiry and knowledge creation; to capitalize on the knowledge building interaction rubric for assessing discussion threads; to reflect using analytical data and designed metacognitive prompt questions; to perform other principle-based knowledge building activities. Collaborative analytics-supported reflective assessment enriched with assessment rubrics may effectively scaffold PSTs to develop their metacognitive skills and gradually help them internalize the metacognitive cycle of goal setting, task analysis, gap analysis and identification, and action planning to regulate further collective inquiry and idea improvement. Such development of metacognitive skills and internalization of the metacognitive cycle may help PSTs to gradually and actively self-direct their knowledge building inquiry as they develop collective agency for productive knowledge building inquiry and knowledge creation, thereby developing their collaborative inquiry and knowledge building competencies.

This study provides one of the few examples demonstrating how the knowledge building design can be enhanced by collaborative analytics-supported reflective assessment in helping PSTs gradually develop their collaborative inquiry competencies. The key contributions of this study includes extending current research in CSCL learning analytics shifting from a focus on researchers and teachers using learning analytics (e.g. dashboards) to learners taking up collective agency using analytics and assessment rubrics to advance their collaborative inquiry and knowledge building. As well, this study broadens CSCL research to preservice teacher learning with implications for examining how to incorporate CSCL designs in teacher education. The findings of this study have practical implications for researchers and teachers who devote their effort to creating CSCL technology-enhanced learning environments as metacognitive and epistemic tools to foster learners’ agentic engagement and higher-order competencies such as collaborative inquiry, agency and knowledge creation. This study also makes theoretical contributions as it provides insights into the nature and mechanisms of analytics-supported reflective assessment, and the connections between instructional practice, collaborative inquiry, and reflective assessment.