Introduction

The emergence of computational analysis methods and machine learning techniques provides new ways to understand classroom interactions and design automated systems that can analyze instructor and student classroom behaviors. Classroom analytics can be displayed on dashboards, offering instructors automated observations and feedback regarding classroom interactions and activities (Sedrakyan et al., 2020). Classroom analytics applications are implemented in higher education institutions to enhance student engagement and learning through timely and evidence-based pedagogical actions (Larrabee Sønderlund et al., 2019). With the ability to process and interpret vast amounts of data, these systems can shift the focus from mere information collection to meaningful action, enhancing both teaching practices and student experiences in higher education classrooms.

Classroom analytical systems face a range of challenges that stem from the complexity of educational environments and the diverse needs of stakeholders. One notable challenge is the sheer volume of data generated by these systems, often overwhelming educators, and administrators. Additionally, ensuring data accuracy and integrity can be a daunting task, as discrepancies can lead to misinformed decision-making. The integration of classroom analytics into existing teaching practices and curriculum alignment poses another hurdle, requiring seamless adoption without disrupting established workflows. However, technological advancements are stepping up to address these challenges. The emergence of advanced machine learning algorithms allows for more efficient data processing and pattern recognition, aiding educators in deriving meaningful insights from vast datasets. Moreover, user-friendly visualization tools are being developed, simplifying the interpretation of analytics and making them accessible to educators who may not be data experts. These advancements also facilitate real-time monitoring, enabling prompt intervention when students disengage or require additional support.

As classroom analytical systems continue to evolve, they are becoming increasingly equipped to navigate the challenges of the educational landscape, providing educators with actionable insights to enhance learning outcomes. However, the design of classroom analytics systems is predominantly influenced by technocentric perspectives, constraining their pedagogical implications (Li et al., 2021). User interaction—whether they are learners, instructors, or administrators—with analytics interfaces isn't isolated; rather, intricate interactions among diverse stakeholders underpin the successful implementation of such systems (Larusson & White, 2014). The drive toward constructing human-centered analytics systems, emphasizing user and stakeholder needs, and their dynamic roles in the design process, is gaining momentum (Boy, 2017). It is important for various stakeholders to actively engage throughout the design, implementation, and evaluation phases, aligning practices with needs and expectations while fostering long-term sustainability. A holistic approach is essential to grasp how analytics systems and interfaces can be formulated through these intricate stakeholder interactions.

This paper presents the design and implementation of a professional development system called TEACHActive, which is founded upon computational analysis and automated context-sensitive feedback (AlZoubi et al., 2021a). TEACHActive introduces an innovative approach to displaying classroom behavioral data, which is automatically analyzed through an automated classroom sensing system. The design and implementation of TEACHActive take place within the context of a project funded by the National Science Foundation (NSF). The primary objective of the project was to enhance student engagement and promote active learning within engineering classrooms, utilizing an analytics-driven faculty professional development framework. This paper outlines the implications of stakeholder engagement and mapping methods that emphasize the interconnectedness and relationships among diverse stakeholders, along with their involvement throughout the design and deployment processes.

TEACHActive: Automated Classroom Analytics System

Classroom interactions can be captured by sensors and wearables in physical classrooms, and quantified through multimodal classroom analytics that provide a deeper understanding of embodied learning and aspects of teaching (Martínez-Maldonado et al., 2022a, b). Computer vision advancements using simple cameras can serve as whole-classroom sensors instead of individual student and teacher wearables (Ahuja et al., 2019). Automated classroom observation and feedback system usage has recently gained popularity and attention. A wide range of research on their effectiveness has both clarified the types of feedback they provide and the ways in which they can be integrated into classrooms (Ndukwe & Daniel, 2020). One area of research has focused on automated observation and feedback implementation to provide formative and/or ongoing assessments that helps instructors understand and respond to student learning in real-time (e.g., Ranalli et al., 2017; Tissenbaum et al., 2016). These systems often track student progress and provide feedback to both instructors and students. Another area of research has focused on automated observation and feedback systems that provide personalized student learning experiences (Bernacki et al., 2021). The findings have shown that these systems can lead to improved student achievement and engagement as well as increased teacher efficiency (Avella et al., 2016).

TEACHActive is a comprehensive automated classroom analytics system that supports instructors’ evidence-based decision making and reflective practices in their classrooms. It was designed and developed using a human-centered design approach, whereby TEACHActive system places the needs, preferences, and behaviors of end-users—in this case, instructors—at the forefront of its design process (Norman, 2013; Brown, 2008; Sanders & Stappers, 2008). Rooted in iterative collaboration with stakeholders, TEACHActive aims to create a user-friendly, efficient, and meaningful system that resonates with instructors' experiences and addresses their challenges.

The TEACHActive system builds upon reflective practice, transforming it into a practical tool for instructors' use. Reflective practice serves as a driver for enhancing instructional methods, enabling instructors to pinpoint instructional challenges (Walkington et al., 2001). The effectiveness of reflective practice is magnified when informed by empirical data, facilitating a more nuanced understanding of educational contexts (Avramides et al., 2015; Wise, 2020). Reflective practice operates in iterative cycles, encompassing the identification of pedagogical challenges, the implementation of targeted interventions, and subsequent reflection on their outcomes (Horton-Deutsch & Sherwood, 2017). The TEACHActive system integrates reflective practice with analytics, thus making it a practical tool for instructors. For example, if classroom analytics indicate that an instructor's facilitation is excessively dominant, they can use reflective practice to allocate more time for collaborative activities in future sessions. Similarly, if students appear disengaged, the instructor can introduce more interactive elements to enhance student participation.

TEACHActive transforms raw classroom data into meaningful metrics, utilizing these outcomes to offer practical feedback to instructors aiming to enhance student engagement and active learning through classroom analytics. The TEACHActive system comprises three key components: 1) Pedagogical training and analytics orientation, 2) automated classroom observation, and 3) feedback and reflection. Figure 1 presents the TEACHActive processes and components.

Fig. 1
figure 1

The classroom analytics system processes and components

Full size image

Component 1. Pedagogical Training and Analytics Orientation

The instructors who participate in the automated classroom observation attend a series of group and one-on-one pedagogical training sessions and workshops to understand and explore how they might integrate classroom analytics into their classroom decision making processes. The training sessions also help instructors become familiar with the system’s capabilities and offer them an opportunity to share feedback regarding their needs before implementation.

Component 2. Automated Classroom Observation

TEACHActive deploys EduSense, a computer vision-based classroom system that relies on passive video footage captured by video cameras placed at two vantage points in the physical classroom space. These cameras capture student and teacher activity (video) and spoken communication (audio) (Ahuja et al., 2019). EduSense’s customized built-in classifiers are adapted to the classroom domain and accurately detect instructors’ features of interest. Edusense is a four-layer full-stack platform that integrates low-cost hardware and open-source software, rendering it scalable and modularized. Three of the four Edusense layers were used to implement this system. The first layer (classroom layer) consisted of two cameras, one facing the students and the other facing the instructor, sense classroom dynamics with a non-intrusive approach. The cameras are wirelessly connected to the campus network and are triggered by a digital scheduler that automatically and selectively records the days and times participants taught. Figure 2 presents the deployment picture from an actual classroom.

Fig. 2
figure 2

Classroom camera view (front and back cameras)

Full size image

In addition to the networked cameras, Edusense requires a dedicated server that obtains audio and video signals through a Real Time Streaming Protocol (RTSP). Audio and video inputs are subsequently processed by the second layer (processing layer) of Edusense. This layer divides the video into thousands of frames (images) and differentially identifies student and instructor bodies from inanimate images using trained classifiers (provided by the OpenPose library). The layer also detects key body points (e.g., neck, hips, knees, feet, chest, elbow, shoulder, wrist) to predict body poses (e.g., hand raised, sit, stand) as well as implements speech detection in the audio input to acoustically differentiate silence and speech using a deep learning model. Finally, all data extracted from each video frame is stored in the third layer (datastore layer) using a database (MongoDB) to save the metadata as a JSON object. Thus, each frame has a JSON associated with it that contains its metadata. The fourth layer of Edusense (Apps layer) is intended to be the end-user application (Ahuja et al., 2019). This layer is not available in the project repository and is not used to implement this system; instead, TEACHActive is implemented on top of the three layers to provide an end-user application for instructors.

Component 3. Feedback and Reflection

A frontend TEACHActive dashboard was designed that included three displays: (a) an in-session display that presented classroom analytics from teacher and student behavioral engagement indicators, (b) a progress display that compared the metrics of different sessions (e.g., bar graphs of teacher vs. student speech displayed for multiple sessions), and (c) a reflection display showing reflection prompts that allowed teachers to reflect on the provided metrics and to set goals accordingly. The dashboard links teaching practices to classroom analytics data captured from teacher and student behavioral indicators (AlZoubi et al., 2021a; Baran et al., 2022). These analytics help teachers identify their actual classroom behaviors and encourage them to work toward improved student engagement through those behaviors. Teacher outcome measures include their kinesthetic patterns, changes in class activity such as sitting vs. standing, and spatial data from body positions. Student behavioral engagement is measured by the number of hand raises, and the number of times students vs. the instructor talk, including the duration. The dashboard aims to personalize classroom feedback and enhance teachers’ ability to transform the information presented on the dashboard into actionable pedagogical practice (Mandinach & Gummer, 2016). Figures 3 and 4 present pages and modules from the dashboard.

Fig. 3
figure 3

Dashboard’s welcome page

Full size image
Fig. 4
figure 4

Dashboard’s metrics module

Full size image

Use Case: Implementation of TEACHActive System in Engineering Classrooms

This use case exemplifies the application of the TEACHActive system within a higher education institution. The implementation process employed a multi-phase approach, engaging 15 engineering instructors across diverse sub-disciplines (Chemical & Biological Engineering, Electrical and Computer Engineering, Industrial & Manufacturing Systems, etc.). Initially, all participating instructors attended two one-hour training sessions designed to focus on the pedagogical implementation of active learning strategies. Following these trainings, a series of user walkthroughs acquainted instructors with the system’s interface. During these walkthroughs, instructors were asked about their perceptions of each feature, specifically concerning its anticipated use for their teaching practices. After the user walkthroughs, the TEACHActive system was deployed in each of the 15 engineering classrooms for a four-week period (approximately three weekly sessions, each lasting 50 min).

To augment instructors' reflective practice, they received reflection prompt questions to complete after reviewing their classroom data. The reflection questions were crafted to prompt instructors to incorporate pedagogical changes for future sessions based on the data they received. Post the four-week implementation period, researchers and developers engaged in discussions with instructors regarding their system usage, comprehension of the data, intended pedagogical modifications, and overall experiences with the system. These discussions served as a gauge of the system’s impact on instructors' reflective practices. The outcomes of this multi-phase implementation suggested that the TEACHActive system triggered instructors' reflective practices concerning their in-classroom pedagogical strategies.

Stakeholder Engagement and Mapping

Successful implementation of the TEACHActive depended on continuous interaction and collaboration with and between multiple stakeholders within and outside the university. There is increasing interest in involving users and stakeholders in the design of analytics systems and interfaces (Sanders & Stappers, 2008). A stakeholder is “any group or individual who can affect or is affected by the achievement of the organization’s objectives” (Freeman, 1984, p. 46). Stakeholders possess invaluable knowledge, experience and interaction with the system. Stakeholder engagement among instructors, administrators, and students is an important consideration in classroom analytics system design and implementation (Cober et al., 2015; Dollinger et al., 2019). These systems involve sensors and devices such as cameras to gather data about student behaviors that potentially provide valuable insights into student learning and classroom dynamics (Blikstein & Worsley, 2016). Thus, the goal of classroom analytics is to provide stakeholders with insights into student learning and behavior, and to support evidence-based decision-making. However, the use of automated classroom analytics systems can also raise concerns about privacy, data security, and the potential for biased or inappropriate data usage (Kitto & Knight, 2019).

While stakeholder involvement in the design processes varies, their input is considered critical in making design decisions, particularly when individuals serve in multiple roles (Grimpe et al., 2014). Stakeholder engagement is critical to ensure that a system is adopted by potential users and addresses privacy, security, and ethical concerns leading to more sustainable practices. It is also important to provide ongoing support and communication to stakeholders, so they have the needed resources and information to effectively use and benefit from an automated classroom analytics system.

Three types of TEACHActive stakeholders are identified based on their level of engagement: key, primary, and secondary. Key stakeholders were a group of decision makers directly involved in the design and implementation. These included instructors (users), research team members, UI designers, and software developers. Primary stakeholders were individuals and/or groups who directly impacted the system, including the system administrator (infrastructure/servers/security), audio visual department, classroom scheduling unit, institutional review board (IRB) administrators, and the EduSense Team. Secondary stakeholders were individuals and/or groups indirectly impacted by both key and primary stakeholders’ efforts, including external evaluators, participating faculty’s students and the College of Engineering faculty. Figure 5 represents the stakeholder diagram that maps the three stakeholder types.

Fig. 5
figure 5

The stakeholder mapping

Full size image

Three factors determined successful stakeholder engagement throughout the design, deployment, and evaluation processes: (a) Implementation of human-centered design methods, (b) active and continuous collaboration, and (c) evaluation for impact and sustainability.

Implementation of Human-Centered Design Methods

Human-centered design methods were implemented throughout all system design processes to ensure that key stakeholders (e.g., UI designers, software developers, instructors, and research team) were involved in the design processes. This involved the gathering of user requirements and creating personas, creating mock-ups, conducting user walkthroughs, and making design decisions after reaching user and research team consensus. The research team, user interface (UI) designers, and software developer met periodically and collaborated on key features and elements. User requirements were gathered through semi-structured interviews. Personas were created to understand users’ needs, goals, barriers, frustrations, expected outcomes, and experiences. UI designers created mock-ups, conducted user walkthroughs and interviews with instructors, collaborated on the system’s steps development, and completed a series of user tests (AlZoubi et al., 2021b). Software developers worked on the system’s deployment and maintenance and investigated the possibility of adding new software features based on the research team’s decisions and themes. Researchers met with instructors (users) on a regular basis and involved them in the dashboard design processes, as well as reported themes and identified key points that could be addressed in future iterations.

Active and Continuous Collaboration

Active, continuous, and collaborative engagement with key stakeholders was central to our design approach, including the system administrators (infrastructure/servers/security), Audio-Visual Department, classroom scheduling unit, Institutional Review Board (IRB) administrators, and EduSense Team. Due to the complexity of implementing a system within a higher education institution’s traditional structure with units at various operational levels, establishing frequent sustained interactions with stakeholders was key to improved user experience and effective design decisions (Buchan et al., 2017). The system’s successful application depended on the inclusion and alignment of multiple stakeholders’ perspectives within and across the institution. Achieving support from various institutional units was challenging due to dissimilar goals, operations, and relationships with the project. Including units such as the University IT services in planning conversations, proved to be helpful in addressing concerns and suggestions during the projects’ early developmental phases. Therefore, the team spent considerable time on relationship building at the beginning of the project, as well as aligning goals throughout the system’s implementation.

The system was implemented within the context of a research project involving human subjects. Therefore, IRB administrators constituted one of the primary stakeholders and ensured that the research followed ethical guidelines. Because the system is dependent on collecting and automatically analyzing video data from classrooms, the IRB team and researchers collaborated to protect instructors and students. After meeting the criteria for ethical data collection and protection, we collaborated with the university’s classroom scheduling services and audio-visual unit to identify potential classrooms that fit system criteria and enabled successful camera installation.

Transparency among stakeholders was critical when addressing privacy and ethical concerns. For example, attributes of power (those who decide which design considerations will move forward), ownership (how and with whom data are shared, the consequences of opting out, and how long data are kept), and responsibility (those responsible for data accuracy) should be addressed transparently while designing systems. Working with system administrators and the IT team was crucial to ensure security and privacy. System admins helped set up a data collection server for the system, and a IT security group ensured that all data collected was securely stored on the university network.

EduSense is a licensed (BSD 3- Clause) open-source system that allows source code modification (Ahuja et al., 2019). The system backend depends on EduSense classifiers, requiring a constant open communication channel between the EduSense development team and the on-site team. Both teams met periodically to communicate EduSense deployment updates and to address arising technical issues.

As users of the interface, instructors teaching within different Engineering departments constituted an essential stakeholder group. Their motivation and interest in analytics-driven teaching were key to their participation and use. Recruitment was challenging because the EduSense software limits the number of students who can be accommodated in a classroom and requires a specific classroom geometry. This restricted the number of potential instructors to only those who taught in classrooms that met the systems’ required parameters. While some instructors were interested in participation, their classroom setups did not fit the system’s specifications. To address the complexity of scheduling instructors for specific classrooms in this large institution, the research team collaborated with the classroom scheduling unit early on and had information about instructors’ classrooms for the upcoming semester. The classroom scheduling unit provided lists of instructors and their assigned classrooms. After identifying potential instructors, the research team collaborated with the Audio-Visual Department to classify which classrooms would fit the requirements. A classroom’s capacity should be limited to 60 students and configured in a wide rectangular shape for the system to work well. In some cases, the research team was able to switch instructors’ classrooms and work with the audio-visual team to install cameras in the newly assigned classrooms. In cases where there was no opportunity to exchange a classroom for one having the required features, the team did not move forward with the instructor interested in being a participant.

Evaluation for Impact and Sustainability

Based on earlier activities, all stakeholders were involved in design evaluation and consolidation with stakeholders. This partnership encouraged all stakeholders to provide feedback and share suggestions with useful resources throughout the different phases. External evaluators who were experts in the field (professional faculty development, engineering education, educational technology, and learning analytics) were consulted during the system’s iterations. They reviewed the progress of the project through periodic team meetings and provided feedback throughout implementation. Formative feedback from all stakeholders was incorporated within the various stages, and a formative evaluation assessed the system’s progress and design methods, deployment and implementation. Formative evaluation reports were provided to the team and external evaluators in a timely manner. Accordingly, adjustments and modifications were incorporated at appropriate junctures.

Once the system is successfully developed and implemented throughout its various phases, it will be available for interested instructors to learn about its features and to understand how they can use it in their classrooms. The system can be integrated into other professional development efforts through similar collaborations and engagements among stakeholders.

Challenges of Stakeholder Mapping and Engagement

Following challenges emerged during the stakeholder engagement processes:

  1. 1.

    Identifying stakeholders: Identifying all relevant stakeholders was a challenging task. To address this challenge, the research team thoroughly investigated the context and environment that would house the system prior to designing, deploying and implementing it.

  2. 2.

    Ensuring representation: Even if all relevant stakeholders were identified, it was difficult to make sure they were all represented in the design process and their needs were adequately addressed. The team met periodically with stakeholders to clarify what points could be moved forward and what points were outside the scope of the project, and to communicate them clearly.

  3. 3.

    Building trust and engagement: Building trust and engagement with stakeholders was a challenging task; it required a deep understanding of the stakeholders’ perspectives, needs, and concerns. This could be difficult if stakeholders were skeptical about using automated systems.

  4. 4.

    Addressing privacy concerns: Privacy concerns such as data security and data governance can be a significant challenge when designing automated systems since these systems collect, store and process data that must be protected. Thus, it was crucial to ensure that the system admin was one of the stakeholders and involved in system deployment and implementation.

  5. 5.

    Ethical considerations: Automated systems can have a significant impact on groups; thus, it is important to ensure that these systems are fair and unbiased. In the implementation of such a system, the IRB office was one of the stakeholders that had the responsibility to anticipate and address potential ethical issues.

  6. 6.

    Limited resources: Stakeholder engagement is resource-intensive and requires a budget. It should be recognized that there may be budget limitations for research or staff time needed for engagement activities.

Key Implications and Final Remarks

The TEACHActive system presented in this paper provides examples that include stakeholders in the design, development, and evaluation processes and support sustainable system design in the classroom analytics field. For successful design and implementation of such systems, it is critical to identify stakeholders early in the process and to include them in design decisions. The stakeholder map presented in this paper illustrates groups and individuals who held key, primary, and secondary roles in the project’s implementation. The map can be updated as conditions change, and newly recognized needs emerge. The building of stakeholder maps and identifying stakeholder engagement strategies help designers and developers improve the impact of their work and contribute to sustainable design and implementation practices.

Human-centered design tools as well as stakeholder engagement and mapping are becoming more important in the design of analytics systems and interfaces. The comprehensive analytics system presented in this paper is uniquely situated within the growing literature on automated classroom observation and multi-modal classroom analytics (Blikstein & Worsley, 2016; Martínez-Maldonado et al., 2022a, b). Further system improvement will depend on aligning the goals of researchers, individual faculty users, and institutional units. The project team’s commitment to agile collaboration with stakeholders within the context of institutional structures and decision processes, as well as prolonged engagement with participating faculty to identify factors that influence their classroom actions, will be important. Further commitment to the dashboards’ interface iterations with participatory or co-design practices will improve users’ experience, promote user trust, agency and dashboard ownership.

Finally, integrating TEACHActive with a community of practice (CoP) can present an opportunity to foster collaborative learning, reflection, and continuous improvement of higher education pedagogies. In a CoP, instructors, researchers, and educational technologists can come together to share their experiences, insights, and best practices related to innovative classroom pedagogies and the utilization of classroom analytics. Such a CoP would provide a space for instructors to discuss and dissect the data generated by TEACHActive, collectively exploring its implications on their teaching methods while engaging in reflective practice. It would also facilitate cross-disciplinary dialogue, enabling educators from various sub-disciplines to learn from each other and adapt successful strategies to their own classrooms, all while reflecting on their pedagogical approaches. Moreover, this CoP can serve as a platform for research-driven discussions, allowing stakeholders to delve deeper into the nuances of classroom analytics and its impact on student engagement and learning outcomes, promoting reflective practice at both the individual and collective levels. Ultimately, the TEACHActive-CoP has the potential to create a thriving ecosystem where the system's benefits are maximized through collaborative knowledge sharing, reflective practice, and continuous professional development.