Every step and saying

That he used, I know,

And with sprites obeying

My arts I will show.

  Johann Wolfgang Von Goethe, The Sorcerer’s Apprentice

  Translated by Zeydel (1955)

In Goethe’s classic poem, an apprentice to a sorcerer finds himself unsupervised. Hoping to demonstrate his own abilities, the apprentice recites a spell, enchanting a broom to fetch water to mop the floor. But quickly the apprentice loses control of the broom, which relentlessly pours buckets of water across the hall, ultimately flooding the house. The apprentice knew how to initiate the spell, but he failed to understand the consequences of his actions, and lacked knowledge of how to control the process that he initiated.

With this poem as context, let us reflect on institutional learning analytics initiatives. Prior to the digital transformation of higher education, teaching and learning were observable only by faculty and their enrolled students. But as technologies for teaching and learning proliferated, an apprentice entered the classroom: the institution’s information technology (IT) division. Faculty, as a whole, are not particularly inclined toward online instructional tools, but this apprenticeship was justified by the new and innovative services made possible with technology (Pomerantz & Brooks, 2017). As a result, many IT divisions now find themselves with a bevy of teaching and learning data, presumably reflecting students’ performance and behaviors spanning the learning technology ecosystem. Whereas IT previously maintained services in support of faculty instruction, they now find themselves unsupervised in their access to teaching and learning data and any corresponding opportunities and insights. Hoping to demonstrate prowess for improving student success, IT leadership has pursued learning analytics initiatives largely in the absence of faculty involvement (Yanosky & Arroway, 2015), at some peril (O’Neil, 2016). Of course, not all organizations fit this analogy (case in point, we are academics who held leadership positions in our local IT division), but this is a common pattern, and one that merits caution and reflection.

We are not the first to suggest that institutional learning analytics initiatives share similarities with the literary trope of a creation getting out of control (Prinsloo, 2017). However, a distinguishing feature of The Sorcerer’s Apprentice, and one that is particularly relevant for learning analytics, is that experts exist who know how to act appropriately, and that things get out of control when this expertise is absent. In our analogy to institutional learning analytics, the sorcerer is the community of faculty, and the apprentice is the IT division. The IT division finds itself in possession of new learning data, largely absent from faculty involvement. In Goethe’s poem, the debacle is resolved, all is made well again, when the apprentice implores the sorcerer to return, and this is the thesis of the current commentary: Institutions of higher education should not advance learning analytics initiatives without the leadership of faculty.

Even though learning analytics is still young, we nevertheless affirm that there exist faculty (teachers, researchers, and teacher-researchers) who do, indeed, know what they’re doing with learning data. While they may be absent from IT’s learning analytics initiatives, experienced “sorcerers” are not hard to find at institutions of higher education. For well over 100 years, psychology has been collecting and analyzing learning data in ways that would be right at home, if not exemplary, in any contemporary learning analytics community (e.g., Bryan & Harter 1899). Similarly, generations of education research have yielded specialization in topics related to student motivation, engagement, and assessment – the very firmament of learning analytics. Behavioral economics has decades of expertise modeling behavior and performance in diverse domains and examining the effects of large-scale interventions on real-world outcomes. And more broadly, social scientists (ranging from philosophy to information sciences) can offer insights into societal and organizational reliance on technologies, and the associated challenges (ethical, sociological, professional, and so on). If an institution has the capacity for a local learning analytics initiative, it likely also has faculty in such areas, and thus, it has experts. Obviously, this expertise does not extend to all faculty, nor even to the majority; we merely affirm that experts exist. Furthermore, such expertise is not isolated in any one individual; the fictional sorcerer is an embodiment of multiple faculty. Even if disciplinary experts are unavailable, instructional faculty have first-hand insight into pedagogical practice (and learning technology utilization) that IT leaders typically lack. Together, these faculty should be leading the constructive and responsible utilization of learning data, in an institutional learning ecosystem that privileges collaboration.

This argument is not merely speculative. Even prior to learning analytics, overwhelmingly, sponsorship and participation outside IT were reported as the top factors in the success of institutional analytics initiatives, despite the fact that fewer than one third of such initiatives made use of any external assistance (Goldstein & Katz, 2005). This pattern extends to the current learning analytics wave (Yanosky & Arroway, 2015), where the majority of institutional learning analytics models are top-down and lack consultation from experts or stakeholders (Dawson et al., 2018), and generally fail to find evidence of effectiveness (Larrabee Sønderlund et al., 2019; Macfadyen, 2022). Given this, it should be no surprise that institutional learning analytics initiatives, as a whole, are poorly grounded in theory (e.g., Jivet et al., 2018), and give minimal attention to ethical and privacy issues (Jones,2019; Viberg et al., 2018). Like the apprentice’s half-baked plan to magically mop the floor, institutional learning analytics tend to be characterized by ambitious but uninformed and uncarefully planned initiatives.

Scholarly research, on the other hand, has constraints that reduce the likelihood it would pursue initiatives carelessly or naively. For one, generalizable research is conducted under the explicit approval of an institutional review board, which means that faculty need to articulate a cogent research plan prior to initiating work, anticipate any adverse consequences of one’s research, and place this plan under scrutiny for compliance with basic legal, ethical, and privacy standards. Unreported deviations from this plan or failure to report adverse consequences pose real threats to a scholar’s career, unlike non-research institutional initiatives which are largely unchecked (Willis et al., 2016). Moreover, a common purpose of scholarly research is to answer generalizable, relevant, and/or unanswered questions, whereas there is a clear shortage of such problem-solving efforts in educational technology and learning analytics (Reeves & Lin, 2020; Motz et al.,in press; Wise et al., 2021), and a similar need in institutional research initiatives more broadly (Borden, 2018). And finally, faculty who conduct scholarly research are incentivized to publish their findings in selective journals under the scrutiny of peer review. This means that many faculty are experienced at building strong inferences based on careful empirical analysis that will withstand expert criticism, a challenging sorcery to which institutional IT may have little exposure.

In his defense, the sorcerer’s apprentice meant no harm, nor even mischievousness. After all, the apprentice did not cast a spell for world domination — he tried to mop the floors. Similarly, we want to avoid characterizing institutional IT as malicious in its aims. Rather, because of the digital transformation of higher education, institutional IT serendipitously found itself in a position of privilege and control, and correspondingly, novel responsibility. As highlighted by Wheeler and Hilton (2012), when the essential services of education became technology decisions, responsibility for education fell to IT, “to inform, influence, engage, debate, and adapt their institutions to an increasingly connected world.” In such a world, institutional IT reasonably sees itself doing good leadership work, improving educational services by selecting tools, aggregating data, and building analytical initiatives previously unimaginable when learning technology consisted of a lectern and a chalkboard. Indeed, some view institutional learning analytics initiatives as IT’s moral obligation (Prinsloo & Slade, 2017). It should be acknowledged that IT is, indeed, capable of conjuring some innovative spells. However, this bold let-us-rise-to-the-occasion optimism is also characteristic of the apprentice’s fundamental mistake – in this case, thinking that education could be improved by novice analytical incantations, ignorant of decades of expertise in the academic buildings across campus.

But by making an analogy to the apprentice’s wild brooms, we suggest that novice and uninformed institutional learning analytics initiatives will not merely be ineffective, they will create real problems. To illustrate these issues, consider the now-classic institutional learning analytics implementation where a risk prediction is exposed to students as a traffic signal (Arnold & Pistilli, 2012). Setting aside the miscue (upon being shown a red traffic light, students are being signaled to stop), students shown a risk flag may reasonably interpret this as a judgment of their ability, possibly reinforcing a fixed mindset or suggesting that success is unattainable, both of which negatively impact engagement, belonging, and performance (respectively, Muenks et al., 2020; Canning et al., 2019). Students with disabilities are more likely to be mistargeted for these flags (Riazy et al., 2020), as is also the case with students from less wealthy backgrounds (Yu et al., 2020). Considering that classification and risk prediction represent core aims of learning analytics, it is critically important that institutions appreciate the dangers of the spells they are casting.

Despite substantive practical risks, there are scant examples of learning analytics causing any problems, let alone a flood, in the published literature. Nevertheless, we caution against interpreting the absence of evidence of problems as evidence of their absence. The apprentice, upon being rescued from his catastrophe, did not seek to publish an article about his ordeal, and learning analytics is known to keep their failures to themselves (Clow et al., 2017). Moreover, we are skeptical that an institution would be aware that a flood is occurring in the first place. Assessing impact at an institutional scale is a steep challenge (Macfadyen et al., 2014) rarely even attempted (Ferguson & Clow, 2017). Rather, assessments of institutional initiatives often favor more convenient methods, such correlating tool usage with academic achievement (e.g., Arnold & Pistilli, 2012; Kia et al., 2020), even though this approach is severely inadequate for demonstrating causal impacts (Reinhart et al., 2013). If a high-performing subset of students use an analytical service more than the average student, correlational statistics would fallaciously imply that the service causes improvements in performance, and this correlation would be even stronger if occasional use harms average or low-performing students. Similarly, if an analytical service causes students to withdraw from a course or an institution, and withdrawing also removes these students from analysis, paradoxically this harmful service may appear to improve student performance when benchmarked against comparison groups (attrition bias; e.g., Robinson 2021), as may have been the case with the traffic signal example above (Straumsheim, 2013). In this way, we imagine institutional analytics initiatives not only incanting dangerous spells, but potentially doing so blindfolded, or worse: under the mistaken impression that the spell is working effectively.

In sum, when institutions enable faculty involvement in learning analytics initiatives, they are making themselves vulnerable to assessment, inviting ethical scrutiny, and opening doors to criticism that may not be readily forthcoming from within IT’s staff hierarchy (e.g., Morrison 2011). Ideally, we believe these will cause the apprentice to proceed with caution when practicing spellcasting. Such caution may hinder an institution’s ability to quickly innovate, but we argue that it will also enable more effective, responsible, and successful learning analytics initiatives.

At our local institution, we have been proud to be members of an IT organization that has valued faculty involvement. The simple fact that we have held leadership positions in IT, as academic appointees, was evidence of this collaborative spirit. Nevertheless, we still observed anecdotal evidence of a sorcerer’s-apprentice-like pattern at play. Our institution’s least successful initiatives have been those where IT has pursued analytical services in isolation, such as investing in a costly predictive analytics solution that failed to deliver value or insight. But we’ve also had successes stemming from efforts to involve faculty in learning technology and learning data efforts, such as through IU’s faculty-driven e-text program (Abaci & Quick, 2020; Abaci et al., 2017), Center for Learning Analytics and Student Success (CLASS; Rehrey et al., 2019), Charting the Future initiative (https://chartingthefuture.iu.edu/), and eLearning Research and Practice Lab (https://pti.iu.edu/elearning-lab). For example, the eLearning Lab has facilitated the multi-institutional ManyClasses research project (Fyfe et al., 2021), population-scale investigations into the digital divide (Jaggars et al., 2021) and students’ grade disclosures (Kolb et al., in press) during the COVID-19 pandemic, and a scalable model for automated nudges that is both ethically responsible (Motz, 2019) and effective (Motz et al.,2021), and more – none of which would have been possible without faculty leadership. Plenty of work still remains, as we continue to negotiate guiding principles and establish organizational structure to sustain inter-unit collaborations. And we are not alone – for example, models of these ecosystems can also be found in the Action lab at Arizona State University, the eCampus Research Unit at Oregon State, and University of Michigan’s Center for Academic Innovation, who all welcome faculty into IT’s learning analytics initiatives. Although anecdotal, we see better value in efforts to build these kinds of action-oriented ecosystems, rather than isolated top-down initiatives.

While these parallels may be compelling, to be clear, we do not propose that The Sorcerer’s Apprentice is a perfect analogy for institutional learning analytics. A prominent discrepancy is that, in Goethe’s poem, the sorcerer is entitled to his own spell book, but in institutional learning analytics, it should not be taken for granted that faculty are entitled to student learning data. Involving faculty in the leadership of learning analytics initiatives carries a set of privacy concerns that need to be carefully navigated, as with any learning analytics initiative (Slade & Prinsloo, 2013). For those supported by our local eLearning Lab, we address such concerns by adopting a role as a trusted independent data manager. This data manager has access to institutional data and acts as a buffer, enforcing policies and standards of data use determined in collaboration with data stewards. Faculty may request data from the data manager, so long as their project is IRB-approved and it complies with the stated policies and standards. All data requests are made public, facilitating transparency and knowledge sharing (eLearning Research and Practice Lab, 2022). In this way, our lab creates connective tissue between faculty researchers, administration, and the IT systems that store learning records, ensuring that the sorcerer has a place of privilege in our local learning ecosystem, while still protecting sensitive data.

Whether through a lab, a center, a data manager, or other structures, we believe that institutions of higher education should build bridges that enable faculty leadership in learning analytics initiatives. However, we do not mean to suggest that faculty will independently seek such a collaboration. While faculty of higher education are highly capable at advancing their own research and teaching agendas, these same faculty may see little incentive to engage in the institution’s initiatives, particularly if this engagement is perceived as a time burden. To achieve truly collaborative institutional ecosystems, it will require movement on the faculty’s behalf, to engage in new partnerships with their institutions. Negotiating these partnerships may be challenging, but such negotiations constitute the “debate” that Wheeler and Hilton (2012) see as IT’s responsibility in an increasingly connected world. Moreover, institutional data are potentially valuable to some faculty researchers, as they present opportunities for scholarly inquiry at a very large scale. At minimum, the sorcerer returns to address an emergent flood in response to the apprentice’s cries, and we hope that collaborations can be established preventatively, before any hazards emerge.

However, one reason that institutional IT may feel comfortable pursuing learning analytics initiatives in isolation is that, in many ways other than learning analytics, IT is hardly a novice apprentice. By establishing networks, web services, student information systems, identity management processes, and more, particularly since the COVID-19 pandemic, IT has plenty of experience building systems and managing initiatives on its own. What makes learning analytics different? One prominent difference is that learning analytics is a means to an end, not an end on its own. How analytics should effectively support the institution’s education mission is not a well-defined problem, and lacks the acceptance criteria that is typical of other IT projects. Furthermore, the institution’s analysis of learning data raises privacy and trust concerns that are inapplicable to traditional IT services. Managing an institution’s email service is fundamentally different from counting students’ clicks in the learning management system. And finally, as we have argued throughout this commentary, there are risks associated with learning analytics initiatives that may not be readily apparent to IT, or that IT might be ill-equipped to effectively assess.

Admittedly it is possible that other analogies might more accurately capture other aspects of institutional learning analytics initiatives. But, while analogies are always imperfect, they nevertheless can offer new and compelling perspectives on an issue. There are plenty of well-written, persuasive articles advocating for broader faculty involvement in institutional analytics initiatives (to name a few: Almond-dannenbring et al., 2022; Dawson et al., 2018; Howell et al., 2018; Macfadyen, 2022; Michos et al., 2020; Tsai et al., 2022), and merely repeating these past arguments is unlikely to be constructive. In writing this commentary we hope that institutions, when considering their learning analytics initiatives, simply ask themselves, “Are we behaving like the sorcerer’s apprentice?” And moreover, we hope this simple heuristic compels institutions, when their honest answer is affirmative, to focus more on building a learning ecosystem, extending leadership roles to those teachers and researchers who know what they’re doing.