Is New Zealand’s Higher Education Sector Ready to Employ Analytics Initiatives to Enhance its Decision-making Process?

Abstract

Interest in the use of analytics to support evidence-based decision-making in higher education is relatively a new phenomenon. The available research suggests that analytics can enhance an institution’s ability to make evidence-based informed decisions that foster growth and increased productivity. The present study explored how institutions of higher education in New Zealand value and utilise analytics to enhance the quality of decision-making. The study involved administering an online questionnaire (n = 82) to senior administrators from seven of the eight research-intensive public universities in New Zealand. Key findings revealed that the use of analytics in the higher education sector could enhance the quality of decision-making. Respondents reported that the use of analytics is likely to advance operational and strategic decisions by monitoring and efficiently optimising the use of resources. Respondents said that analytics provide better planning of issues about students (e.g. enrolment, retention, and completion rate). Furthermore, analytics can improve institutional research administration (e.g. performance outcomes in finance and human resources). Despite the stated benefits of analytics in enhancing decision-making in many operational and strategic areas, the study found that the use of analytics in the higher education sector in New Zealand is limited to monitoring operational activities, rather than improving the quality of learning, teaching and strategic initiatives. Further, the study identified several concerns regarding the ability of institutions to find the capacity and expertise to extract useful information from the available data sets and to turn such data into usable knowledge to support students and educators. Also, respondents were concerned that the lack of staff capacity and training to use analytics effectively could be detrimental. Others feared that analytics could be used as surveillance tools and enforce compliance and control. It was mentioned that over-reliance on analytics could easily lead to a breach of individuals’ and institutional privacy, which is a threat to information and data security. Also, there was a concern that analytics could be used to perpetuate inequity and inequality among and within institutions. Results of the study presented in this article serve as a baseline for future studies about the use of analytics in the higher education sector in New Zealand. Additionally, it contributes to the growing debate about the value and challenges of deploying analytics and Big Data in the higher education sector worldwide.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

References

  1. Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan, V. (2017). NMC horizon report: 2017 higher education edition. The New Media Consortium, Austin. Available at: http://cdn.nmc.org/media/2017-nmc-horizon-report-he-EN.pdf. Accessed 11 Dec 2020.

  2. Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the second International Conference on Learning Analytics and Knowledge (pp. 267–270). Vancouver, British Columbia, Canada. Retrieved on 18 October 2020 from http://dl.acm.org/citation.cfm?id=2330666.

  3. Atif, A., Richards, D., Bilgin, A., & Marrone, M. (2013). Learning analytics in higher education: A summary of tools and approaches. In 30th Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2013), Sydney, (pp. 68–72). Retrieved on 9 November 2020 from http://www.learntechlib.org/p/162009.

  4. Avella, J. T., Kebritchi, M., Nunn, S. G., & Kanai, T. (2016). Learning analytics methods, benefits, and challenges in higher education: A systematic literature review. Online Learning, 20(2), 13–29.

    Google Scholar 

  5. Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. Learning analytics (pp. 61–75). Springer New York: Springer. https://doi.org/10.1007/978-1-4614-3305-7_4.

  6. Bakshi, K. (2012). Considerations for big data: architecture and approach. Paper presented at the IEEE Aerospace Conference, Big Sky, MT, 2012, pp. 1–7. https://doi.org/10.1109/AERO.2012.6187357.

  7. Brooks, C., & Greer, J. (2014, March). Explaining predictive models to learning specialists using personas. In Proceedings of the 4th International Conference on Learning Analytics and Knowledge (pp. 26–30). https://doi.org/10.1145/2567574.2567612.

  8. Campbell, J. P., & Oblinger, D. G. (2007). Academic analytics. Retrieved on 9 Nov 2020 from https://library.educause.edu/resources/2007/10/academic-analytics.

  9. Charlton, P., Mavrikis, M., & Katsifli, D. (2013). The potential of learning analytics and Big Data. Ariadne, 71. Retrieved on 9 Nov 2020 from http://www.ariadne.ac.uk/issue71/charlton-et-al.

  10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Erlbaum.

    Google Scholar 

  11. Daniel, B. (2015). Big Data and analytics in higher education: Opportunities and challenges. British Journal of Educational Technology, 46(5), 904–920. https://doi.org/10.1111/bjet.12230.

    Article  Google Scholar 

  12. Daniel, B. K. (2019). Artificial reality: The practice of analytics and big data in educational research. In J. S. Pedersen & A. Wilkinson (Eds.), Big data: Promise, application and pitfalls (pp. 287–300). Cheltenham: Edward Elgar. https://doi.org/10.4337/9781788112352.00018.

  13. Dawson, S., Bakharia, A., & Heathcote, E. (2010, May). SNAPP: Realizing the affordances of real-time SNA within networked learning environments. Proceedings of the 7th International Conference on Networked Learning. 125–133. Retrieved on 9 Nov 2020 from https://at.doit.wisc.edu/wp-content/uploads/2019/03/LA-SNAPP_Dawson.pdf.

  14. de Winter, J. C. F., Dodou, D., & Wieringa, P. A. (2009). Exploratory factor analysis with small sample sizes. Multivariate Behavioral Research, 44(2), 147–181. https://doi.org/10.1080/00273170902794206.

    Article  Google Scholar 

  15. El Alfy, S., Gómez, J. M., & Dani, A. (2019). Exploring the benefits and challenges of learning analytics in higher education institutions: a systematic literature review. Information Discovery and Delivery, 47(1), 25–34. https://doi.org/10.1108/IDD-06-2018-0018.

    Article  Google Scholar 

  16. Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4.

    Article  Google Scholar 

  17. Fritz, J. (2011). Classroom walls that talk: using online course activity data of successful students to raise self-awareness of underperforming peers. Internet and Higher Education, 14(2), 89–97. https://doi.org/10.1016/j.iheduc.2010.07.007.

    Article  Google Scholar 

  18. Gašević, D., Tsai, Y. S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 36(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024.

    Article  Google Scholar 

  19. Goldstein, P. J., & Katz, R. N. (2005). Academic Analytics: The Uses of Management Information and Technology in Higher Education, ECAR Research Study Vol. 8. Retrieved on 9 Nov 2020 from https://www.educause.edu/ir/library/pdf/ers0508/rs/ers0508w.pdf.

  20. Greer, J., Thompson, C., Banow, R., & Frost, S. (2016a). Data-driven programmatic change at universities: What works and how. In Proceedings of the 1st Learning Analytics for Curriculum and Program Quality Improvement Workshop (PCLA 2016), Vol. 25, pp. 30–33.

  21. Greer, J., Molinaro, M., Ochoa, X., & McKay, T. (2016b). Learning analytics for curriculum and program quality improvement (PCLA, 2016). In Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 494–495). https://doi.org/10.1145/2883851.2883899.

  22. Hazelkorn, E. (2007). The impact of league tables and ranking systems on higher education decision- making. Higher Education Management and Policy, 19(2), 1–24. https://doi.org/10.1787/hemp-v19-art12-en.

    Article  Google Scholar 

  23. Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: the teachers’ role and perspective. Educational Technology Research and Development, 67(5), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0.

    Article  Google Scholar 

  24. Hilbert, M. (2013). Big data for development: from information to knowledge societies (January 15, 2013). Available at SSRN: https://ssrn.com/abstract=2205145 or https://doi.org/10.2139/ssrn.2205145. Accessed 11 Dec 2020.

  25. Hrabowski, F. A., III, Suess, J., & Fritz, J. (2011a). Analytics in institutional transformation. Educause Review (pp. 15–28). Retrieved on 9 Nov 2020 from http://hdl.handle.net/11603/19019.

  26. Hrabowski, F. A., III, Suess, J., & Fritz, J. (2011b). Assessment and analytics in institutional transformation. Assessment and analytics in institutional transformation. Educause Review, 46(5) (September/October 2011). Retrieved on 9 Nov 2020 from http://hdl.handle.net/11603/19019.

  27. Jones, K. M., Rubel, A., & LeClere, E. (2020). A matter of trust: higher education institutions as information fiduciaries in an age of educational data mining and learning analytics. Journal of the Association for Information Science and Technology, 71(10), 1227–1241. https://doi.org/10.1002/asi.24327.

    Article  Google Scholar 

  28. Jones, S. J. (2012). Technology review: The possibilities of learning analytics to improve learner-centred decision-making. Community College Enterprise, 18(1), 89–93.

    Google Scholar 

  29. Kaisler, S., Armour, F., Espinosa, J. A., & Money, W. (2013). Big Data: Issues and challenges moving forward. Paper presented at the System Sciences (HICSS), 2013 46th Hawaii International Conference on System Sciences, Wailea, Maui, HI, 2013, pp. 995–1004, https://doi.org/10.1109/HICSS.2013.645.

  30. Lawson, C., Beer, C., Rossi, D., Moore, T., & Fleming, J. (2016). Identification of ‘at risk’ students using learning analytics: the ethical dilemmas of intervention strategies in a higher education institution. Educational Technology Research and Development, 64(5), 957–968.

    Article  Google Scholar 

  31. Li, K.-C., & Wong, B. T.-M. (2020). Trends of learning analytics in STE(A)M education: a review of case studies. Interactive Technology and Smart Education, 17(3), 323–335. https://doi.org/10.1108/ITSE-11-2019-0073.

  32. Liu, D. Y. T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-driven personalization of student learning support in higher education (pp. 143–169). Cham: Springer. https://doi.org/10.1007/978-3-319-52977-6_5.

  33. Mahroeian, H., Daniel, B., & Butson, R. (2017). The perceptions of the meaning and value of analytics in New Zealand higher education institutions. International Journal of Educational Technology in Higher Education, 14(1), 35. https://doi.org/10.1186/s41239-017-0073-y.

    Article  Google Scholar 

  34. Moreno-Marcos, P. M., Pong, T. C., Muñoz-Merino, P. J., & Kloos, C. D. (2020). Analysis of the factors influencing learners’ performance prediction with learning analytics. IEEE Access: Practical Innovations, Open Solutions, 8, 5264–5282. https://doi.org/10.1109/ACCESS.2019.2963503.

    Article  Google Scholar 

  35. Moussavi, M., Amannejad, Y., Moshirpour, M., Marasco, E., & Behjat, L. (2020). Importance of data analytics for improving teaching and learning methods. In Data Management and Analysis (pp. 91–101). Cham: Springer. https://doi.org/10.1007/978-3-030-32587-9_6.

  36. Negash, S., & Gray, P. (2008). Business intelligence. Handbook on decision support systems 2 (pp. 175–193). Berlin: Springer.

    Google Scholar 

  37. Niemi, D., & Gitin, E. (2012). Using big data to predict student dropouts: technology affordances for research, IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012). Retrieved on 9 Nov 2020 from https://eric.ed.gov/?id=ED542777.

  38. Norris, D., Baer, L., Leonard, J., Pugliese, L., & Lefrere, P. (2008). Action analytics: Measuring and improving performance that matters in highereducation. EDUCAUSE Review, 43(1), 42. https://eric.ed.gov/?id=EJ783251. Accessed 11 Dec.

  39. Oblinger, D. G. (2012a). Game changers: Education and information technologies. In D. G. Oblinger (Ed.), Design. Educause. Available from: http://net.educause.edu/ir/library/pdf/pub7203.pdf. Accessed March 2013.

  40. Oblinger, D. G. (2012b). (18) (PDF) Review of game changers: Education and information technologies. Available from: https://www.researchgate.net/publication/286689965_Review_of_Game_Changers_Education_and_Information_Technologies_by_Diana_G_Oblinger. Accessed Dec 11 2020.

  41. OECD (2013). OECD report: the state of higher education 2013. Retrieved on 9 Nov 2020 from http://www.oecd.org/edu/imhe/thestateofhighereducation2013.htm.

  42. Phillips, E. D. (2013). Improving advising using technology and data analytics. Change: The Magazine of Higher Learning, 45(1), 48–55. https://doi.org/10.1080/00091383.2013.749151.

    Article  Google Scholar 

  43. Picciano, A. G. (2012). The evolution of Big Data and learning analytics in American higher education. Journal of Asynchronous Learning Networks, 16(3), 9–20. Retrieved on 9 Nov 2020 from https://eric.ed.gov/?id=EJ982669.

  44. Prinsloo, P. (2019). A social cartography of analytics in education as performative politics. British Journal of Educational Technology, 50(6), 2810–2823. https://doi.org/10.1111/bjet.12872.

    Article  Google Scholar 

  45. Rajesh, K. V. N. (2013). Big data analytics: applications and benefits. IUP Journal of Information Technology, 9(4), 41–51.

    Google Scholar 

  46. Sclater, N., Peasgood, A., & Mullan, J. (2016). Learning analytics in higher education. A review of UK and international practice. Jisc Report (April 2016). Retrieved on 9 Nov 2020 from https://www.jisc.ac.uk/sites/default/files/learning-analytics-in-he-v3.pdf.

  47. Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioural Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851.

    Article  Google Scholar 

  48. Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 30–40. Retrieved on 9 Nov 2020 from https://eric.ed.gov/?id=EJ950794.

  49. Tsai, Y. S., Rates, D., Moreno-Marcos, P. M., Muñoz-Merino, P. J., Jivet, I., Scheffel, M., Drachslered, H., Kloosc, C. D., & Gašević, D. (2020). Learning analytics in European higher education–trends and barriers. Computers & Education, 155, 103933. https://doi.org/10.1016/j.compedu.2020.103933.

  50. Tulasi, B. (2013). Significance of big data and analytics in higher education. International Journal of Computer Applications, 68(14), 23–25.

    Article  Google Scholar 

  51. U.S. Department of Education, National Center for Education Statistics (2011). The condition of education 2011 (NCES 2011-033). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

  52. Van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a common language. Educause Learning Initiative, 1(1), l–ll.

  53. Vytasek, J. M., Patzak, A., & Winne, P. H. (2020). Analytics for student engagement. Machine learning paradigms (pp. 23–48). Cham: Springer. https://doi.org/10.1007/978-3-030-13743-4_3.

  54. Wagner, E., & Ice, P. (2012). Data changes everything: Delivering on the promise of learning analytics in higher education. Educause Review, 47(4), 33–42. Retrieved on 9 Nov 2020 from https://eric.ed.gov/?id=EJ981214.

  55. West, D. M. (2012). Big Data for education: Data mining, data analytics, and web dashboards. Governance Studies at Brookings, 4(1), 1–10. Retrieved on 18 Oct 2020 from https://www.brookings.edu/research/Big-Data-for-education-data-mining-data-analytics-and-web-dashboards/.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Hamid Mahroeian.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Questionnaire

figureafigureafigureafigureafigureafigureafigureafigureafigurea

Appendix 2: Thematic Analysis of Responses to Open-ended Questions

What are the potential benefits of using analytics in higher education?

Theme Example of quotations
Efficiency and effectiveness in resource utilization Optimal and targeted use of finite resources requires an understanding of the use of those resources [1].
Optimizing the use of resources [2].
Analysing data helps to understand what is happening and potentially why. This can lead to making efficiencies and streamlining to make things better for everyone [3].
Positive contribution to efficiency in the use of resources [8]
The focus is definitely on using data to find ways to minimize loss and keep costs low. I do not think they are well used for student progress and support [9].
With little opportunity for growth in revenues, analytics are used mainly to reduce costs or promote efficiencies in the services provided [10].
To me, these are the most numerous benefits and main ones; especially resources used in the institution can be well optimized using a specific sort of analysis, improvement of faculty and research performance by extracting the right data and getting insightful ('meaningful') data [11].
I support the use of analytics to improve effectiveness and excellence, not to promote cost reduction (which usually involves job losses) [12].
Our office provides a lot of the information senior management uses to make decisions, and I have strong faith in our ability/capacity to provide high-quality information [13]
Improving administrative services Improving administrative services [1]
Reduce costs, improve performance, improve knowledge, and improve security [2].
Income generation efficiency [3].
It seems primarily used for KPI measurement and reporting [4].
All about performance and optimization [6].
Knowing where the wasted resources are would enable better use of money, etc. [7].
Data analysis has always been used to monitor performance, compare competitors' progress and assist with strategic planning in areas such as highlighted above--sophisticated analytics can be useful to understand more massive data sets and give either more detailed performance information or a clearer 'big picture' view of performance [8].
There have been no apparent decreases in costs, nor effectiveness seen elsewhere—just more magnificent gnashing of teeth [9].
Analytics gives you a sense of where you are at, so you can then put in place plans to improve your situation or prioritize between different options [10].
Analytics can show what is going on rather than just using guesswork and have real-time data [12].
Important decisions should be based on analytics wherever possible [13].
Universities are large businesses with high levels of accountability, and they need the best information and evidence available to support their decision-making [14].
Improving student services Understanding student demographics and behaviours [1].
I prefer analytics to be used to improve student learning rather than monitor students or staff [2].
From my perspective, the most significant benefit is information on what is happening on the broader market and how can we use this to increase sales/enrolments [3].
Analytics can improve learning and teaching by understanding behavioural responses to teaching methods. I can help finances by looking at patterns and identifying areas to economize [4].
We are very ignorant of what drives students and the different sorts of student we have, and how to help them succeed [5].
The main reasons would be to do with recruitment and then quality enhancement and quality assurance [6].
Better tracking of student performance over time will allow for the identification of both students who excel. It could be targeted/nurtured for further education (post-grad), as well as early identification and intervention for students who may be struggling before they fail their course[7].
I see the potential power of analytics in improving student retention, engagement and learning [8].
Current and future prediction Analytics provides a measure to calculate/estimate or forecast what is currently happening and what could happen in the future [1].
Support understanding of equity Analytics provide the basis for making decisions. For example to understand whether there is a gender pay gap at the university data is required to know how men and women are being placed on grades, the rates at which they are promoted and the pay and the reasons why the payment might be different. By analysing the data, you can determine whether you need to respond to an issue, and if so, what is the best way of resolving or improving the situation [1].
Theme Example of quotations [participants]
Financial decision-making Increasingly restricted funding, government directives on acceptable outcomes and government interference on graduate preferences [1].
Financial provision will depend on metrics around employability and not just results. Staffing will depend on metrics around sensible FTE/papers taught, not FTES [2].
Student recruitment and success Enrolments fell at our institution this year, but through analytics, we can see how to boost numbers by enhancing entrance scholarships. Also, to improve retention, we are using analytics to monitor student progress and to intervene when progress is unsatisfactory [1].
Having spent some time reading in this area, I think it is the only way for universities to continue to demonstrate value by ensuring student success [2].
I can see recruitment, admission and selection all being heavily influenced by analytics and in the future, curriculum and pedagogy design [3].
Business intelligence Business intelligence and analytics play a vital role. Though the access has become relatively straightforward, it is hard for people to make sense as so many tools offer so many different views [1]. We need to have a better understanding of our business to stay in business [2].
Analytics will form an essential source of data. Still, the most impactful metrics will be around business intelligence―who is enrolling, who is leaving, who is succeeding primarily factors affecting financial bottom line [3].
Being able to spot well-performing departments and poor performing departments would enable corrective action to be taken [4].
The university is partway through a business intelligence implementation. The data is being provided more widely than before and will undoubtedly aid data transparency. Provided we also invest in capability, we should, as an institution, be able to use the data to support targeted student learning support, target resources where required, and identify performance issues across faculty and administrative processes which should enable improved performance in these functions over time [5].
Strategic planning Analytics will be essential to ongoing strategic planning and institutional improvement. If you cannot measure progress, you will not know if you are improving [1].
The need to provide back up for strategy and approaches to issues is vital. Without understanding, for example, the workforce, then you cannot develop appropriate strategies and policies to deal with issues such as recruitment retention etc.[2]
The more information we have about a particular area of interest, the better the future outcome should be [3].
It will be instrumental; the information is out there and ignoring it means you cannot improve or target accurately [4].
An alternative source of evidence for enhanced decision-making In the near future, the almost overall success will be highly dependent on analytics use for evidence-based decision-making [1].
We could say in the near future perhaps as we can see in already in business such as Facebook, Netflix, Google, etc. I believe, yes, to some extent talking about the future success of HEIs will be depending somehow to some extent on analytics use of data for decision-making [2].
One cannot make decisions without understanding the immediate environment — analytics guides better decisions [3].
While it is not a perfect solution, the more information considered and available when making a decision contributes to the quality and rigidity of that decision [4].
I agree that success will be enhanced by more data-driven decision-making [5].
Having analytics provides an opportunity to make more informed business decisions [6].
We are an evidence-based organization so data analysis will always play a part in our decision-making but tempering that is the need for human judgment and intuition to guide the areas to explore for future directions [7].
Reliable analytics should translate into better decision-making [8].
Accurate information is crucial for informed decision making [9].
Analytics can significantly enhance efficiency [10].
Without sufficient information on what is, and what is not, the intended results or goals would rely more heavily on luck [11].
Analytics is only one-way to guide decision-making [12].
Addressing challenges institutions face Given the range of challenges facing HEI, the more accurate data that can be used, the better we know our current situation, and where we can go from there [1].
We have constrained resources and operate in a fiercely competitive environment. We need to allocate resources optimally to be able to compete [2].
I do believe there is a place for data in decision-making, but not every decision can be based on hard data. A balance of quantitative and qualitative is vital [3].
Human-in-the loop analytics I believe good leadership and passion for education and research are more important than analytics. It will not 'rely' on it, but it will play a part [1].
Still need to ensure that humans are considered the critical elements in the activity (students, the staff of all types) [2].
Analytics will be a component in success, not the underpinning factor. Many factors are influencing the tertiary education sector, and the overall quality of thought is most famous for decision-making [4].
Theme Example of quotations
Access and interpretation Information required is widely dispersed and not easily interpreted. Much of the easily accessible information is not accessed or well utilized [1].
The decision-makers in my institution have great access to information they can request reports from several departments. The reason I said 'agree' and not 'strongly agree' is that it sometimes takes time to put those reports together, so it is 'easy' but not always speedy! [2].
The information is not well understood. The information is not disseminated to the people who can 'make a difference'[3].
There are many decision-makers (e.g. teachers, managers, HoDs, etc.), and I am not sure all of them have easy access to useful data [4].
I think the decision-makers are given access to some of the information they need to make crucial decisions. Still, I believe it would be inconsistent and not always clearly explained or interpreted [5].
I believe they have access to information, but it is not necessarily easy [6].
The data is hard to attain as not all processes can be streamlined or monitored with online tools or computer tools currently, as it is still very paper-based systems [7].
I think decision-makers have access to data if they know what to ask for; access, in my experience, is not easy [8].
Can be very difficult to get the information, and can be out of date once it is obtained [9].
For us, this is a work in progress. Decision-makers have access to excellent information in some areas and limited access in others [10].
I think decision-makers who are interested in making evidence-based decisions seek out the appropriate information some of it is easy to get, some of it is hard [11].
Not easy access, perhaps, it could be difficult but still should be available and accessible to significant decision-makers and critical people for pressing issues [12].
Challenges in meeting different information needs We have some decision-makers who require a high level of information and others who need more detail. The detailed requests can be challenging to meet [1].
Decision-makers are positioned at different levels, and their data needs change. At the macro level, senior managers (VC and DVC, even faculty/division Heads) their data needs are more around reporting/measuring activity and outcomes they have this data. Fine-grained data, though, that gives meaningful insight into learning behaviour/engagement and outcomes is limited, although emerging and nascent [2].
Different administrators, suggesting that they are not collated and widely available, often at short notice, which reduces their accuracy [3], often request the same data.
Challenges in data extraction Data are sometimes challenging to extract [2].
Information is dispersed We struggle to pull together simple metrics around research performance, revenue and expenditure from across the institution. I would imagine that other departments face similar problems[1]
Much information is lost in layers and silos [2].
Limited information Decisions are always made based on limited information. Still, decision-makers need to make sure they have the appropriate sources of information (because they often neglect to ask their staff what they think) [1].
There is little or no data warehousing of information [2].
Inaccurate and unreliable data I think their data is unreliable and inaccurate [1].
The reporting functionality is only really starting to be explored, and some of the products are poor and do not always show information in a useful way for analysis [2].
Lack of coordination and data governance Devolved nature of the university leads to lack of coordination or information sharing across different parts of the university [1].
Lack of technical infrastructure We are just instituting a range of platforms such as "dashboards" to make data access easier for Heads of Schools and their administrators [1].
Our BI journey is in its early stages, but this will improve in the next 18 months [2].
I do not think our institution has discovered the power of business analytics to use it to its best advantage [3].
Currently changing the system, so it has been hard to get the usual data will be better in the long run [4].
Several in house bespoke services and systems have not been built with the opportunity of analytics in mind [5].
I believe that, to some certain extent, my university has easy access to make significant decisions, but not sure, whether they are well-utilizing analytics tools and implementation for its extraction or not [6].
They have access to information but not easy access. This is because our systems are too devolved, and often information is only accessible by a limited group of people [7].
Data literacy In all my dealings within the leadership, data sorting has been quite difficult. They are always looking for easier methods of sorting data to inform decision-making [1].
I work with the people extracting the data. There is lot there, but a lot yet to be made digestible. Some data are readily available, but is it the data that should be driving decisions, or it leads to an obtuse, target-based decision making process?[2]
Leadership buy-in of information Relevant necessary information is readily available, but it would appear often not sought [1].
The influence of analytics does not reach the top levels of leadership [2].
Bureaucratic barriers make access to information complicated [3].
Theme Example of quotations
Privacy and user consent Privacy and informed consent are not easy concepts to apply in the context of analytics, yet they are principles underlying access and use of data [1].
Privacy issues and the potential for misuse [2].
Privacy and consent are always issues [3].
As Big Data develops, there comes a time when privacy concerns become prevalent [4].
Privacy and consent are always issues to deal with [5].
Privacy issues and potential for misuse[6]
Security Security of data [1].
Ethics and security of the data [2].
Protection of the data [3].
Data security, primarily [4].
Data ownership Who owns the data?[1]
Ethics Ethical considerations for both groups and individuals, power concerns, no direct comparisons between institutions due to no requirement for comparable standards [1].
Dehumanization Dehumanising leads to decisions that are perhaps not student-centred [1].
Neoliberal compliance and control Development of strategic capability, making sure that institutional leaders have basic data literacy and understanding what is necessary to make their institutions data-informed [1].
Development of implementation capability making sure that institution lined up all the required capabilities that the implementation of analytics may happen (technologies, relevant questions, professional development, easy access to analytics results/tools etc.) [2].
Fear of surveillance. Concerns about efficacy. Managerialism in higher education [3].
Massaging data to achieve an end [4].
Could get driven by numbers, which could stifle imagination [5].
Becomes a low-trust surveillance culture rather than a high trust professional culture
Any data can be manipulated [6].
May make things too impersonal. Also, you might have said a student, based on analytics might not look as though they will do well, when in fact, they do end up doing well. However, if decisions are made based purely on analytics, that person may never, in fact, end up in a particular programme [7].
I believe analytics is becoming too prevalent in terms of world university rankings, etc. Universities risk becoming also standardized in a neoliberal economic context [8].
Analytics used to put pressure on us to save money [9].
I believe analytics is becoming too prevalent in terms of world university rankings, etc. Universities risk becoming also standardized in a neoliberal economic context [10].
Data literacy and capability Superficial use without a deep understanding of data can lead to inaccurate conclusions (e.g. league tables) [2].
Lack of training Systems training resources [1].
Lack of trained staff in its use [2].
The challenge is ensuring you can collect relevant information quickly that is easy to extract in a useful way, which does not create a lot of extra work [3].
Data analysis may not always lead to appropriate decisions—i.e. causation, where there is only correlation [4].
Lack of skills inappropriate management [5].
Staff skills, costs, online availability [6].
Insufficient attention to learning statistics, financial analysis and basic mathematics [7].
Valid and rigorous interpretation of data is critical [8].
Superficial use without a deep understanding of data can lead to inaccurate conclusions (e.g. league tables) [11].
Training and skills [12].
Valid and rigorous interpretation of data is critical. Expertise to ensure this is still growing [13].
Keeping pace with rapidly changing technology Keeping pace with technology and knowing what is available [1].
Inequity and inequality Too much reliance on analytics can lead universities to perhaps focus only on high performing areas or success models, rather than providing a broad base of programmes and approaches to problems [1].
Analytics should only be one method of determining strategy and desired outcomes and not be the be-all and end-all of the response to particular issues [2].
Too many decisions being based on data and not enough on real-life impacts of findings [3].
Others Lack of infrastructure, traditional values, lack of insight, lack of empathy, lack of privacy, lack of expertise, lack of software, lack of data management, lack of holism, lack of dissemination, lack of feedback, lack of loops, lack of connectivity[1].

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mahroeian, H., Daniel, B. Is New Zealand’s Higher Education Sector Ready to Employ Analytics Initiatives to Enhance its Decision-making Process?. Int J Artif Intell Educ (2021). https://doi.org/10.1007/s40593-020-00234-y

Download citation

Keywords

  • Analytics
  • Higher education
  • Decision-making
  • Challenges