Abstract
Quantitative studies on technology integration often examined general quantity of classroom technology use or teacher-reported accounts of integration practices. There is a current need for a measurement tool that links students’ technology use and cognitive engagement, which will allow researchers to better illustrate how technology is woven into the learning process. The purpose of this study is to develop a scale to measure how students use technology for different cognitive tasks, following theoretical conceptions from Bloom’s Digital Taxonomy and Multiple-Document Task-based Relevance Assessment and Content Extraction. We employed Confirmatory Factor Analysis, as well as both classical test theory and item response theory over three studies to validate our newly created scale. The new Cognitive Engagement with Technology (CET) scale showed good psychometric properties, item functioning, and construct validity. The CET scale can be used to triangulate students’ technology use patterns with other research methods. It can also help extend past findings by taking into account how students use technology to aid in the cognitive processes of learning.
Similar content being viewed by others
References
Ajzen, I. (1985). From intentions to actions: a theory of planned behavior. In J. Kuhl & J. Beckmann (Eds.), Action control: From cognition to behavior (pp. 11–39). Springer.
Ajzen, I. (2002). Perceived behavioral control, self-efficacy, locus of control, and the theory of planned behavior. Journal of Applied Social Psychology, 32(4), 665–683. https://doi.org/10.1111/j.1559-1816.2002.tb00236.x
Al-Adwan, A. A., Al-Adwan, A. A., & Smedley, J. (2013). Exploring students acceptance of e-learning using Technology Acceptance Model in Jordanian universities. International Journal of Education and Development Using ICT, 9(2), 2.
Anderson, L. W., & Krathworthl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assisting: A revision of Bloom’s taxonomy of education objectives. Longman.
Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41(5), 873–894. https://doi.org/10.1007/s11251-013-9263-8
Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research study on technology in K–12 schools: Technology integration as it relates to the National Technology Standards. Journal of Research on Technology in Education, 35(4), 489–507.
Böckenholt, U., & Meiser, T. (2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70(1), 159–181.
Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84(1), 58–85. https://doi.org/10.1111/bjep.12005
Bråten, I., Stadtler, M., & Salmerón, L. (2017). The role of sourcing in discourse comprehension. The Routledge Handbook of Discourse Processes, Second Edition. https://doi.org/10.4324/9781315687384
Cameron, C., Van Meter, P., & Long, V. A. (2017). The effects of instruction on students’ generation of self-questions when reading multiple documents. Journal of Experimental Education, 85(2), 334–351. https://doi.org/10.1080/00220973.2016.1182884
Chen, R. J. (2010). Investigating models for preservice teachers’ use of technology to support student-centered learning. Computers and Education, 55(1), 32–42. https://doi.org/10.1016/j.compedu.2009.11.015
Cheng, E. W. L. (2018). Choosing between the theory of planned behavior (TPB) and the technology acceptance model (TAM). Educational Technology Research and Development, 67(1), 1–17. https://doi.org/10.1007/s11423-018-9598-6
Cheung, R., & Vogel, D. (2013). Predicting user acceptance of collaborative technologies: An extension of the technology acceptance model for e-learning. Computers and Education, 63, 160–175. https://doi.org/10.1016/j.compedu.2012.12.003
Churches, A. (2009). Bloom's digital taxonomy. Retrieved 6th June, 2020, from http://edorigami.wikispaces.com/Bloom%27s+Digital+Taxonomy.
Clarebout, G., & Elen, J. (2009). The complexity of tool use in computer-based learning environments. Instructional Science, 37(5), 475–486. https://doi.org/10.1007/s11251-008-9068-3
Crompton, H., Burke, D., & Lin, Y. C. (2019). Mobile learning and student cognition: A systematic review of PK-12 research using Bloom’s Taxonomy. British Journal of Educational Technology, 50(2), 684–701. https://doi.org/10.1111/bjet.12674
Crook, S. J., & Sharma, M. D. (2013). Bloom-ing heck! the activities of australian science teachers and students two years into a 1:1 laptop program across 14 high schools. International Journal of Innovation in Science and Mathematics Education, 21(1), 54–69.
Crow, S. R. (2015). The information-seeking behavior of intrinsically motivated elementary school children of a collectivist culture. School Library Media Research, 18.
Davis Jr, F. D. (1986). A technology acceptance model for empirically testing new end-user information systems: Theory and results [Doctoral dissertation, Massachusetts Institute of Technology]. https://dspace.mit.edu/handle/1721.1/15192#files-area.
D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082–1099. https://doi.org/10.1037/a0032674
Diacopoulos, M. M. (2015). Untangling web 2.0: Charting web 2.0 tools, the NCSS guidelines for effective use of technology, and bloom’s taxonomy. The Social Studies, 106(4), 139–148. https://doi.org/10.1080/00377996.2015.1015711
Dietrich, T., & Balli, S. J. (2014). Digital natives: Fifth-grade students’ authentic and ritualistic engagement with technology. International Journal of Instruction, 7(2), 21–34.
Dugdale, S., Legare, O., Matthews, J. I., & Ju, M. K. (1998). Mathematical problem solving and computers: A study of learner-initiated application of technology in a general problem-solving context. Journal of Research on Computing in Education, 30(3), 239–252. https://doi.org/10.1080/08886504.1998.10782225
Edmunds, R., Thorpe, M., & Conole, G. (2012). Student attitudes towards and use of ICT in course study, work and social activity: A technology acceptance model approach. British Journal of Educational Technology, 43(1), 71–84. https://doi.org/10.1111/j.1467-8535.2010.01142.x
Embretson, S. E., & Reise, S. P. (2000). Item response theory. Psychology Press.
Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.
Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128. https://doi.org/10.18608/jla.2017.42.10
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. https://doi.org/10.1002/RRQ.027
Heflin, H., Shewmaker, J., & Nguyen, J. (2017). Impact of mobile technology on student attitudes, engagement, and learning. Computers and Education, 107, 91–99. https://doi.org/10.1016/j.compedu.2017.01.006
Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers and Education, 90, 36–53. https://doi.org/10.1016/j.compedu.2015.09.005
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135.
Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223–252.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
Inan, F. A., Lowther, D. L., Ross, S. M., & Strahl, D. (2010). Pattern of classroom activities during students’ use of computers: Relations between instructional strategies and computer applications. Teaching and Teacher Education, 26(3), 540–546. https://doi.org/10.1016/j.tate.2009.06.017
Jiang, L., Elen, J., & Clarebout, G. (2009). The relationships between learner variables, tool-usage behaviour and performance. Computers in Human Behavior, 25(2), 501–509. https://doi.org/10.1016/j.chb.2008.11.006
Kim, M., Cheng, S. L., & Xie, K. (2017). The validation of a systemic evaluation framework to investigate the multi-layered impacts of technology integration projects. Paper presented at the annual meeting of the American Educational Research Association, San Antonio, TX.
Kline, R. B. (2011). Principles and practice of structural equation modeling. Guilford Publications.
Kopcha, T. J. (2012). Teachers’ perceptions of the barriers to technology integration and practices with technology under situated professional development. Computers & Education, 59, 1109–1121. https://doi.org/10.1016/j.compedu.2012.05.014
Linderholm, T., Kwon, H., & Therriault, D. J. (2014). Instructions that enhance multiple-text comprehension for college readers. Journal of College Reading and Learning, 45(1), 3–19. https://doi.org/10.1080/10790195.2014.906269
Lust, G., Elen, J., & Clarebout, G. (2013). Students’ tool-use within a web enhanced course: Explanatory mechanisms of students’ tool-use pattern. Computers in Human Behavior, 29(5), 2013–2021. https://doi.org/10.1016/j.chb.2013.03.014
Makransky, G., Lilleholt, L., & Aaby, A. (2017). Development and validation of the Multimodal Presence Scale for virtual reality environments: A confirmatory factor analysis and item response theory approach. Computers in Human Behavior, 72, 276–285. https://doi.org/10.1016/j.chb.2017.02.066
Miranda, H. P., & Russell, M. (2012). Understanding factors associated with teacher-directed student use of technology in elementary classrooms: A structural equation modeling approach. British Journal of Educational Technology, 43(4), 652–666.
Mueller, J., Wood, E., Willoughby, T., Ross, C., & Specht, J. (2008). Identifying discriminating variables between teachers who fully integrate computers and teachers with limited integration. Computers and Education, 51(4), 1523–1537. https://doi.org/10.1016/j.compedu.2008.02.003
Park, S. Y., Nam, M. W., & Cha, S. B. (2012). University students’ behavioral intention to use mobile learning: Evaluating the technology acceptance model. British Journal of Educational Technology, 43(4), 592–605. https://doi.org/10.1111/j.1467-8535.2011.01229.x
Pellas, N. (2014). The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning programs: Evidence from the virtual world of Second Life. Computers in Human Behavior, 35, 157–170. https://doi.org/10.1016/j.chb.2014.02.048
Pittman, T., & Gaines, T. (2015). Technology integration in third, fourth and fifth grade classrooms in a Florida school district. Educational Technology Research and Development, 63(4), 539–554. https://doi.org/10.1007/s11423-015-9391-8
Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013). How teachers are using technology at home and in their classrooms. Pew Research Center.
Rashid, T., & Asghar, H. M. (2016). Technology use, self-directed learning, student engagement and academic performance: Examining the interrelations. Computers in Human Behavior, 63, 604–612. https://doi.org/10.1016/j.chb.2016.05.084
Rouet, J. F., & Britt, A. M. (2011). Relevance processes in multiple document comprehension. Text Relevance and Learning from Text, June, 19–52.
Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R., Abrami, P. C., Wade, C. A., & Lowerison, G. (2009). Technology’s effect on achievement in higher education: A Stage I meta-analysis of classroom applications. Journal of Computing in Higher Education, 21(2), 95–109.
Sun, J. C. Y., & Rueda, R. (2012). Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191–204. https://doi.org/10.1111/j.1467-8535.2010.01157.x
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4–28.
Teo, T. (2009). Modelling technology acceptance in education: A study of pre-service teachers. Computers & Education, 52(2), 302–312. https://doi.org/10.1016/j.compedu.2008.08.006
Teo, T. S., Srivastava, S. C., & Jiang, L. (2008). Trust and electronic government success: An empirical study. Journal of Management Information Systems, 25(3), 99–132.
Tondeur, J., Aesaert, K., Prestridge, S., & Consuegra, E. (2018). A multilevel analysis of what matters in the training of pre-service teacher’s ICT competencies. Computers and Education, 122, 32–42. https://doi.org/10.1016/j.compedu.2018.03.002
Tondeur, J., van Keer, H., van Braak, J., & Valcke, M. (2008). ICT integration in the classroom: Challenging the potential of a school policy. Computers and Education, 51(1), 212–223. https://doi.org/10.1016/j.compedu.2007.05.003
Vannatta, R. A., & O’Bannon, B. (2002). Beginning to put the pieces together: A technology infusion model for teacher education. Journal of Computing in Teacher Education, 18(4), 112–123.
Vongkulluksn, V. W., Xie, K., & Bowman, M. A. (2018). The role of value on teachers’ internalization of external barriers and externalization of personal beliefs for classroom technology integration. Computers & Education, 118, 70–81.
Wang, S. K., Hsu, H. Y., Campbell, T., Coster, D. C., & Longhurst, M. (2014). An investigation of middle school science teachers and students use of technology inside and outside of classrooms: Considering whether digital natives are more technology savvy than their teachers. Educational Technology Research and Development, 62(6), 637–662.
Xie, K., & Luthy, N. (2017). Textbooks in the digital world. The Conversation. Retrieved from https://theconversation.com/textbooks-in-the-digital-world-78299.
Acknowledgements
The study reported in this paper is based upon work in the EDCITE: Evaluating Digital Content for Instructional and Teaching Excellence project and the College Ready Ohio project supported by the Straight A fund from the Ohio Department of Education. The grant organization had no involvement in the design, data collection, data analysis, writing, or decision on article submission. The conclusions and recommendations within this article do not necessarily reflect the views of the Ohio Department of Education.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Vongkulluksn, V.W., Lu, L., Nelson, M.J. et al. Cognitive engagement with technology scale: a validation study. Education Tech Research Dev 70, 419–445 (2022). https://doi.org/10.1007/s11423-022-10098-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-022-10098-9