Assessing learning, quality and engagement in learning objects: the Learning Object Evaluation Scale for Students (LOES-S)

Research Article

Abstract

Learning objects are interactive web-based tools that support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners. Research on the impact, effectiveness, and usefulness of learning objects is limited, partially because comprehensive, theoretically based, reliable, and valid evaluation tools are scarce, particularly in the K-12 environment. The purpose of the following study was to investigate a Learning Object Evaluation Scale for Students (LOES-S) based on three key constructs gleaned from 10 years of learning object research: learning, quality or instructional design, and engagement. Tested on over 1100 middle and secondary school students, the data generated using the LOES-S showed acceptable internal reliability, face validity, construct validity, convergent validity and predictive validity.

Keywords

Evaluate Assess Quality Scale Secondary school Middle school Learning object 

References

  1. Acovelli, M., & Gamble, M. (1997). A coaching agent for learners using multimedia simulations. Educational Technology, 37(2), 44–49.Google Scholar
  2. Adams, A., Lubega, J., Walmsley, S., & Williams, S. (2004). The effectiveness of assessment learning objects produced using pair programming. Electronic Journal of e-Learning, 2(2). Retrieved July 28, 2005 from http://www.ejel.org/volume-2/vol2-issue2/v2-i2-art1-adams.pdf.
  3. Akpinar, Y., & Hartley, J. R. (1996). Designing interactive learning environments. Journal of Computer Assisted Learning, 12(1), 33–46.CrossRefGoogle Scholar
  4. Albanese, M. A., & Mitchell, S. A. (1993). Problem-based learning: A review of the literature on its outcomes and implementation issues. Academic Medicine, 68, 52–81.CrossRefGoogle Scholar
  5. Alonso, F., Lopez, G., Manrique, D., & Vines, J. M. (2005). An instructional model for web-based e-learning education with a blended learning process approach. British Journal of Educational Technology, 36(2), 217–235.CrossRefGoogle Scholar
  6. Baruque, L. B., & Melo, R. N. (2004). Learning theory and instructional design using learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 343–370.Google Scholar
  7. Baser, M. (2005). Promoting conceptual change through active learning using open source software for physic simulations. Australasian Journal of Educational Technology, 22(3), 336–354.Google Scholar
  8. Bennett, K., & McGee, P. (2005). Transformative power of the learning object debate. Open Learning, 20(1), 15–30.CrossRefGoogle Scholar
  9. Bradley, C., & Boyle, T. (2004). The design, development, and use of multimedia learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 371–389.Google Scholar
  10. Brown, A. L., & Palinscar, A. S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 393–451). Hillsdale, NJ: Erlbaum Associates.Google Scholar
  11. Brown, A. R., & Voltz, B. D. (2005). Elements of effective e-learning design. The International Review of Research in Open and Distance Learning, 6(1). Retrieved June 1, 2007 from http://www.irrodl.org/index.php/irrodl/article/view/217/300.
  12. Bruner, J. (1983). Child’s talk. Learning to use language. Toronto, Canada: George J. McLeod Ltd.Google Scholar
  13. Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.Google Scholar
  14. Brush, T., & Saye, J. (2001). The use of embedded scaffolds with hypermedia-supported student-centered learning. Journal of Educational Multimedia and Hypermedia, 10(4), 333–356.Google Scholar
  15. Buzzetto-More, N. A., & Pinhey, K. (2006). Guidelines and standards for the development of fully online learning objects. Interdisciplinary Journal of Knowledge and Learning Objects, 2006(2), 96–104.Google Scholar
  16. Cafolla, R. (2006). Project MERLOT: Bringing peer review to web-based educational resources. Journal of Technology and Teacher Education, 14(2), 313–323.Google Scholar
  17. Calvi, L. (1997). Navigation and disorientation: A case study. Journal of Educational Multimedia and Hypermedia, 6(3/4), 305–320.Google Scholar
  18. Caws, C., Friesen, N., & Beaudoin, M. (2006). A new learning object repository for language learning: Methods and possible outcomes. Interdisciplinary Journal of Knowledge and Learning Objects, 2, 112–124.Google Scholar
  19. Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293–332.CrossRefGoogle Scholar
  20. Chenail, R. J. (2004). When Disney meets the research park: Metaphors and models for engineering an online learning community of tomorrow. Internet and Higher Education, 7(2), 107–121.CrossRefGoogle Scholar
  21. Chi, M. T. H., & Bassok, M. (1989). Learning from examples via self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 251–282). Hillsdale, NJ: Erlbaum Associates.Google Scholar
  22. Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education and training trial of online curriculum content with Indigenous students, pp. 1–14.Google Scholar
  23. Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of education and training trial of online curriculum content with ESL students, pp. 1–16.Google Scholar
  24. Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning objects to enhance both face-to-face and distance e-learning environments. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved August 3, 2005 from http://ijklo.org/Volume1/v1p033-054Cochrane.pdf.
  25. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 453–494). Hillsdale, NJ: Erlbaum Associates.Google Scholar
  26. Convertini, V. C., Albanese, D., Marengo, A., Marengo, V., & Scalera, M. (2006). The OSEL taxonomy for the classification of learning objects. Interdisciplinary Journal of Knowledge and Learning Objects, 2, 125–138.Google Scholar
  27. Docherty, C., Hoy, D., Topp, H., & Trinder, K. (2005). E-Learning techniques supporting problem based learning in clinical simulation. International Journal of Medical Informatics, 74(7–8), 527–533.CrossRefGoogle Scholar
  28. Downes, S. (2003). Design and reusability of learning objects in an academic context: A new economy of education? USDLA, 17(1). Retrieved June 1, 2007 from http://www.usdla.org/html/journal/JAN03_Issue/article01.html.
  29. Field, A. (2005). Discovering statistics using SPSS (2nd ed.). Thousand Oaks, CA: SAGE Publications.Google Scholar
  30. Friesen, N., & Anderson, T. (2004). Interaction for lifelong learning. British Journal of Educational Technology, 35(6), 679–687.CrossRefGoogle Scholar
  31. Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets in the lesson planning of pre-service mathematics teachers. Journal of Computers in Mathematics and Science Teaching, 22(4), 323–344.Google Scholar
  32. Gadanidis, G., Sedig, K., & Liang, H. (2004). Designing online mathematical investigation. Journal of Computers in Mathematics and Science Technology, 23(3), 275–298.Google Scholar
  33. Guadagnoli, E., & Velicer, W. (1988). Relation of sample size to the stability of component patters. Psychological Bulletin, 103, 265–275.CrossRefGoogle Scholar
  34. Hanna, L., Risden, K., Czerwinski, M., & Alexander, K. J. (1999). The Role of usability in designing children’s computer products. In A. Druin (Ed.), The design of children’s technology. San Francisco: Morgan Kaufmann Publishers, Inc.Google Scholar
  35. Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90(3), 414–434.CrossRefGoogle Scholar
  36. Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional Sciences and Technology, 8(1). Retrieved June 1, 2007 from http://www.usq.edu.au/electpub/e-jist/docs/vol8_no1/fullpapers/eval_learnobjects_school.htm.
  37. Holzinger, A. (2004). Rapid prototyping for virtual medical campus interface. IEEE Software, 21(1), 92–99.CrossRefGoogle Scholar
  38. Howard-Rose, D., & Harrigan, K. (2003). CLOE learning impact studies lite: Evaluating learning objects in nine Ontario university courses. Retrieved July 3, 2007 from http://cloe.on.ca/documents/merlotconference10.doc.
  39. Jonassen, D., & Churchhill, D. (2004). Is there a learning orientation in learning objects? International Journal on E-Learning, 3(2), 32–41.Google Scholar
  40. Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995). Using metacognitive theories to design user interfaces for computer-based learning. Educational Technology, 35(4), 12–22.Google Scholar
  41. Kay, R., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-component model. Interdisciplinary Journal of Knowledge and Learning Objects, 2005(1), 229–254.Google Scholar
  42. Kay, R. H., & Knaack, L. (2007a). Evaluating the learning in learning objects. Open Learning, 22(1), 5–28.CrossRefGoogle Scholar
  43. Kay, R. H., & Knaack, L. (2007b). Teacher evaluation of learning objects in middle and secondary school classrooms. Retrieved November 1, 2007 from http://faculty.uoit.ca/kay/papers/LOES_Teacher_2007.doc.
  44. Kennedy, D. M., & McNaught, C. (1997). Design elements for interactive multimedia. Australian Journal of Educational Technology, 13(1), 1–22.Google Scholar
  45. Kenny, R. F., Andrews, B. W., Vignola, M. V., Schilz, M. A., & Covert, J. (1999). Towards guidelines for the design of interactive multimedia instruction: Fostering the reflective decision-making of preservice teachers. Journal of Technology and Teacher Education, 7(1), 13–31.Google Scholar
  46. Kester, L., Lehnen, C., Van Gerven, P. W. M., & Kirschner, P. A. (2006). Just-in-time schematic supportive information presentation during cognitive skill acquisition. Computers in Human Behavior, 22(1), 93–116.CrossRefGoogle Scholar
  47. Kline, P. (1999). The handbook of psychological testing (2nd ed.). London: Routledge.Google Scholar
  48. Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fractions: A model of affordances. Computers and Education, 45(2), 245–265.CrossRefGoogle Scholar
  49. Koohang, A., & Du Plessis, J. (2004). Architecting usability properties in the e-learning instructional design process. International Journal on E-Learning, 3(3), 38–44.Google Scholar
  50. Koppi, T., Bogle, L., & Bogle, M. (2005). Learning objects, repositories, sharing and reusability. Open Learning, 20(1), 83–91.CrossRefGoogle Scholar
  51. Kramarski, B., & Zeichner, O. (2001). Using technology to enhance mathematical reasoning: Effects of feedback and self-regulation learning. Education Media International, 38(2/3), 77–82.Google Scholar
  52. Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and implications for content development. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved June 1, 2007 from http://ijklo.org/Volume1/v1p001-022Krauss.pdf.
  53. Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environment based on recreational learning objects. Computers and Education, 48(4), 618–641.CrossRefGoogle Scholar
  54. Lim, C. P., Lee, S. L., & Richards, C. (2006). Developing interactive learning objects for a computing mathematics models. International Journal on E-Learning, 5(2), 221–244.Google Scholar
  55. Lin, A., & Gregor, S. (2006). Designing websites for learning and enjoyment: A study of museum experiences. International Review of Research in Open and Distance Learning, 7(3), 1–21.Google Scholar
  56. Littlejohn, A. (2003). Issues in Reusing Online Resources. Journal of Interactive Media in Education, 1, Special Issue on Reusing Online Resources. Retrieved July 1, 2005 from www-jime.open.ac.uk/2003/1/.
  57. Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology, Research and Development, 53(1), 5–21.CrossRefGoogle Scholar
  58. MacDonald, C. J., Stodel, E., Thompson, T. L., Muirhead, B., Hinton, C., Carson, B., et al. (2005). Addressing the eLearning contradiction: A collaborative approach for developing a conceptual framework learning object. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved August 2, 2005 from http://ijklo.org/Volume1/v1p079-098McDonald.pdf.
  59. Madhumita, & Kumar, K.L., (1995). Twenty-one guidelines for effective instructional design. Educational Technology, 35(3), 58–61.Google Scholar
  60. Maslowski, R., & Visscher, A. J. (1999). Formative evaluation in educational computing research and development. Journal of Research on Computing in Education, 32(2), 239–255.Google Scholar
  61. Mayer, R., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning and Instruction, 12, 107–119.CrossRefGoogle Scholar
  62. McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Technology and Distance Learning, 1(9). Retrieved August 5, 2005 from http://www.itdl.org/Journal/Sep_04/article02.htm.
  63. McGreal, R., Anderson, T., Babin, G., Downes, S., Friesen, N., Harrigan, K., et al. (2004). EduSource: Canada’s learning object repository network. International Journal of Instructional Technology and Distance Learning, 1(3). Retrieved July 24, 2005 from http://www.itdl.org/Journal/Mar_04/article01.htm.
  64. Muzio, J. A., Heins, T., & Mundell, R. (2002). Experiences with reusable e-learning objects from theory to practice. Internet and Higher Education, 2002(5), 21–34.CrossRefGoogle Scholar
  65. Nesbit, J., & Belfer, K. (2004). Collaborative evaluation of learning objects. In R. McGreal (Ed.), Online education using learning objects (pp. 138–153). New York: RoutledgeFalmer.Google Scholar
  66. Nesbit, J., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3). Retrieved July 1, 2005 from http://www.cjlt.ca/content/vol28.3/nesbit_etal.html.
  67. Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.Google Scholar
  68. Nurmi, S., & Jaakkola, T. (2005). Problems underlying the learning object approach. International Journal of Instructional Technology and Distance Learning, 2(11). Retrieved April 9, 2007 at http://www.itdl.org/Journal/Nov_05/article07.htm.
  69. Nurmi, S., & Jaakkola, T. (2006a). Effectiveness of learning objects in various instructional settings. Learning, Media, and Technology, 31(3), 233–247.CrossRefGoogle Scholar
  70. Nurmi, S., & Jaakkola, T. (2006b). Promises and pitfall of learning objects. Learning, Media, and Technology, 31(3), 269–285.CrossRefGoogle Scholar
  71. Ohl, T. M. (2001). An interaction-centric learning model. Journal of Educational Multimedia and Hypermedia, 10(4), 311–332.Google Scholar
  72. Oliver, R., & McLoughlin, C. (1999). Curriculum and learning-resources issues arising from the use of web-based course support systems. International Journal of Educational Telecommunications, 5(4), 419–435.Google Scholar
  73. Parrish, P. E. (2004). The trouble with learning objects. Educational Technology Research & Development, 52(1), 49–67.CrossRefGoogle Scholar
  74. Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19(3), 285–307.CrossRefGoogle Scholar
  75. Schell, G. P., & Burns, M. (2002). A repository of e-learning objects for higher education. e-Service Journal, 1(2), 53–64.Google Scholar
  76. Schoner, V., Buzza, D., Harrigan, K., & Strampel, K. (2005). Learning objects in use: ‘Lite’ assessment for field studies. Journal of Online Learning and Teaching, 1(1), 1–18.Google Scholar
  77. Sedig, K., & Liang, H (2006). Interactivity of visual mathematical representations: Factors affecting learning and cognitive processes. Journal of Interactive Learning Research. 17(2), 179–212.Google Scholar
  78. Siqueira, S. W. M., Melo, R. N., & Braz, M. H. L. B. (2004). Increasing the semantics of learning objects. International Journal of Computer Processing of Oriental Languages, 17(1), 27–39.CrossRefGoogle Scholar
  79. Sosteric, M., & Hesemeier, S. (2002). When is a learning object not an object: A first step towards a theory of learning objects. International Review of Research in Open and Distance Learning, 3(2), 1–16.Google Scholar
  80. Stevens, J. P. (1992). Applied multivariate statistics for the social science applications (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  81. Sweller, J. (1988). Cognitive load during problem solving: effects on learning. Cognitive Science, 12, 257–285.CrossRefGoogle Scholar
  82. Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.CrossRefGoogle Scholar
  83. Vacik, H., Wolfslehner, B., Spork, J., & Kortschak, E. (2006). The use of COCOON in teaching silviculture. Computers and Education, 47(3), 245–259.CrossRefGoogle Scholar
  84. Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25(3), 1–8.Google Scholar
  85. Van Gerven, P. W. M., Paas, F., & Tabbers, H. K. (2006). Cognitive aging and computer-based instructional design: Where do we go from here? Educational Psychology Review, 18(2), 141–157.CrossRefGoogle Scholar
  86. Van Merriënboer, J. J. G., & Ayres, P. (2005). Research on cognitive load theory and its design implications for e-learning. Education Theory, Research and Development, 53(3), 1042–1629.Google Scholar
  87. Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.Google Scholar
  88. Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementation and evaluation of a course concept based on reusable learning objects. Journal of Educational Computing and Research, 28(4), 355–372.CrossRefGoogle Scholar
  89. Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the limitations of learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 507–521.Google Scholar
  90. Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. Retrieved July 1, 2005 from http://reusability.org/read/chapters/williams.doc.
  91. Windschitl, M., & Andre, T. (1998). Using computer simulations to enhance conceptual change: The roles of constructivist instruction and student epistemological beliefs. Journal of Research in Science Teaching, 35(2), 145–160.CrossRefGoogle Scholar
  92. Zammit, K. (2000). Computer icons: A picture says a thousand words. Or does it? Journal of Educational Computing Research, 23(2), 217–231.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2008

Authors and Affiliations

  1. 1.Faculty of EducationUniversity of Ontario Institute of TechnologyOshawaCanada

Personalised recommendations