Skip to main content
SpringerLink
Log in

Assessing learning, quality and engagement in learning objects: the Learning Object Evaluation Scale for Students (LOES-S)

  • Research Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Learning objects are interactive web-based tools that support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners. Research on the impact, effectiveness, and usefulness of learning objects is limited, partially because comprehensive, theoretically based, reliable, and valid evaluation tools are scarce, particularly in the K-12 environment. The purpose of the following study was to investigate a Learning Object Evaluation Scale for Students (LOES-S) based on three key constructs gleaned from 10 years of learning object research: learning, quality or instructional design, and engagement. Tested on over 1100 middle and secondary school students, the data generated using the LOES-S showed acceptable internal reliability, face validity, construct validity, convergent validity and predictive validity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Acovelli, M., & Gamble, M. (1997). A coaching agent for learners using multimedia simulations. Educational Technology, 37(2), 44–49.

    Google Scholar 

  • Adams, A., Lubega, J., Walmsley, S., & Williams, S. (2004). The effectiveness of assessment learning objects produced using pair programming. Electronic Journal of e-Learning, 2(2). Retrieved July 28, 2005 from http://www.ejel.org/volume-2/vol2-issue2/v2-i2-art1-adams.pdf.

  • Akpinar, Y., & Hartley, J. R. (1996). Designing interactive learning environments. Journal of Computer Assisted Learning, 12(1), 33–46.

    Article  Google Scholar 

  • Albanese, M. A., & Mitchell, S. A. (1993). Problem-based learning: A review of the literature on its outcomes and implementation issues. Academic Medicine, 68, 52–81.

    Article  Google Scholar 

  • Alonso, F., Lopez, G., Manrique, D., & Vines, J. M. (2005). An instructional model for web-based e-learning education with a blended learning process approach. British Journal of Educational Technology, 36(2), 217–235.

    Article  Google Scholar 

  • Baruque, L. B., & Melo, R. N. (2004). Learning theory and instructional design using learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 343–370.

    Google Scholar 

  • Baser, M. (2005). Promoting conceptual change through active learning using open source software for physic simulations. Australasian Journal of Educational Technology, 22(3), 336–354.

    Google Scholar 

  • Bennett, K., & McGee, P. (2005). Transformative power of the learning object debate. Open Learning, 20(1), 15–30.

    Article  Google Scholar 

  • Bradley, C., & Boyle, T. (2004). The design, development, and use of multimedia learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 371–389.

    Google Scholar 

  • Brown, A. L., & Palinscar, A. S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 393–451). Hillsdale, NJ: Erlbaum Associates.

    Google Scholar 

  • Brown, A. R., & Voltz, B. D. (2005). Elements of effective e-learning design. The International Review of Research in Open and Distance Learning, 6(1). Retrieved June 1, 2007 from http://www.irrodl.org/index.php/irrodl/article/view/217/300.

  • Bruner, J. (1983). Child’s talk. Learning to use language. Toronto, Canada: George J. McLeod Ltd.

    Google Scholar 

  • Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Brush, T., & Saye, J. (2001). The use of embedded scaffolds with hypermedia-supported student-centered learning. Journal of Educational Multimedia and Hypermedia, 10(4), 333–356.

    Google Scholar 

  • Buzzetto-More, N. A., & Pinhey, K. (2006). Guidelines and standards for the development of fully online learning objects. Interdisciplinary Journal of Knowledge and Learning Objects, 2006(2), 96–104.

    Google Scholar 

  • Cafolla, R. (2006). Project MERLOT: Bringing peer review to web-based educational resources. Journal of Technology and Teacher Education, 14(2), 313–323.

    Google Scholar 

  • Calvi, L. (1997). Navigation and disorientation: A case study. Journal of Educational Multimedia and Hypermedia, 6(3/4), 305–320.

    Google Scholar 

  • Caws, C., Friesen, N., & Beaudoin, M. (2006). A new learning object repository for language learning: Methods and possible outcomes. Interdisciplinary Journal of Knowledge and Learning Objects, 2, 112–124.

    Google Scholar 

  • Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293–332.

    Article  Google Scholar 

  • Chenail, R. J. (2004). When Disney meets the research park: Metaphors and models for engineering an online learning community of tomorrow. Internet and Higher Education, 7(2), 107–121.

    Article  Google Scholar 

  • Chi, M. T. H., & Bassok, M. (1989). Learning from examples via self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 251–282). Hillsdale, NJ: Erlbaum Associates.

    Google Scholar 

  • Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education and training trial of online curriculum content with Indigenous students, pp. 1–14.

  • Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of education and training trial of online curriculum content with ESL students, pp. 1–16.

  • Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning objects to enhance both face-to-face and distance e-learning environments. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved August 3, 2005 from http://ijklo.org/Volume1/v1p033-054Cochrane.pdf.

  • Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 453–494). Hillsdale, NJ: Erlbaum Associates.

    Google Scholar 

  • Convertini, V. C., Albanese, D., Marengo, A., Marengo, V., & Scalera, M. (2006). The OSEL taxonomy for the classification of learning objects. Interdisciplinary Journal of Knowledge and Learning Objects, 2, 125–138.

    Google Scholar 

  • Docherty, C., Hoy, D., Topp, H., & Trinder, K. (2005). E-Learning techniques supporting problem based learning in clinical simulation. International Journal of Medical Informatics, 74(7–8), 527–533.

    Article  Google Scholar 

  • Downes, S. (2003). Design and reusability of learning objects in an academic context: A new economy of education? USDLA, 17(1). Retrieved June 1, 2007 from http://www.usdla.org/html/journal/JAN03_Issue/article01.html.

  • Field, A. (2005). Discovering statistics using SPSS (2nd ed.). Thousand Oaks, CA: SAGE Publications.

    Google Scholar 

  • Friesen, N., & Anderson, T. (2004). Interaction for lifelong learning. British Journal of Educational Technology, 35(6), 679–687.

    Article  Google Scholar 

  • Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets in the lesson planning of pre-service mathematics teachers. Journal of Computers in Mathematics and Science Teaching, 22(4), 323–344.

    Google Scholar 

  • Gadanidis, G., Sedig, K., & Liang, H. (2004). Designing online mathematical investigation. Journal of Computers in Mathematics and Science Technology, 23(3), 275–298.

    Google Scholar 

  • Guadagnoli, E., & Velicer, W. (1988). Relation of sample size to the stability of component patters. Psychological Bulletin, 103, 265–275.

    Article  Google Scholar 

  • Hanna, L., Risden, K., Czerwinski, M., & Alexander, K. J. (1999). The Role of usability in designing children’s computer products. In A. Druin (Ed.), The design of children’s technology. San Francisco: Morgan Kaufmann Publishers, Inc.

    Google Scholar 

  • Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90(3), 414–434.

    Article  Google Scholar 

  • Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional Sciences and Technology, 8(1). Retrieved June 1, 2007 from http://www.usq.edu.au/electpub/e-jist/docs/vol8_no1/fullpapers/eval_learnobjects_school.htm.

  • Holzinger, A. (2004). Rapid prototyping for virtual medical campus interface. IEEE Software, 21(1), 92–99.

    Article  Google Scholar 

  • Howard-Rose, D., & Harrigan, K. (2003). CLOE learning impact studies lite: Evaluating learning objects in nine Ontario university courses. Retrieved July 3, 2007 from http://cloe.on.ca/documents/merlotconference10.doc.

  • Jonassen, D., & Churchhill, D. (2004). Is there a learning orientation in learning objects? International Journal on E-Learning, 3(2), 32–41.

    Google Scholar 

  • Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995). Using metacognitive theories to design user interfaces for computer-based learning. Educational Technology, 35(4), 12–22.

    Google Scholar 

  • Kay, R., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-component model. Interdisciplinary Journal of Knowledge and Learning Objects, 2005(1), 229–254.

    Google Scholar 

  • Kay, R. H., & Knaack, L. (2007a). Evaluating the learning in learning objects. Open Learning, 22(1), 5–28.

    Article  Google Scholar 

  • Kay, R. H., & Knaack, L. (2007b). Teacher evaluation of learning objects in middle and secondary school classrooms. Retrieved November 1, 2007 from http://faculty.uoit.ca/kay/papers/LOES_Teacher_2007.doc.

  • Kennedy, D. M., & McNaught, C. (1997). Design elements for interactive multimedia. Australian Journal of Educational Technology, 13(1), 1–22.

    Google Scholar 

  • Kenny, R. F., Andrews, B. W., Vignola, M. V., Schilz, M. A., & Covert, J. (1999). Towards guidelines for the design of interactive multimedia instruction: Fostering the reflective decision-making of preservice teachers. Journal of Technology and Teacher Education, 7(1), 13–31.

    Google Scholar 

  • Kester, L., Lehnen, C., Van Gerven, P. W. M., & Kirschner, P. A. (2006). Just-in-time schematic supportive information presentation during cognitive skill acquisition. Computers in Human Behavior, 22(1), 93–116.

    Article  Google Scholar 

  • Kline, P. (1999). The handbook of psychological testing (2nd ed.). London: Routledge.

    Google Scholar 

  • Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fractions: A model of affordances. Computers and Education, 45(2), 245–265.

    Article  Google Scholar 

  • Koohang, A., & Du Plessis, J. (2004). Architecting usability properties in the e-learning instructional design process. International Journal on E-Learning, 3(3), 38–44.

    Google Scholar 

  • Koppi, T., Bogle, L., & Bogle, M. (2005). Learning objects, repositories, sharing and reusability. Open Learning, 20(1), 83–91.

    Article  Google Scholar 

  • Kramarski, B., & Zeichner, O. (2001). Using technology to enhance mathematical reasoning: Effects of feedback and self-regulation learning. Education Media International, 38(2/3), 77–82.

    Google Scholar 

  • Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and implications for content development. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved June 1, 2007 from http://ijklo.org/Volume1/v1p001-022Krauss.pdf.

  • Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environment based on recreational learning objects. Computers and Education, 48(4), 618–641.

    Article  Google Scholar 

  • Lim, C. P., Lee, S. L., & Richards, C. (2006). Developing interactive learning objects for a computing mathematics models. International Journal on E-Learning, 5(2), 221–244.

    Google Scholar 

  • Lin, A., & Gregor, S. (2006). Designing websites for learning and enjoyment: A study of museum experiences. International Review of Research in Open and Distance Learning, 7(3), 1–21.

    Google Scholar 

  • Littlejohn, A. (2003). Issues in Reusing Online Resources. Journal of Interactive Media in Education, 1, Special Issue on Reusing Online Resources. Retrieved July 1, 2005 from www-jime.open.ac.uk/2003/1/.

  • Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology, Research and Development, 53(1), 5–21.

    Article  Google Scholar 

  • MacDonald, C. J., Stodel, E., Thompson, T. L., Muirhead, B., Hinton, C., Carson, B., et al. (2005). Addressing the eLearning contradiction: A collaborative approach for developing a conceptual framework learning object. Interdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved August 2, 2005 from http://ijklo.org/Volume1/v1p079-098McDonald.pdf.

  • Madhumita, & Kumar, K.L., (1995). Twenty-one guidelines for effective instructional design. Educational Technology, 35(3), 58–61.

  • Maslowski, R., & Visscher, A. J. (1999). Formative evaluation in educational computing research and development. Journal of Research on Computing in Education, 32(2), 239–255.

    Google Scholar 

  • Mayer, R., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning and Instruction, 12, 107–119.

    Article  Google Scholar 

  • McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Technology and Distance Learning, 1(9). Retrieved August 5, 2005 from http://www.itdl.org/Journal/Sep_04/article02.htm.

  • McGreal, R., Anderson, T., Babin, G., Downes, S., Friesen, N., Harrigan, K., et al. (2004). EduSource: Canada’s learning object repository network. International Journal of Instructional Technology and Distance Learning, 1(3). Retrieved July 24, 2005 from http://www.itdl.org/Journal/Mar_04/article01.htm.

  • Muzio, J. A., Heins, T., & Mundell, R. (2002). Experiences with reusable e-learning objects from theory to practice. Internet and Higher Education, 2002(5), 21–34.

    Article  Google Scholar 

  • Nesbit, J., & Belfer, K. (2004). Collaborative evaluation of learning objects. In R. McGreal (Ed.), Online education using learning objects (pp. 138–153). New York: RoutledgeFalmer.

    Google Scholar 

  • Nesbit, J., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3). Retrieved July 1, 2005 from http://www.cjlt.ca/content/vol28.3/nesbit_etal.html.

  • Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.

    Google Scholar 

  • Nurmi, S., & Jaakkola, T. (2005). Problems underlying the learning object approach. International Journal of Instructional Technology and Distance Learning, 2(11). Retrieved April 9, 2007 at http://www.itdl.org/Journal/Nov_05/article07.htm.

  • Nurmi, S., & Jaakkola, T. (2006a). Effectiveness of learning objects in various instructional settings. Learning, Media, and Technology, 31(3), 233–247.

    Article  Google Scholar 

  • Nurmi, S., & Jaakkola, T. (2006b). Promises and pitfall of learning objects. Learning, Media, and Technology, 31(3), 269–285.

    Article  Google Scholar 

  • Ohl, T. M. (2001). An interaction-centric learning model. Journal of Educational Multimedia and Hypermedia, 10(4), 311–332.

    Google Scholar 

  • Oliver, R., & McLoughlin, C. (1999). Curriculum and learning-resources issues arising from the use of web-based course support systems. International Journal of Educational Telecommunications, 5(4), 419–435.

    Google Scholar 

  • Parrish, P. E. (2004). The trouble with learning objects. Educational Technology Research & Development, 52(1), 49–67.

    Article  Google Scholar 

  • Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19(3), 285–307.

    Article  Google Scholar 

  • Schell, G. P., & Burns, M. (2002). A repository of e-learning objects for higher education. e-Service Journal, 1(2), 53–64.

    Google Scholar 

  • Schoner, V., Buzza, D., Harrigan, K., & Strampel, K. (2005). Learning objects in use: ‘Lite’ assessment for field studies. Journal of Online Learning and Teaching, 1(1), 1–18.

    Google Scholar 

  • Sedig, K., & Liang, H (2006). Interactivity of visual mathematical representations: Factors affecting learning and cognitive processes. Journal of Interactive Learning Research. 17(2), 179–212.

    Google Scholar 

  • Siqueira, S. W. M., Melo, R. N., & Braz, M. H. L. B. (2004). Increasing the semantics of learning objects. International Journal of Computer Processing of Oriental Languages, 17(1), 27–39.

    Article  Google Scholar 

  • Sosteric, M., & Hesemeier, S. (2002). When is a learning object not an object: A first step towards a theory of learning objects. International Review of Research in Open and Distance Learning, 3(2), 1–16.

    Google Scholar 

  • Stevens, J. P. (1992). Applied multivariate statistics for the social science applications (2nd ed.). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Sweller, J. (1988). Cognitive load during problem solving: effects on learning. Cognitive Science, 12, 257–285.

    Article  Google Scholar 

  • Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.

    Article  Google Scholar 

  • Vacik, H., Wolfslehner, B., Spork, J., & Kortschak, E. (2006). The use of COCOON in teaching silviculture. Computers and Education, 47(3), 245–259.

    Article  Google Scholar 

  • Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25(3), 1–8.

    Google Scholar 

  • Van Gerven, P. W. M., Paas, F., & Tabbers, H. K. (2006). Cognitive aging and computer-based instructional design: Where do we go from here? Educational Psychology Review, 18(2), 141–157.

    Article  Google Scholar 

  • Van Merriënboer, J. J. G., & Ayres, P. (2005). Research on cognitive load theory and its design implications for e-learning. Education Theory, Research and Development, 53(3), 1042–1629.

    Google Scholar 

  • Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementation and evaluation of a course concept based on reusable learning objects. Journal of Educational Computing and Research, 28(4), 355–372.

    Article  Google Scholar 

  • Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the limitations of learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 507–521.

    Google Scholar 

  • Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. Retrieved July 1, 2005 from http://reusability.org/read/chapters/williams.doc.

  • Windschitl, M., & Andre, T. (1998). Using computer simulations to enhance conceptual change: The roles of constructivist instruction and student epistemological beliefs. Journal of Research in Science Teaching, 35(2), 145–160.

    Article  Google Scholar 

  • Zammit, K. (2000). Computer icons: A picture says a thousand words. Or does it? Journal of Educational Computing Research, 23(2), 217–231.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Education, University of Ontario Institute of Technology, 2000 Simcoe St. North, Oshawa, ON, Canada, L1H 7L7

    Robin H. Kay & Liesel Knaack

Authors
  1. Robin H. Kay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liesel Knaack
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robin H. Kay.

Appendices

 

 

Appendix A List of learning objects used in the study
Full size table

 

 

Appendix B Learning object evaluation survey—students
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kay, R.H., Knaack, L. Assessing learning, quality and engagement in learning objects: the Learning Object Evaluation Scale for Students (LOES-S). Education Tech Research Dev 57, 147–168 (2009). https://doi.org/10.1007/s11423-008-9094-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-008-9094-5

Keywords

Search

Navigation