Abstract
The purpose of this study was to identify and validate items applicable to evaluating online courses at the postsecondary level. Items were derived from a review of the literature. Four judges rated the similarity of the items by making pair-wise comparisons utilizing multidimensional scaling (MDS). The study consisted of five stages. Stage I involved identifying items. Stage II involved validating items. Stage III involved sampling. Stage IV involved developing an online MDS instrument accommodating the 4,851 ([99 * 98]/2) pair-wise comparisons and rating them. Stage V involved data collection and analysis. The results of the MDS study indicated a three-dimensional solution (Accessibility, Adaptability, and Clarity of Communication) as the appropriate model for analysis. The three-dimensional solution indicated .24222 for the fit values of STRESS and .74 for R-squared. Four clusters were identified as Contextual Accommodation, Instructional Access, Guided Learning, and Organizational Clarity.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
American Psychological Association, & National Council on Measurement in Education (1985). Standards for educational and psychological testing. Washington: American Psychological Association, Inc.
Anderson, T. (2004). Teaching in online learning context. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 273–294). Athabasca: Athabasca University.
Anderson, T. (2004). Toward a theory of online learning. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 33–60). Athabasca: Athabasca University.
Ardovino, J., Hollingsworth, J., & Ybarra, S. (2000). Multiple measures. Thousand Oaks: Corwin Press, Inc.
Bastiaens, T., Boon, J., & Martens, R. (2003). Evaluating integrated e-learning. In W. Jochems, V. J. Merrienboer, & R. Koper (Eds.), Integrated e-learning (pp. 187–198). New York: RoutledgeFalmer.
Belanger, F., & Jordan, D. (2000). Evaluating and implementation of distance learning: Technologies, tools, and techniques. Hershey: Idea Group Publishing.
Childress, R., Heaton, L. A., & Pauley, R. (2002). Quality control for online graduate course delivery: A case study. Computers in the School, 19(3/4), 103–114.
Ciavarelli, A. (2003). Assessing the quality of online instruction: Integrating instructional quality and web usability assessments (Report No. CG032657). (ERIC Document Reproduction Service No. ED480084).
Davison, M. R. (1983). Multidimensional scaling. Malabar, FL: Krieger Publishing Company.
Davey, L. P. (2002). The e-volving practitioner: A heuristic formative evaluation of an online course based on an action research methodology (Mater thesis, Concordia University in Canada, 2002). Master Abstracts International, 41, 374.
Ebel, L. R., & Frisbie, A. D. (1991). Essentials of educational measurement (5th ed.). Englewood Cliff: Prentice Hall.
Fabro, K. R., & Garrison, D. R. (1998). Computer conferencing and higher-order learning. Indian Journal of Open Learning, 7(1), 41–54.
Garrison, D. R. (1991). Critical thinking in adult education: A conceptual model for developing critical thinking in adult learners. International Journal of Lifelong Education, 10(4), 287–303.
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical thinking in text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2), 87–105.
Gunawardena, C. N. (2001). Reflections on evaluating online learning and teaching. In E. Burge (Ed.),Using learning technologies; international perspectives on practice (pp. 115–124). New York: RoutledgeFalmer.
Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality guidelines for online courses: the development of an instrument to audit online units (Report No. IR021463). Australia; Western Australia. (Eric Document Reproduction Service No. ED467 943).
Lindeman, H. R. (1967). Educational masurement. Glenview: Scott, Foresman, C1979.
McLoughlin, C., & Luca, J. (2001). A learner-centered approach to developing team skills through web-based learning and assessment. British Journal of Educational Technology, 33(5), 571–582.
McPeck, J. (1990). Teaching critical thinking. New York: Routledge.
Messick, S. (1988). Validity. In R. L. Linn (Ed.). Measurement (3rd ed.) (pp. 13–103). New York: Macmillan Publishing Company.
Onay, Z. (2002). Leveraging distance education through the Internet: A paradigm shift in higher education. In R. Discenza, C. Howard, & K. Schenk (Eds.), The design & management of effective distance learning programs (pp. 233–261). Hershey: Idea group publishing.
Pedone, D. M. (2003). A qualitative analysis of student learning experiences in online community college undergraduate education courses (Doctoral dissertation, University of Central Florida, 2003). Dissertation Abstracts International, 64, 3576.
Rovai, A. P. (2003). A practical framework for evaluating online distance education programs. Internet and Higher Education, 6(2), 109–124.
Schiffman, S. S., Reynolds, M. L., & Yong, F. W. (1981). Introduction to multidimensional scaling: Theory, methods, and applications. San Diego: Academic Press.
Sireci, S. G., & Davison, M. L. (2000). Multidimensional scaling. In H. E. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 324–352). New York: Academic Press.
Sireci, S. G., Rogers, H. J., Swaminathan, H. K., Meara, K., & Robin, F. (2000). An external evaluation of the 1996 grade 8 NAEP science framework. In N. S. Raju, J. W. Pellegrino, M. W. Bertenthal, K. J. Mitchell, & L. R. Jones (Eds.), Grading the nation’s report card (pp. 74–100). Washington: National Academy Press.
Stalans, L. J. (1995). Multidimensional scaling. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 137–168). Washington: American Psychological Association.
Statistical Package for the Social Sciences (Version 14) [Computer software] (2000). Chicago: SPSS Inc.
Stewart, I. (2001). Development and validation of an instrument for student evaluation of Web-based instruction (Doctoral dissertation, University of Nevada, 2001). Dissertation Abstracts International, 63, 1250.
Streveler, A. R., Miller, L. R., & Boyd, M. T. (2001). Using an on-line tool to investigate chemical engineering seniors’ concept of the design process (Report No. HE 033 966). (ERIC Document Reproduction Service No. ED452 751).
Tagg, A. C., & Dickenson, J. A. (1995). Tutor messaging and its effectiveness in encouraging student participation on computer conferences. Journal of Distance Education, 10(2), 33–55.
Thorndike, R. M. (1997). Measurement and evaluation in psychology and education (6th ed.). Upper Saddle River: Prentice Hall.
Wagner, J. G. (2001). Assessing online learning (Report No. CE 081 359). Reston, VA: National Business Education Association. (Eric Document Reproduction Service No. ED449 386).
Young, F. W. (1985). Multidimensional scaling. Retrieved March 19, 2006 from http://www.forrest.psych.unc.edu/teaching/p208a/mds/mds.html
Young, F. W., & Lewyckyj, R. (1979). ALSCAL-4 user’s guide. Carrboro: Data Analysis and Theory Associates.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Seok, S. Item validation of online postsecondary courses: rating the proximity between similarity and dissimilarity among item pairs (Validation study series I: multidimensional scaling). Education Tech Research Dev 57, 665–684 (2009). https://doi.org/10.1007/s11423-007-9072-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-007-9072-3