Designing for Quality?

  • Allison LittlejohnEmail author
  • Nina Hood
Part of the SpringerBriefs in Education book series (BRIEFSEDUCAT)


There are significant complexities in interpreting and measuring quality in MOOCs. In this chapter, we examine experts’ perceptions of how to measure quality in MOOCs, using empirical data we gathered through conversations with MOOC specialists. In their experience, while data can be helpful in understanding quality, the metrics measured are shaped by underpinning assumptions and biases. In conventional education, it is assumed that the learner wants to follow a course pathway and complete a course. However, this assumption may not be valid in a MOOC. Quality data might not capture the underlying goals and intentions of MOOC learners. Therefore, it is difficult to measure whether or not a learner has achieved his or her goals. We stress the need to explore quality metrics from the learner’s point of view and to encompass the variability in motivations, needs and backgrounds, which shape conceptions of quality for individuals.



The authors wish to thank Vasudha Chaudhari of The Open University for comments and for proofing this chapter.


  1. Adamopoulos, A. (2013). What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. Paper presented at the Thirty-Fourth International Conference on Information Systems, Milan, Italy. Retrieved from
  2. Alario-Hoyos, C., Perez-Sanagustin, M., Cormier, D., & Delgado-Kloos, C. (2014). Proposal for a conceptual framework for educators to describe and design MOOCs. Journal of Universal Computer Science, 20(1), 6–23.Google Scholar
  3. Amo, D. (2013, November). MOOCs: Experimental approaches for quality in pedagogical and design fundamentals. Paper presented at TEEM’13, Salamanca, Spain.Google Scholar
  4. Bayne, S., & Ross, J. (2014). MOOC pedagogy. In P. Kim (Ed.), Massive open online courses: The MOOC revolution (pp. 23–45). New York, NY: Routledge.Google Scholar
  5. Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the demcrative deficit in educational research. Educational Theory, 57(1), 1–22.CrossRefGoogle Scholar
  6. Biggs, J. (1993). From theory to practice: A cognitive systems approach. Higher Education Research & Development, 12(1), 73–85.CrossRefGoogle Scholar
  7. Chandrasekaran, M., Ragupathi, K., Kan, M., & Tan, B. (2015, December). Towards feasible instructor intervention in MOOC discussion forums. Paper presented at the Thirty-Sixth International Conference on Information Systems, Fort Worth, TX.Google Scholar
  8. Coetzee, D., Lim, S., Fox, A., Hartmann, B., & Hearst, M. A. (2015). Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer- Supported Cooperative Work and Social Computing (CSCW), Vancouver, Canada (pp. 1139–1152). New York, NY: ACM.Google Scholar
  9. Conole, G. (2013). MOOCs as disruptive technologies: Strategies for enhancing the learner experience and quality of MOOCs. RED—Revista de Educación a Distancia, 39. Retrieved from
  10. Daradoumis, T., Bassi, R., Xhafa, F., & Caballé, S. (2013, October). A review on massive e-learning (MOOC) design, delivery and assessment. In 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC) (pp. 208–213). Piscataway, NJ: IEEE.Google Scholar
  11. DeBoer, J., Ho, A., Stump, G. S., & Breslow, L. (2014). Changing “course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74–84. Scholar
  12. Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrolment physics class. Science, 332(6031), 862–864.CrossRefGoogle Scholar
  13. Dillenbourg, P., Fox, A., Kirchner, C., Mitchell, J., & Wirsing, M. (2014). Massive open online courses: Current state and perspectives. Manifesto from Dagstuhl Perspectives Workshop. Scholar
  14. Downes, S. (2013, April 24). The quality of massive open online courses. Retrieved from
  15. Fenwick, T. (2015). Sociomateriality and learning: A critical approach. In D. Scott & E. Hargreaves (Eds.), The SAGE handbook of learning (online). London: SAGE.Google Scholar
  16. Gibbs, G. (2010). Dimensions of quality. York, UK: The Higher Education Academy.Google Scholar
  17. Gillani, N., & Eynon, R. (2014). Communication patterns in massively open online courses. Internet and Higher Education, 23, 18–26.CrossRefGoogle Scholar
  18. Grover, S., Franz, P., Schneider, E., & Pea, R. (2013). The MOOC as distributed intelligence: Dimension of a framework an evaluation of MOOCs. Paper presented at the 10th Annual International Conference on Computer Supported Collaborative Learning, Madison, WI. Retrieved from
  19. Grunewald, F., Meinel, C., Totschnig, M., & Willems, C. (2013). Designing MOOCs for the support of multiple learning styles. In Conference Proceedings from EC-TEL 2013, LNCS (pp. 371–382). Berlin, Germany: Springer.Google Scholar
  20. Guardia, L., Maina, M., & Sangra, A. (2013). MOOC design principles: A pedagogical approach from the learner’s perspective. eLearning Papers, 33, 1–5.Google Scholar
  21. Guo, P., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In Proceedings of the First ACM conference on Learning @ Scale Conference (pp. 41–50). New York, NY: ACM.Google Scholar
  22. Hew, K. (2014). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS? British Journal of Educational Technology, 47(2), 320–342. Scholar
  23. Hood, N., & Littlejohn, A. (2016a). MOOC Quality: A call for new quality measures. Journal of Learning for Development, 3(3), 28–42
  24. Hood, N., & Littlejohn, A. (2016b). Quality in MOOCs, surveying the terrain (Commonwealth for Learning Report).
  25. Howley, I., Mayfield, E., & Rosé, C. P. (2013). Linguistic analysis methods for studying small groups. In C. Hmelo-Silver, A. O’Donnell, C. Chan, & C. Chin (Eds.), The international handbook of collaborative learning (pp. 184–202). New York, NY: Routledge.Google Scholar
  26. Illeris, K. (2007). How we learn: Learning and non-learning in school and beyond. London: Routledge.Google Scholar
  27. Istrate, O., & Kestens, A. (2015, April). Developing and monitoring a MOOC: The IFRC experience. Paper presented at the 11th International Scientific Conference eLearning and Software for Education, Bucharest, Romania. Retrieved from
  28. Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16(3), 341–358.CrossRefGoogle Scholar
  29. Kanwar, A. (2013, October 16). Quality vs. quantity: Can technology help? Opening keynote presentation at the 25th ICDE World Conference, Tianjin, China.Google Scholar
  30. Kay, J., Reimann, P., Diebold, E., & Kummerfeld, B. (2013). MOOCs: So many learners, so much potential …. IEEE Intelligent Systems, 28(3), 70–77 (Kizilcic et al., 2013).CrossRefGoogle Scholar
  31. Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. LAK’13 Leuven, Belgium.Google Scholar
  32. Kling, R., & Courtright, C. (2003). Group behavior and learning in electronic forums: A sociotechnical approach. The Information Society, 19, 221–235.CrossRefGoogle Scholar
  33. Kolowich, S. (2013, March 18). The professors behind the MOOC hype. The Chronicle of Higher Education. Retrieved from
  34. Lackner, E., Ebner, M., & Khalil, M. (2015). MOOCs as granular systems: Design patterns to foster participant activity. eLearning Papers, 42, 28–37.Google Scholar
  35. Lin, Y.-L., Lin, H.-W., & Hung, T.-T. (2015). Value hierarchy for massive open online courses. Computers in Human Behaviour, 53, 408–418.CrossRefGoogle Scholar
  36. Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self-regulated learning in MOOCs. The Internet and Higher Education, 29, 40–48. Scholar
  37. Mak, S., Williams, R., & Mackness, J. (2010). Blogs and forums as communication and learning tools in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg (Eds.), In Proceedings of the 7th International Conference on Networked Learning 2010 (pp. 275–284). Lancaster, UK: Lancaster University. Retrieved from https://
  38. Mamgain, N., Sharma, A., & Goyal, P. (2014). Learner’s perspective on video-viewing features offered by MOOC providers: Coursera and edX. Paper presented at the 2014 IEEE International Conference on MOOC, Innovation and Technology in Education (MITE).
  39. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80, 77–83.CrossRefGoogle Scholar
  40. Morozov, E. (2014). The planning machine. The New Yorker, 13 October (
  41. Morrison, D. (2014, January 18). Need-to-know-news: MOOC mentors for hire, Coursera’s MOC$s, edX shares MOOC data and more. Online Learning Insights. Retrieved from
  42. Perna, L., Ruby, A., Boruch, R., Wang, N., Scull, J., Ahmad, S., et al. (2014). Moving through MOOCs: Understanding the progression of users in massive open online courses. Education Researcher, 43(9), 421–432.CrossRefGoogle Scholar
  43. Rodriguez, C. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning. Retrieved from
  44. Ross, J., Sinclair, C., Knox, J., & Macleod, H. (2014). Teacher experiences and academic identity: The missing components of MOOC pedagogy. Journal of Online Learning and Teaching, 10(1), 57.Google Scholar
  45. Scagnoli, N. (2012). Thoughts on instructional design for MOOCs. Retrieved from
  46. Selwyn, N. (2014). Distrusting educational technology. London and New York: Routledge.Google Scholar
  47. Selwyn, N. (2016). Is technology good for education. Cambridge, UK: Polity Books.Google Scholar
  48. Shah, D. (2016). By the numbers: MOOCS in 2016. [Online]. Retrieved from:
  49. Sharples, M., & Domingue, J. (2016, September). The blockchain and kudos: A distributed system for educational record, reputation and reward. In European Conference on Technology Enhanced Learning (pp. 490–496). Springer International Publishing.CrossRefGoogle Scholar
  50. Sinha, T., Li, N., Jermann, P., & Dillenbourg, P. (2014). Capturing “attrition intensifying” structural traits from didactic interaction sequences of MOOC learners. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses (pp. 42–49). Retrieved from
  51. Tabba, Y., & Medouri, A. (2013). LASyM: A learning analytics system for MOOCs. International Journal of Advanced Computer Science and Applications, 4(5), 113–119.Google Scholar
  52. Tyler, K. (1939). Recent developments in radio education. The English Journal, 28(3), 193–199.CrossRefGoogle Scholar
  53. Warburton, S., & Mor, Y. (2015). Double loop design: Configuring narratives, patterns and scenarios in the design of technology enhanced learning. In M. Maina et al. (Eds.), The art and science of learning design (pp. 93–104). Rotterdam, The Netherlands: Sense Publishers.CrossRefGoogle Scholar
  54. Wen, M., Yang, D., & Rosé, C. P. (2014a). Linguistic reflections of student engagement in massive open online courses. In Proceedings of the International Conference on Weblogs and Social Media. Retrieved from
  55. Wen, M., Yang, D., & Rosé, C. P. (2014b). Sentiment analysis in MOOC discussion forums: What does it tell us? In Proceedings of Educational Data Mining. Retrieved from
  56. Yang, D., Wen, M., Kumar, A., Xing, E., & Rosé, C. (2014). Towards an integration of text and graph clustering methods as a lens for studying social interaction in MOOCs. International Review of Research in Open and Distributed Learning, 15(5). Retrieved from

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Open UniversityMilton KeynesUK
  2. 2.University of AucklandAucklandNew Zealand

Personalised recommendations