Skip to main content
SpringerLink
Log in

Challenges in Assessments of Soft Skills: Towards Unobtrusive Approaches to Measuring Student Success

  • Chapter
  • First Online:
Unobtrusive Observations of Learning in Digital Environments

Part of the book series: Advances in Analytics for Learning and Teaching ((AALT))

Abstract

Rapid technological advances, coupled with globalization, have resulted in a changing economy, requiring graduates and students to master not only technical and subject knowledge but also broad, transferable skills for workplace readiness. However, assessing these essential soft skills and competencies beyond the cognitive domain has often relied on questionnaires, surveys and other self-rated scales, which are subjective, often obtrusive in nature, subject to response biases, and lack scalability. In contrast, the pervasive use of educational technology has provided researchers with the opportunity to unobtrusively collect enormous amounts of factual learners’ data which has the potential to overcome some of the challenges with questionnaire-based approaches. These unobtrusive measures increase the possibilities of passively evaluating skill acquisition and supporting learners by personalizing learning according to their needs. This chapter outlines a multi-tiered case study and proposes a novel blended methodology, marrying measurement models and learning analytics techniques to mitigate some of these challenges and unobtrusively measure leadership skills in a workplace learning context. Using learners’ reflection assessments, several leadership-defining course objectives were quantified, and their progress was assessed over time. The implications of this evidence-based assessment approach, informed by theory, to measure and model soft skills acquisition are further discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For convenience and to minimize the multiplicity of terms used to describe the same skills and competencies, we refer to them as soft skills throughout this chapter.

References

  • Amagoh, F. (2009). Leadership development and leadership effectiveness. Management Decision, 47(6), 989–999. https://doi.org/10.1108/00251740910966695

    Article  Google Scholar 

  • Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, Nay-saying, and going to extremes: Black-White differences in response styles. Public Opinion Quarterly, 48(2), 491–509. https://doi.org/10.1086/268845

    Article  Google Scholar 

  • Barthakur, A., Joksimovic, S., Kovanovic, V., Corbett, F. C., Richey, M., & Pardo, A. (2022a). Assessing the sequencing of learning objectives in a study program using evidence-based practice. Assessment & Evaluation in Higher Education, 1–15. https://doi.org/10.1080/02602938.2022.2064971

  • Barthakur, A., Joksimovic, S., Kovanovic, V., Ferreira Mello, R., Taylor, M., Richey, M., & Pardo, A. (2022b). Understanding depth of reflective writing in workplace learning assessments using machine learning classification. IEEE Transactions on Learning Technologies, 1. https://doi.org/10.1109/TLT.2022.3162546

  • Barthakur, A., Kovanovic, V., Joksimovic, S., Zhang, Z., Richey, M., & Pardo, A. (2022c). Measuring leadership development in workplace learning using automated assessments: Learning analytics and measurement theory approach. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13218

  • Bergner, Y. (2017). Measurement and its uses in learning analytics. In C. Lang, G. Siemens, A. Wise, & D. Gasevic (Eds.), Handbook of learning analytics (1st ed., pp. 35–48). Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17.003

  • Buckingham Shum, S., Sándor, Á., Goldsmith, R., Bass, R., & McWilliams, M. (2017). Towards reflective writing analytics: Rationale, methodology and preliminary results. Journal of Learning Analytics, 4(1), 10.18608/jla.2017.41.5.

    Article  Google Scholar 

  • Burke, V., & Collins, D. (2005). Optimising the effects of leadership development programmes: A framework for analysing the learning and transfer of leadership skills. Management Decision, 43(7/8), 975–987. https://doi.org/10.1108/00251740510609974

    Article  Google Scholar 

  • Care, E., Griffin, P., & Wilson, M. (Eds.). (2018). Assessment and teaching of 21st century skills: Research and applications. Springer. https://doi.org/10.1007/978-3-319-65368-6

    Book  Google Scholar 

  • Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. In Partnership for 21st century skills. Partnership for 21st Century Skills. https://eric.ed.gov/?id=ED519465

  • Chauhan, A. (2014). Massive open online courses (MOOCS): Emerging trends in assessment and accreditation (p. 12).

    Google Scholar 

  • Corbett, F. (2021). Emergence of the connectivist leadership paradigm: A grounded theory study in the Asia region. Theses and dissertations. https://digitalcommons.pepperdine.edu/etd/1194

  • Corbett, F., & Spinello, E. (2020). Connectivism and leadership: Harnessing a learning theory for the digital age to redefine leadership in the twenty-first century. Heliyon, 6(1), e03250. https://doi.org/10.1016/j.heliyon.2020.e03250

    Article  Google Scholar 

  • Dawson, S., & Hubball, H. (2014). Curriculum analytics: Application of social network analysis for improving strategic curriculum decision-making in a research – Intensive university. Learning Inquiry, 2(2), 59–74.

    Article  Google Scholar 

  • Dawson, S., & Siemens, G. (2014). Analytics to literacies: The development of a learning analytics framework for multiliteracies assessment. International Review of Research in Open and Distance Learning, 15, 284–305. https://doi.org/10.19173/irrodl.v15i4.1878

    Article  Google Scholar 

  • Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. In Proceedings of the 9th international conference on learning analytics & knowledge (pp. 446–455). https://doi.org/10.1145/3303772.3303784

  • Densten, I., & Gray, J. (2001). Leadership development and reflection: What is the connection? (p. 15). http://lst-iiep.iiep-unesco.org/cgi-bin/wwwi32.exe/[in=epidoc1.in]/?T2000=013168/(100). https://doi.org/10.1108/09513540110384466

  • Drachsler, H., & Goldhammer, F. (2020). Learning analytics and eAssessment – Towards computational psychometrics by combining psychometrics with learning analytics. In D. Burgos (Ed.), Radical solutions and learning analytics: Personalised learning and teaching through big data (pp. 67–80). Springer. https://doi.org/10.1007/978-981-15-4526-9_5

    Chapter  Google Scholar 

  • Ebrahimi, M. S., & Azmi, M. N. (2015). New approach to leadership skills development (developing a model and measure). Journal of Management Development, 34(7), 821–853. https://doi.org/10.1108/JMD-03-2013-0046

    Article  Google Scholar 

  • Ginda, M., Richey, M. C., Cousino, M., & Börner, K. (2019). Visualizing learner engagement, performance, and trajectories to evaluate and optimize online course design. PLoS ONE, 14(5), e0215964. https://doi.org/10.1371/journal.pone.0215964

    Article  Google Scholar 

  • Gray, G., & Bergner, Y. (2022). A practitioner’s guide to measurement in learning analytics – Decisions, opportunities, and challenges. In Handbook of learning analytics (2nd ed., pp. 20–28).

    Google Scholar 

  • Haste, H. (2001). Ambiguity, autonomy and agency: Psychological challenges to new competence. Defining and Selecting Key Competencies, 93–120.

    Google Scholar 

  • Helyer, R. (2015). Learning through reflection: The critical role of reflection in work-based learning (WBL). Journal of Work-Applied Management, 7(1), 15–27. https://doi.org/10.1108/JWAM-10-2015-003

    Article  Google Scholar 

  • Jenkins, H., Clinton, K., Purushotma, R., Robison, A. J., & Weigel, M. (2006). Confronting the challenges of participatory culture: Media education for the 21st century. MacArthur Foundation.

    Google Scholar 

  • Joksimovic, S., Kovanovic, V., & Dawson, S. (2019). The journey of learning analytics. HERDSA Review of Higher Education, 6, 37–63.

    Google Scholar 

  • Joksimovic, S., Siemens, G., Wang, Y. E., San Pedro, M. O. Z., & Way, J. (2020). Editorial: Beyond cognitive ability. Journal of Learning Analytics, 7(1), 1–4. https://doi.org/10.18608/jla.2020.71.1

    Article  Google Scholar 

  • Jung, Y., & Wise, A. F. (2020). How and how well do students reflect? Multi-dimensional automated reflection assessment in health professions education. In Proceedings of the tenth international conference on learning analytics & knowledge (pp. 595–604). https://doi.org/10.1145/3375462.3375528

  • Kember, D., McKay, J., Sinclair, K., & Wong, F. K. Y. (2008). A four-category scheme for coding and assessing the level of reflection in written work. Assessment & Evaluation in Higher Education, 33(4), 369–379. https://doi.org/10.1080/02602930701293355

    Article  Google Scholar 

  • Kovanović, V., Joksimović, S., Mirriahi, N., Blaine, E., Gašević, D., Siemens, G., & Dawson, S. (2018). Understand students’ self-reflections through learning analytics. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 389–398). https://doi.org/10.1145/3170358.3170374.

  • Krippendorff, K. (2003). Content analysis: An introduction to its methodology (p. 8). Sage.

    Google Scholar 

  • Kyllonen, P. C. (2012). Measurement of 21st century skills within the common core state standards. In Invitational research symposium on technology enhanced assessments (p. 24).

    Google Scholar 

  • Lai, E. R., & Viering, M. (2012). Assessing 21st century skills: Integrating research findings. Pearson.

    Google Scholar 

  • Lee, Y.-W., & Sawaki, Y. (2009). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6(3), 172–189. https://doi.org/10.1080/15434300902985108

    Article  Google Scholar 

  • Martin, C. K., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 37–87.

    Article  Google Scholar 

  • Milligan, S. (2020). Standards for developing assessments of learning using process data. In M. Bearman, P. Dawson, R. Ajjawi, J. Tai, & D. Boud (Eds.), Re-imagining university assessment in a digital world (Vol. 7, pp. 179–192). Springer. https://doi.org/10.1007/978-3-030-41956-1_13

    Chapter  Google Scholar 

  • Milligan, S. K., & Griffin, P. (2016). Understanding learning and learning design in MOOCs: A measurement-based interpretation. Journal of Learning Analytics, 3(2), 88–115. https://doi.org/10.18608/jla.2016.32.5

    Article  Google Scholar 

  • Mislevy, R. J., Behrens, J. T., Dicerbo, K. E., & Levy, R. (2012). Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining. Journal of Educational Data Mining, 4(1), 11–48.

    Google Scholar 

  • Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3), 263–280. https://doi.org/10.1002/ejsp.2420150303

    Article  Google Scholar 

  • Pellegrino, J. W. (2017). Teaching, learning and assessing 21st century skills (pp. 223–251). https://doi.org/10.1787/9789264270695-12-en

  • Pongpaichet, S., Nirunwiroj, K., & Tuarob, S. (2022). Automatic assessment and identification of leadership in college students. IEEE Access, 1. https://doi.org/10.1109/ACCESS.2022.3193935

  • Quinn, R. E. (1988). Beyond rational management: Mastering the paradoxes and competing demands of high performance (pp. xxii, 199). Jossey-Bass.

    Google Scholar 

  • Rios, J. A., Ling, G., Pugh, R., Becker, D., & Bacall, A. (2020). Identifying critical 21st-century skills for workplace success: A content analysis of job advertisements. Educational Researcher, 49(2), 80–89. https://doi.org/10.3102/0013189X19890600

    Article  Google Scholar 

  • Rohs, F. R., & Langone, C. A. (1997). Increased accuracy in measuring leadership impacts. Journal of Leadership Studies, 4(1), 150–158. https://doi.org/10.1177/107179199700400113

    Article  Google Scholar 

  • Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6(4), 219–262. https://doi.org/10.1080/15366360802490866

    Article  Google Scholar 

  • Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

    Google Scholar 

  • Salleb-Aouissi, A., Vrain, C., & Nortel, C. (2007). QuantMiner: A genetic algorithm for mining quantitative association rules (pp. 1035–1040).

    Google Scholar 

  • Sartori, L., & Theodorou, A. (2022). A sociotechnical perspective for the future of AI: Narratives, inequalities, and human control. Ethics and Information Technology, 24(1), 4. https://doi.org/10.1007/s10676-022-09624-3

    Article  Google Scholar 

  • Shum, S. B., & Crick, R. D. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21. https://doi.org/10.18608/jla.2016.32.2

    Article  Google Scholar 

  • Sondergeld, T. A., & Johnson, C. C. (2019). Development and validation of a 21st century skills assessment: Using an iterative multimethod approach. School Science and Mathematics, 119(6), 312–326. https://doi.org/10.1111/ssm.12355

    Article  Google Scholar 

  • Sutherland, M. A., Amar, A. F., & Laughon, K. (2013). Who sends the email? Using electronic surveys in violence research. The Western Journal of Emergency Medicine, 14(4), 363–369. https://doi.org/10.5811/westjem.2013.2.15676

    Article  Google Scholar 

  • Ullmann, T. D. (2019). Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257. https://doi.org/10.1007/s40593-019-00174-2

    Article  Google Scholar 

  • Vockley, M. (2007). Maximizing the impact: The pivotal role of technology in a 21st century education system. In Partnership for 21st century skills. Partnership for 21st Century Skills. https://eric.ed.gov/?id=ED519463

  • Wu, Y., & Crocco, O. (2019). Critical reflection in leadership development. Industrial and Commercial Training, 51(7/8), 409–420. https://doi.org/10.1108/ICT-03-2019-0022

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of South Australia, Adelaide, SA, Australia

    Abhinava Barthakur, Srecko Joksimovic & Abelardo Pardo

  2. Centre for Change and Complexity in Learning, University of South Australia, Adelaide, SA, Australia

    Vitomir Kovanovic

Authors
  1. Abhinava Barthakur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vitomir Kovanovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Srecko Joksimovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abelardo Pardo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abhinava Barthakur .

Editor information

Editors and Affiliations

  1. Centre for Change and Complexity, University of South Australia, Adelaide, SA, Australia

    Vitomir Kovanovic

  2. School of Modeling Simulation and Training, University of Central Florida, Orlando, FL, USA

    Roger Azevedo

  3. Data Science in Higher Education Learning and Teaching, Curtin University, Bentley, WA, Australia

    David C. Gibson

  4. Learning, Design and Technology, University of Mannheim, Mannheim, BW, Germany

    Dirk lfenthaler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Barthakur, A., Kovanovic, V., Joksimovic, S., Pardo, A. (2023). Challenges in Assessments of Soft Skills: Towards Unobtrusive Approaches to Measuring Student Success. In: Kovanovic, V., Azevedo, R., Gibson, D.C., lfenthaler, D. (eds) Unobtrusive Observations of Learning in Digital Environments. Advances in Analytics for Learning and Teaching. Springer, Cham. https://doi.org/10.1007/978-3-031-30992-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30992-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30991-5

  • Online ISBN: 978-3-031-30992-2

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation