Abstract
This article examines a critical issue in digital literacy assessment design when technological changes are happening with escalating speed in our society. There have been many assessment studies of digital literacy (DL) for diverse purposes and across different geographic and socioeconomic (geo-socioeconomic) contexts. While the assessment framework, instrument design, and technology platforms used for conducting these assessments differ, what remains common is the lack of explicit discussion about the possible role of the technology used and item design in affecting the measure DL. There is an apparent, implicit assumption that DL assessment is similar to the assessment of other academic achievements such as reading literacy and numeracy, which should ideally be measured independent of the specific technologies or task contexts adopted in the assessment. Recent evidence from a United Nations Educational, Scientific and Cultural Organization (UNESCO) commissioned study on a Digital Literacy Global Framework (DLGF) shows that the DL needed to accomplish the same task is heavily dependent on the devices and tools used under different the geo-socioeconomic contexts (Law et al. in A global framework of reference on digital literacy skills for indicator 4.4.2, 2018). Drawing on the DLGF findings and a critical examination of the assessment designs in large-scale international assessment tests, this paper puts forward a product-lifecycle and experience dependence (PLED) perspective to guide the design and interpretation of DL assessment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Figures in brackets refer to the DigComp 2.0 competences listed in Appendix 1.
References
Ahonen, A., & Harding, S.-M. (2018). Assessing online collaborative problem solving among school children in Finland: A case study using ATC21S™ in a national context. International Journal of Learning, Teaching and Educational Research, 17(2), 138–158.
Alarcón, R., del Pilar Jiménez, E., & de Vicente-Yagüe, M. I. (2020). Development and validation of the DIGIGLO, a tool for assessing the digital competence of educators. British Journal of Educational Technology, 51, 2407.
Almerich, G., Orellana, N., Suárez-Rodríguez, J., & Díaz-García, I. (2016). Teachers’ information and communication technology competences: A structural approach. Computers & Education, 100, 110–125.
Appelbaum, S. (1997). Socio-technical systems theory: An intervention strategy for organizational development. Management Decision, 35(6), 452–463.
Audrin, C., & Audrin, B. (2022). Key factors in digital literacy in learning and education: A systematic literature review using text mining. Education and Information Technologies, 27, 1–25.
Baker, E. L., O’Neil, H. F., & Linn, R. L. (1993). Policy and validity prospects for performance-based assessment. American Psychologist, 48(12), 1210–1218.
Bawden, D. (2008). Origins and concepts of digital literacy. In C. Lankshear & M. Knobel (Eds.), Digital literacies: Concepts, policies and practices (Vol. 30, pp. 17–32). Peter Lang.
Berezner, A., & Adams, R. J. (2017). Why large-scale assessments use scaling and item response theory. In P. Lietz, J. C. Creswell, K. Rust, & R. J. Adams (Eds.), Implementation of large-scale education assessments (pp. 323–356). Wiley.
Bilousova, L., Gryzun, L., & Zhytienova, N. (2021). Interactive methods in blended learning of the fundamentals of UI/UX design by pre-service specialists. Educational Technology Quarterly, 2021(3), 415–428.
Borna, S., & Wahlers, R. (2018). Product identity over time and the concept of product life cycle. Journal of Management and Strategy, 9(2), 27–33.
Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1: The digital competence framework for citizens with eight proficiency levels and examples of use (No. JRC106281). Joint Research Centre (Seville site). Retrieved from http://publications.jrc.ec.europa.eu/repository/bitstream/JRC-106281/web-digcomp2.1pdf_(online).pdf
Chemero, A. (2018). An outline of a theory of affordances. In K. S. Jones (Ed.), How shall affordances be refined? Four perspectives (pp. 181–195). Routledge.
Cox, W. E. (1967). Product life cycles as marketing models. The Journal of Business, 40(4), 375–384.
Day, G. S. (1981). The product life cycle: Analysis and applications issues. The Journal of Marketing, 45(4), 60–67.
Ferrari, A. (2013). DIGCOMP: A framework for developing and understanding digital competence in Europe. In Y. Punie & B. N. Brečko (Eds.), JRC scientific and policy reports (pp. 79–92). Publications Office of the European Union.
Fraillon, J. (2018). International large-scale computer-based studies on information technology literacy in education. In J. Voogt, G. Knezek, R. Christensen, & K.-W. Lai (Eds.), Second handbook of information technology in primary and secondary education (pp. 1161–1179). Springer.
Fraillon, J., Ainley, J., Schulz, W., Duckworth, D., & Friedman, T. (2019). IEA international computer and information literacy study 2018 assessment framework. Springer.
Fraillon, J., Schulz, W., & Ainley, J. (2013). International computer and information literacy study: Assessment framework. IEA.
Gibson, J. J. (2014). The ecological approach to visual perception (Classic). Psychology Press.
Gilster, P. (1997). Digital literacy. Wiley.
Gmelin, H., & Seuring, S. (2014). Determinants of a sustainable new product development. Journal of Cleaner Production, 69, 1–9.
Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. Springer.
Haßler, B., Major, L., & Hennessy, S. (2016). Tablet use in schools: A critical review of the evidence for learning outcomes. Journal of Computer Assisted Learning, 32(2), 139–156.
Hatlevik, O. E., Guðmundsdóttir, G. B., & Loi, M. (2015). Digital diversity among upper secondary students: A multilevel analysis of the relationship between cultural capital, self-efficacy, strategic use of information and digital competence. Computers & Education, 81, 345–353.
Jara, I., Claro, M., Hinostroza, J. E., San Martín, E., Rodríguez, P., Cabello, T., Ibieta, A., & Labbé, C. (2015). Understanding factors related to Chilean students’ digital skills: A mixed methods analysis. Computers & Education, 88, 387–398.
Jin, K. Y., Reichert, F., Cagasan, L. P., Jr., de La Torre, J., & Law, N. (2020). Measuring digital literacy across three age cohorts: Exploring test dimensionality and performance differences. Computers & Education, 157, 103968.
Laanpere, M. (2018). Recommendations on assessment tools for monitoring digital literacy within UNESCO DLGF. Retrieved from Hamburg, Germany http://gaml.uis.unesco.org/wp-content/uploads/sites/2/2018/12/4.4.2_02-Assessment-tools-for-monitoring-digital-literacy.pdf
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.
Law, N., Woo, D., de la Torre, J., & Wong, G. K. W. (2018). A global framework of reference on digital literacy skills for indicator 4.4.2. Retrieved from Montreal, Quebec http://uis.unesco.org/sites/default/files/documents/ip51-global-framework-reference-digital-literacy-skills-2018-en.pdf
Levitt, T. (1965). Exploit the product life cycle. Harvard Business Review, 43, 81–94.
Li, Y., & Ranieri, M. (2010). Are ‘digital natives’ really digitally competent?—A study on Chinese teenagers. British Journal of Educational Technology, 41(6), 1029–1042.
Martínez-Bravo, M. C., Sádaba Chalezquer, C., & Serrano-Puche, J. (2022). Dimensions of digital literacy in the 21st century competency frameworks. Sustainability, 14, 1867. https://doi.org/10.3390/su14031867
Mattar, J., Ramos, D. K., & Lucas, M. R. (2022). DigComp-based digital competence assessment tools: Literature review and instrument analysis. Education and Information Technologies. https://doi.org/10.1007/s10639-022-11034-3
McDougall, J., Readman, M., & Wilkinson, P. (2018). The uses of (digital) literacy. Learning, Media and Technology, 43(3), 263–279.
Meyers, E. M., Erickson, I., & Small, R. V. (2013). Digital literacy and informal learning environments: An introduction. Learning, Media and Technology, 38(4), 355–367.
Mossberger, K., Tolbert, C. J., & McNeal, R. S. (2008). Digital citizenship: The internet, society, and participation. MIT Press.
Norman, D. A. (1999). Affordance, conventions, and design. Interactions, 6(3), 38–43. https://doi.org/10.1145/301153.301168
Raynard, M. (2017). Understanding academic e-books through the diffusion of innovations theory as a basis for developing effective marketing and educational strategies. The Journal of Academic Librarianship, 43(1), 82–86.
Reedy, K., & Parker, J. (Eds.). (2018). Digital literacy unpacked. Facet Publishing.
Reichert, F., Zhang, J., Law, N., Wong, G., & de la Torre, J. (2020). Exploring the structure of digital literacy competence assessed using authentic software applications. Educational Technology Research and Development, 68(6), 2991–3013. https://doi.org/10.1007/s11423-020-09825-x
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Rohatgi, A., Scherer, R., & Hatlevik, O. E. (2016). The role of ICT self-efficacy for students’ ICT use and their achievement in a computer and information literacy test. Computers & Education, 102, 103–116.
Rolf, E., Knutsson, O., & Ramberg, R. (2019). An analysis of digital competence as expressed in design patterns for technology use in teaching. British Journal of Educational Technology, 50(6), 3361–3375.
Scherer, R., Siddiq, F., & Teo, T. (2015). Becoming more specific: Measuring and modeling teachers’ perceived usefulness of ICT in the context of teaching and learning. Computers & Education, 88, 202–214.
Siddiq, F., Gochyyev, P., & Wilson, M. (2017). Learning in digital networks—ICT literacy: A novel assessment of students’ 21st century skills. Computers & Education, 109, 11–37.
Siddiq, F., Hatlevik, O. E., Olsen, R. V., Throndsen, I., & Scherer, R. (2016a). Taking a future perspective by learning from the past—A systematic review of assessment instruments that aim to measure primary and secondary school students’ ICT literacy. Educational Research Review, 19, 58–84.
Siddiq, F., Scherer, R., & Tondeur, J. (2016b). Teachers’ emphasis on developing students’ digital information and communication skills (TEDDICS). A new construct in 21st century education. Computers & Education, 92, 1–14.
Van Laar, E., van Deursen, A. J., van Dijk, J. A., & de Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72, 577–588.
Vernon, R. (1966). International investment and international trade in the product cycle. The Quarterly Journal of Economics, 80(2), 190–207.
Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44, 299–332.
Vuorikari, R., Kluzer, S., & Punie, Y. (2022). DigComp 2.2: The digital competence framework for citizens—With new examples of knowledge, skills and attitudes. Publications Office of the European Union.
Vuorikari, R., Punie, Y., Carretero, S., & Van den Brande, L. (2016). DigComp 2.0: The digital competence framework for citizens. Update phase 1: The conceptual reference model. Publications Office of the European Union.
Wilson, M., Gochyyev, P., & Scalise, K. (2017). Modeling data from collaborative assessments: Learning in digital interactive social networks. Journal of Educational Measurement, 54(1), 85–102.
Wilson, M., & Scalise, K. (2015). Assessment of learning in digital network. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach. Springer.
Woo, D., & Law, N. W. Y. (2021). A methodology for deploying a digital literacy framework for diverse socioeconomic and sector contexts. In I. A. Lubin (Ed.), ICT-enabled learning ecologies: Representation and sustainability across contexts. Routledge.
Wu, M., Adams, R., Wilson, M., & Haldane, S. (2007). ConQuest: Generalised item response modelling software (Version 2.0). ACER Press.
Zane, T. W. (2009). Performance assessment design principles gleaned from constructivist learning theory (Part 2). TechTrends, 53(3), 86–94.
Funding
This study was funded by the Research Grants Council of the HKSAR Government, University Grants Committee (#T44-707/16N), under the Theme-based Research Scheme.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Research involving human participants
This study is a conceptual exploration and did not involve data collection from human subjects.
Informed consent
No informed consent is required given no data collection from human subject was involved.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
DIGCOMP 2.2 framework (Vuorikari et al., 2022)
Competence area (Dimension 1) | Competence title and descriptor (Dimension 2) |
|---|---|
1. Information and data literacy | 1.1 Browsing, searching, filtering data, information and digital content |
1.2 Evaluating data, information and digital content | |
1.3 Managing data, information and digital content | |
2. Communication and collaboration | 2.1 Interacting through digital technologies |
2.2 Sharing through digital technologies | |
2.3 Engaging in citizenship through digital technologies | |
2.4 Collaborating through digital technologies | |
2.5 Netiquette | |
2.6 Managing digital identity | |
3. Digital content creation | 3.1 Developing digital content |
3.2 Integrating and re-elaborating digital content | |
3.3 Copyright and licences | |
3.4 Programming | |
4. Safety | 4.1 Protecting devices |
4.2 Protecting personal data and privacy | |
4.3 Protecting health and well-being | |
4.4 Protecting the environment | |
5. Problem solving | 5.1 Solving technical problems |
5.2 Identifying needs and technological responses | |
5.3 Creatively using digital technologies | |
5.4 Identifying digital competence gaps |
Appendix 2
Assessment framework in ICILS 2018 (Fraillon et al., 2019)
Modules | Dimension | Aspects |
|---|---|---|
CIL | Strand 1: understanding computer use | 1.1 Foundation of computer use (e.g. input/output, memory storage, basic technical support) |
1.2 Computer use conventions (e.g. navigate within webpage using hyperlink, save file as another format, open a file of a specified type) | ||
Stand 2: gathering information | 2.1 Accessing and evaluating information | |
2.2 Managing information | ||
Strand 3: producing information | 3.1 Transforming information | |
3.2 Creating information | ||
Strand 4: digital communication | 4.1 Sharing information | |
4.4 Using information responsibly and safely | ||
CT | Strand 1: conceptualizing problems | 1.1 Knowing about and understanding digital systems |
1.2 Formulating and analyzing problems | ||
1.3 Collecting and representing relevant data | ||
Strand 2: operationalizing solutions | 2.1 Planning and evaluating solutions |
Appendix 3
DL assessment framework in ATC21S (LDN-ICT) (Wilson & Scalise, 2015)
Dimension | Aspects |
|---|---|
Strand 1: consumer in networks (CiN) | Obtaining, managing and utilizing information and knowledge from shared digital resources and experts |
Stand 2: producer in networks (PiN) | Creating, developing, organizing and re-organizing information/knowledge |
Strand 3: social capital through networks (SCN) | Using, developing, moderating, leading and brokering the connectivities within and between individuals and social groups |
Strand 4: intellectual capital through networks (ICN) | Understanding how tools, media and social networks operate with appropriate techniques |
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wong, G.KW., Reichert, F. & Law, N. Reorienting the assessment of digital literacy in the twenty-first century: a product-lifecycle and experience dependence perspective. Education Tech Research Dev 71, 2389–2412 (2023). https://doi.org/10.1007/s11423-023-10278-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-023-10278-1