Abstract
Many school change efforts aim to foster conditions for the development of students’ twenty-first century skills in an equitable and comprehensive way, but methods to measure the frequency and quality of opportunities that students receive in school are lacking. In this study, we describe the design, testing, and initial evaluation of a self-report measure of high-school students’ opportunity to learn and apply a core set of intrapersonal and interpersonal twenty-first century skills—collaboration, communication, creativity, critical thinking, and self-directed learning. Our sample included 9th graders (n = 522) from one large comprehensive high school in central California. Confirmatory factor analysis and multivariate regression modeling suggest promising reliability and validity to evaluate opportunities in high school learning environments. Based on this early evidence, the Student MetaSkills Survey appears to provide a way for school leaders and educators to evaluate the frequency and quality of key twenty-first century skills and take responsibility for providing equitable opportunities for students. Measuring inputs and processes, such as students’ opportunity-to-learn, can play an important role in the process of educational improvement.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alspaugh, J. W. (1998). Achievement loss associated with the transition to middle school and high school. The Journal of Educational Research, 92, 20–25.
American Institutes for Research. (2016). Does deeper learning improve student outcomes? Results from the Study of Deeper Learning: Opportunities and outcomes. Washington, DC: American Institutes for Research
Ananiadou, K., & Claro, M. (2009). 21st century skills and competences for new millennium learners in OECD countries (OECD Education Working Papers, No. 41). OECD Publishing.
Beghetto, R. A. (2016). Creative learning: A fresh look. Journal of Cognitive Education and Psychology, 15, 6–23.
Beghetto, R. A., & Kaufman, J. C. (2007). Toward a broader conception of creativity: A case for mini-c creativity. Psychology of Aesthetics, Creativity, and the Arts, 1, 73–79.
Bitter, C., Taylor, J., Zeiser, K. L., & Rickles, J. (2014). Providing opportunities for deeper learning. American Institutes for Research.
Bolhuis, S., & Voeten, M. J. M. (2001). Toward self-directed learning in secondary schools: What do teachers do?. Teacher and Teacher Education, 17, 837–855.
Booth, W. C. (2004). The rhetoric of rhetoric: The quest for effective communication. Malden, MA: Blackwell.
Bronson, M. B. (2001). Self-regulation in early childhood: Nature and nurture. New York, NY: Guilford Press.
Brown, T. (2006). Confirmatory factor analysis for applied research. Guildford.
Care, E., Anderson, K., & Kim, H. (2016). Visualizing the breadth of skills movement across educational systems. Center for Universal Education at Brookings.
Cohen, J. (1992). A power primer. Pyschological Bulletin, 112(1), 155–159.
Conley, D. T. (2013). Rethinking the notion of “noncognitive.” Education Week, 32(18), 20–21.
Costa, A. L., & Kallick, B. (2004). Assessment strategies for self-directed learning. Thousand Oaks, CA: Corwin Press.
Council of Chief State School Officers. (2013). Knowledge, skills, and dispositions: The Innovation Lab Network state framework for college, career, and citizenship readiness, and implications for state policy. Retrieved from www.ccsso.org
Darling-Hammond, L. (2010). The flat world and education: How America’s commitment to equity will determine our future. Teachers College Press.
Datnow, A., & Park, V. (2018). Opening or closing doors for students: Equity and data use in schools. Journal of Educational Change, 19, 131–152.
Davis, G. (1999). Barriers to creativity and creative attitudes. In M. A. Runco & S. Pritzker (Eds.), Encyclopedia of creativity (pp. 165–174). Academic Press.
Dillman, D., Smyth, J., & Christian, L. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. Wiley.
Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition (No. ORC-80–2). Berkeley, CA: University of California Operations Research Center.
Dreyfus, H. L., & Dreyfus, S. E. (2005). Peripheral vision: Expertise in real world contexts. Organizational Studies, 26, 779–792.
Elder, L., & Paul, R. (2007). Critical thinking: The nature of critical and creative thought, Part II. Journal of Developmental Education, 30(3), 36–37.
Finke, R. A., Ward, T. B., & Smith, S. M. (1992). Creative cognition: Theory, research, and application. MIT Press.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34, 906–911.
Geisinger, K. (2016). 21st century skills: What are they and how do we assess them? Applied Measurement in Education, 29, 245–249.
Green, J. O., & Burleson, B. R. (Eds.). (2008). Handbook of communication and social interaction skills. Mahwah, NJ: Erlbaum.
Guglielmino, L. M., & Guglielmino, P. (2015). Self-Directed Learning Readiness Scale (Learning Preference Assessment). Retrieved June 25, 2015, from http://www.lpasdlrs.com
Guha, R., Caspary, K., Stites, R., Padilla, C., Arshan, N., Park, C., Tse, V., Astudillo, S., Black, A., & Adelman, N. (2014). Taking stock of the California Linked Learning District Initiative: Fifth-year evaluation report. Menlo Park, CA: SRI International.
Heckman, J. J., & Rubinstein, Y. (2001). The importance of noncognitive skills: Lessons from the GED testing program. American Economic Review, 91, 145–149.
Katz-Buonincontro, J. & Anderson, R. C. (2020). Observation methods for researching creativity: Past approaches and recommendations for advancing the field. Journal of Creative Behavior, 54(3), 508–524. https://doi.org/10.1002/jocb.385
Kline, R. (2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press.
Ladson-Billings, G. (2006). From the achievement gap to the education debt: Understanding achievement in U.S. schools. Educational Researcher, 35, 3–12.
Lai, E. R. (2011). Collaboration: A literature review. Retrieved from http://images.pearsonassessments.com/images/tmrs/Collaboration-Review.pdf
Lench, S., Fukuda, E., & Anderson, R. C. (2015). Essential skills and dispositions: Developmental frameworks for collaboration, creativity, communication, and self-direction. Lexington, KY: Center for Innovation in Education at the University of Kentucky. Retrieved at https://www.leadingwithlearning.org/esd
Lucas, B. (2016). A five-dimensional model of creativity and its assessment in schools. Applied Measurement in Education, 29, 278–290.
MacCallum, R., Widaman, K., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4, 84–99.
Madison, E., Anderson, R. C., & Bousselot, T. (2019). Self-determined to write: Catalyzing critical thinking, collaboration, and self-direction in middle school. Reading and Writing Quarterly, 35(5), 473–495. https://doi.org/10.1080/10573569.2019.1579127
Maine Department of Education. (2007). Chapter 132 – Learning results: Parameters for essential instruction summary. Retrieved from http://www.maine.gov/education/lres/pei/summary102207.pdf
Martin, C., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3, 37–87.
Muthén, L., & Muthén, B. (2012). Mplus user’s guide (7th ed.). Los Angeles: Muthén & Muthén.
National Research Council. (2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academy Press
National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st Century (Committee on defining deeper learning and 21st century skills, J.W. Pellegrino and M. L. Hilton, Eds., Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education). Washington, DC: The National Academies Press.
New Hampshire Department of Education. (2014). New Hampshire work-study practices: Rationale for work-study practices–June 2014. Retrieved fromhttps://education.nh.gov/innovations/hs_redesign/documents/nhsbea-approved-final.pdf
Orfield, G. (2014). Tenth annual Brown lecture in education research: A new civil rights agenda for American education. Educational Researcher, 43, 273–292.
Perry, L., Their, M., Beach, P., Anderson, R. C., Roberts, P., & Thoennessen, M. (under review). Opportunities and conditions to learn (OCL): A conceptual framework.
Pitts, C., Anderson, R. C., & Haney, M. (2018). Measure of instruction for creative engagement: Making metacognition, modeling, and creative thinking visible. Learning Environments Research, 21(1), 43–59. https://doi.org/10.1007/s10984-017-9238-9
Plucker, J. A. (2005). The (relatively) generalist view of creativity. In J. C. Kaufman & J. Baer (Eds.), Creativity across domains: Faces of the muse (pp. 307–312). Mahwah, NJ: Erlbaum.
Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35, 299–331.
Rickheit, G., & Strohner, H. (Eds.). (2008). Handbook of communication competence. Berlin, Germany: Mounton de Gruyter.
Ritchhart, R. (2002). Intellectual character: What it is, why it matters, and how to get it. San Francisco, CA: Jossey-Bass.
Runco, M. A., & Chand, I. (1995). Cognition and creativity. Educational Psychology Review, 7, 243–267.
Schacter, J., Thum, Y., & Zifkin, D. (2006). How much does creative teaching enhance elementary school students’ achievement? Journal of Creative Behavior, 40(1), 47–72.
Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental design for generalized causal inference. CA: Wadsworth Cengage Learning.
Soland, J., Stecher, B. M., & Hamilton, L. S. (2013). Measuring 21st-century competencies: Guidance for educators. Asia Society & RAND.
South Carolina Education Oversight Committee (2015). 2015 Annual Report. Retrieved from http://www.eoc.sc.gov/Reports%20%20Publications/Annual%20Repor%20t2015/Feb1report2015.pdf
Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). HarperCollins.
The College Board. (2013). A review of the connections between the common core state standards and the next generation arts standards. New York, NY: Author.
The College Board. (2014). The arts and the Common Core: A comparison of the national core arts standards and the common core state standards. New York, NY: Author.
The Secretary's Commission on Achieving Necessary Skills. (1991). What work requires of schools: A SCANS report for America 2000. Washington DC: U.S. Department of Labor
Treffinger, D. J., & Isaksen, S. G. (2005). Creative problem solving: The history, development, and implications for gifted education and talent development. Gifted Child Quarterly, 49(4), 342–353.
Volet, S., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: How does it emerge and how is it sustained? Learning and Instruction, 19, 128–143.
Wallas, G. (1926). The art of thought. Harcourt Brace and World.
Weinberger, A., Stegmann, K., & Fischer, F. (2007). Knowledge convergence in collaborative learning: Concepts and assessment. Learning and Instruction, 17, 416–426.
Zerihun, Z., Beishuizen, J., & Van Os, W. (2012). Student learning experience as indicator of teaching quality. Educational Assessment, Evaluation and Accountability, 24, 99–111.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix
Table 8.
MetaSkills survey
Hello!
We are researchers and need your help to learn more about your learning experiences at school. In this survey we will ask you questions about a variety of skills you use while learning. Skills like collaboration and creativity are important and can take time to learn. Schools need to provide students sufficient opportunities to learn and practice these skills. Your school wants to know more about your experience using these skills so they can make improvements.
In this survey, we ask you questions that relate to your learning experience in school. Please respond to the questions and share how you really think and behave and not how you would like to. Your opinion matters most, not what someone else thinks. Try not to dwell on each question too long. Your first thought is usually a good indication of the best response for you. Your responses in this survey will remain private.
For most of the questions we ask how often you find yourself doing the things described in each of the statements. For each question, mark the response option that most closely represents being right for you in your current educational role.
Throughout the survey we will ask you to complete most questions using the following response options:
Never = did not happen during the most recent school semester.
Once a semester = about once a semester
Monthly = about once or twice a month
Weekly = about once or twice a week
Daily = at least once a day
Rights and permissions
About this article
Cite this article
Anderson, R.C., Beach, P. Measure of opportunity: assessing equitable conditions to learn twenty-first century thinking skills. Learning Environ Res 25, 741–774 (2022). https://doi.org/10.1007/s10984-021-09388-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10984-021-09388-5