Measuring experiences of interest-related pursuits in connected learning

  • Andrew Maul
  • William R. Penuel
  • Nathan Dadey
  • Lawrence P. Gallagher
  • Timothy Podkul
  • Emily Price
Research Article


This paper describes an effort to develop a survey instrument capable of measuring important aspects of adolescents’ experiences of interest-related pursuits that are supported by technology. The measure focuses on youths’ experiences of connected learning (Ito et al. in Connected learning: an agenda for research and design. Digital Media and Learning Research Hub, Irvine, 2013), an emerging model of learning across settings supported by digital media. Specifically, the instrument aims to measure the depth with which youth are able to engage in an interest-related pursuit, the level of support and encouragement they receive from peers, and the degree to which their pursuit involves performance or media production as an essential feature. The survey also elicits information regarding the connections between youths’ interest-related pursuits and academic goals, the involvement of adults as co-participants in pursuits, and youths’ access to technology tools they deem necessary for their pursuits. The paper reports on results from a pilot study and two rounds of field-testing, in which we evaluated the validity and reliability of the instrument and compared results with evidence from interviews with youth. Our aim was to investigate the feasibility of an approach to measuring youths’ interest-related pursuits to inform future research and evaluation of initiatives focused on digital media and learning.


Connected learning Survey Measurement Interest development 



Funding for this survey development effort comes from the MacArthur Foundation. All opinions expressed herein are the sole responsibility of the authors. We wish to thank the youth leaders in the sites that facilitated data collection for the field test and members of the Connected Learning Research Network for their generous input into the survey development process and feedback on the survey itself.


  1. Abramovich, S., Schunn, C. D., & Higashi, R. M. (2012). Are badges useful in education? It depends on the type of badge and the expertise of learner. Educational Technology Research and Development, 61(2), 217–232.CrossRefGoogle Scholar
  2. Barron, B., Gomez, K., Pinkard, N., & Martin, C. K. (2014). The Digital Youth Network: Cultivating digital media citizenship in urban communities. Boston, MA: MIT Press.Google Scholar
  3. Barron, B., Levinson, A., Martin, C. K., Mertl, V., Stringer, D., Rogers, M., et al. (2010). Supporting young new media producers across learning spaces: A longitudinal study of the Digital Youth Network. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the disciplines: Proceedings of the 9th international conference of the learning sciences (Vol. 2, pp. 203–211). Chicago, IL: International Society of the Learning Sciences.Google Scholar
  4. Barron, B., Martin, C. K., Takeuchi, L., & Fithian, R. (2009). Parents as learning partners in the development of technological fluency. International Journal of Learning and Media, 1(2), 55–77.CrossRefGoogle Scholar
  5. Black, R. W. (2005). Access and affiliation: The literacy and composition practices of English-language learners in an online fan fiction community. Journal of Adolescent & Adult Literacy, 49(2), 118–128.CrossRefGoogle Scholar
  6. Buechley, L., Peppler, K. A., Eisenberg, M., & Kafai, Y. B. (2013). Textile messages: Dispatches from the world of e-textiles and education. New York, NY: Peter Lang.CrossRefGoogle Scholar
  7. Burke, Q., & Kafai, Y. B. (2014). A decade of game-making for learning: From tools to communities. In H. Agius & M. C. Angelides (Eds.), The handbook of digital games (pp. 689–709). New York, NY: Wiley.Google Scholar
  8. Catalano, R. F., Berglund, M. L., Ryan, J. A. M., Lonczak, H. S., & Hawkins, J. D. (2004). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Annals of the American Academy of Political and Social Science, 591, 98–124.CrossRefGoogle Scholar
  9. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.CrossRefGoogle Scholar
  10. Dadey, N., Penuel, W. R., & Maul, A. (under review). Looking at differences and changes in the outcomes of connected learning. Applied Developmental Science.Google Scholar
  11. Davis, K., & Singh, S. (2015). Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Computers & Education, 88(1), 72–83.Google Scholar
  12. DeNavas-Walt, C., Proctor, B. D., & Smith, J. C. (2012). Income, poverty, and health insurance coverage in the United States: 2011. Washington, DC: U.S. Census Bureau.Google Scholar
  13. Devedžić, V., & Jovanović, J. (2015). Developing Open Badges: A comprehensive approach. Educational Technology Research and Development, 63(4), 603–620.CrossRefGoogle Scholar
  14. Duffy, T. M., & Jonassen, D. (Eds.). (1992). Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Erlbaum.Google Scholar
  15. Dugan, T. E., Stevens, R., & Mehus, S. (2010). From show, to room, to world: A cross-context investigation of how children learn from media programming. In K. Gomez & J. Radinsky (Eds.), Proceedings of the 9th international conference of the learning sciences (pp. 992–999). Chicago, IL: ISLS.Google Scholar
  16. Engeström, Y., & Sannino, A. (2010). Studies of expansive learning: Foundations, findings and future challenges. Educational Research Review, 5, 1–24.CrossRefGoogle Scholar
  17. Garcia, A., & Morrell, E. (2013). City youth and the pedagogy of participatory media. Learning, Media and Technology, 38(2), 123–127.CrossRefGoogle Scholar
  18. Gee, J. P. (2007). Good video games and good learning: Collected essays on video games, learning, and literacy. London: Routledge.CrossRefGoogle Scholar
  19. Gee, J. P. (2010). New digital media and learning as an emerging area and “worked examples” as one way forward. Cambridge, MA: MIT Press.Google Scholar
  20. Gee, J. P., & Hayes, E. R. (2010). Women and gaming: The Sims and 21st Century Learning. New York, NY: Palgrave Macmillan.CrossRefGoogle Scholar
  21. Goldman, S., Booker, A., & McDermott, M. (2008). Mixing the digital, social, and cultural: Learning, identity, and agency in youth participation. In D. Buckingham (Ed.), Youth, identity, and digital media (pp. 185–206). Cambridge, MA: MIT Press.Google Scholar
  22. Gutiérrez, K. D., Baquedano-Lopez, P., & Tejada, C. (2000). Rethinking diversity: Hybridity and hybrid language practices in the third space. Mind, Culture, and Activity, 6(4), 286–303.CrossRefGoogle Scholar
  23. Harel, I., & Papert, S. A. (1991). Constructionism. New York: Ablex.Google Scholar
  24. Ito, M. (Ed.). (2009). Hanging out, messing around, and geeking out: Kids living and learning with new media. Cambridge, MA: MIT Press.Google Scholar
  25. Ito, M., Gutiérrez, K. D., Livingstone, S., Penuel, W. R., Rhodes, J. E., Salen, K., et al. (2013). Connected learning: An agenda for research and design. Irvine, CA: Digital Media and Learning Research Hub.Google Scholar
  26. Ito, M., Horst, H. A., Bittani, M., Boyd, D., Stephenson, B. H., Lange, P. G., et al. (2009). Living and learning with new media. Cambridge, MA: MIT Press.Google Scholar
  27. Jenkins, H. (2009). Confronting the challenges of participatory culture: Media education for the 21st century. Cambridge, MA: MIT Press.Google Scholar
  28. Jenkins, H. (2010). Transmedia storytelling and entertainment: An annotated syllabus. Continuum: Journal of Media and Cultural Studies, 24(6), 943–958.CrossRefGoogle Scholar
  29. Jenkins, H. (2012). “Cultural acupuncture”: Fan activism and the Harry Potter Alliance. Transformative Works and Cultures, 10.
  30. Kafai, Y. B., & Peppler, K. A. (2011). Youth, technology, and DIY: Developing participatory competencies in creative media production. Review of Research in Education, 35(1), 89–119.CrossRefGoogle Scholar
  31. Larson, K., Riemer, N., Ackerman, C., Mishel, E., Trent, R., Bradley, E., & Arum, R. (2014). AY2013-14 Hive Networks final report: Connecting youth: Digital Learning Research Project. New York, NY: New York University.Google Scholar
  32. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
  33. Lemke, J. L. (1983). Thematic analysis: Systems, structures, and strategies. Semiotic Inquiry, 3(2), 159–187.Google Scholar
  34. Masters, G. M. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.CrossRefGoogle Scholar
  35. Middaugh, E., & Kirshner, B. (Eds.). (2015). #youthaction: Becoming political in the digital age. Charlotte, NC: Information Age Publishing.Google Scholar
  36. Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6–20.CrossRefGoogle Scholar
  37. National Research Council. (2011). Learning science through computer games and simulations. Washington, DC: National Academies Press.Google Scholar
  38. National Research Council. (2015). Identifying and supporting productive STEM programs in out-of-school settings. Washington, DC: National Research Council.Google Scholar
  39. Penuel, W. R., DiGiacomo, D., & Van Horne, K. (2015). FUSE Studios evaluation report. Boulder, CO: University of Colorado Boulder.Google Scholar
  40. Penuel, W. R., Lee, T. R., & Bevan, B. (2014). Designing for equitable learning across settings. San Francisco, CA: Exploratorium.Google Scholar
  41. Polman, J. L., Saul, E. W., Newman, A., Farrar, C., Singer, N., Turley, E., et al. (2010). A cognitive apprenticeship for science literacy based on journalism. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences (Vol. 2, pp. 61–68). Chicago, IL: International Society of the Learning Sciences.Google Scholar
  42. Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Oxford, UK: Nielsen & Lydiche.Google Scholar
  43. Riconscente, M. M., Kamareinen, A., & Honey, M. (2013). Badge-based STEM Assessment: Current terrain and the road ahead. New York, NY: New York Hall of Science.Google Scholar
  44. Rogoff, B. (2003). The cultural nature of human development. New York: Oxford University Press.Google Scholar
  45. Rogoff, B., Baker-Sennett, J., Lacasa, P., & Goldsmith, D. (1995). Development through participation in sociocultural activity. In J. Goodnow, P. Miller, & F. Kessel (Eds.), Cultural practices as contexts for development. San Francisco: Jossey-Bass.Google Scholar
  46. Salen, K. (Ed.). (2008). The ecology of games: Connecting youth, games, and learning. Cambridge, MA: MIT Press.Google Scholar
  47. Salen, K., Torres, R., Wolozin, L., Rufo-Tepper, R., & Shapiro, A. (2011). Quest to Learn: Developing the school for digital kids. Chicago, IL: The John D. and Catherine T. MacArthur Foundation.Google Scholar
  48. Schwartz, L., & Gutiérrez, K. D. (2013, March). Turn-taking and inventos: Examining the everyday lives of Latino families and designing learning ecologies with youth and undergraduates. Paper presented at the digital media and learning conference, Chicago, IL.Google Scholar
  49. Sebring, P. B., Brown, E. C., Julian, K., Ehrlich, S. B., Sporte, S. E., Bradley, E., & Meyer, L. (2013). YOUmedia Chicago: Teens, digital media, and the Chicago Public Library. Chicago, IL: University of Chicago Consortium on Chicago School Research.Google Scholar
  50. Shute, V. J., Ventura, M., & Torres, R. (2012). Formative evaluation of students at Quest to Learn. International Journal of Learning and Media, 4(1), 55–69.CrossRefGoogle Scholar
  51. Soep, E. (2006). Critique: Assessment and the production of learning. Teachers College Record, 108(4), 748–777.CrossRefGoogle Scholar
  52. Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, “translations” and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–1939. Social Studies of Science, 19(3), 387–420.CrossRefGoogle Scholar
  53. Stevens, R., Satwicz, T., & McCarthy, L. (2008). In-game, in-room, in-world: Reconnecting video game play to the rest of kids’ lives. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning (pp. 41–66). Cambridge, MA: MIT Press.Google Scholar
  54. Subramaniam, M., Ahn, J., & Waugh, A. (2015). The role of school librarians in enhancing science learning. Journal of Librarianship and Information Science, 47(1), 3–16.CrossRefGoogle Scholar
  55. Traphagen, K., & Traill, S. (2014). How cross-sector collaborations are advancing STEM learning. Los Altos, CA: Noyce Foundation.Google Scholar
  56. Tripp, L. (2011). Digital youth, libraries, and new media literacy. The Reference Librarian, 52(4), 329–341.CrossRefGoogle Scholar
  57. Ünlüsoy, A., de Haan, M., Leander, K., & Volker, B. (2013). Learning potential in youth’s online networks: A multilevel approach. Computers & Education, 68, 522–533.CrossRefGoogle Scholar
  58. Watkins, S. C. (2011). Digital divide: Navigating the digital edge. International Journal of Learning and Media, 3(2), 1–12.CrossRefGoogle Scholar
  59. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum.Google Scholar
  60. Wu, M., & Adams, R. J. (2012). Properties of Rasch residual fit statistics. Journal of Applied Measurement, 14, 339–355.Google Scholar
  61. Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A. (2007). Acer ConQuest. Version 2.0. Camberwell, VIC: ACER Press, Australian Council for Educational Research.Google Scholar
  62. Zickuhr, K. (2013). Reading, writing, and research in the digital age. Paper presented at the edUi 2013 annual conference, Richmond, VA.Google Scholar

Copyright information

© Association for Educational Communications and Technology 2016

Authors and Affiliations

  1. 1.Education 3109University of California, Santa BarbaraSanta BarbaraUSA
  2. 2.UCB 249, School of EducationUniversity of Colorado BoulderBoulderUSA
  3. 3.The National Center for the Improvement of Educational Assessment, Inc.DoverUSA
  4. 4.Office of the Vice Provost for Teaching and Learning DepartmentStanford UniversityStanfordUSA
  5. 5.SRI InternationalMenlo ParkUSA

Personalised recommendations