Measuring Literacy, Attitudes, and Capacities to Solve Societal Problems

  • Ellen R. IversonEmail author
  • David Steer
  • Lisa A. Gilbert
  • Kim A. Kastens
  • Kristin O’Connell
  • Cathryn A. Manduca
Part of the AESS Interdisciplinary Environmental Studies and Sciences Series book series (AESS)


Effective assessments offer more than just a measurement of learning. When sequenced and aligned to evidence-based curriculum, assessments can facilitate learning and inform instruction. The Interdisciplinary Teaching about Earth for a Sustainable Future (InTeGrate) project viewed assessment as a core strategy of the curriculum being developed. InTeGrate used a community-based approach to assessment. An assessment team engaged with materials development teams to facilitate module and course assessment design, developed common instruments to assess student learning, analyzed student data from pilot tests to inform curriculum revisions, and used the findings to fine-tune assessment instruments and to shed light on opportunities for faculty development within the geoscience education community. The community approach ensured high-quality curricula and assessments and strategies to investigate learning at the module, course, and project level. Perhaps more importantly, the approach increased the capacity of the geoscience education community by harnessing the expertise from within the community to promote a culture of evaluative thinking.


Assessment Science literacy Developmental evaluation Summative assessment 




David Steer, University of Akron Main Campus (2011–2017)

Ellen Iverson, Science Education Resource Center, Carleton College (2011–2017)

Stuart Birnbaum, University of Texas at San Antonio (2012–2017)

Team Members

Leilani Arthurs, University of Nebraska at Lincoln (2013–2017)

Aida Awad, Broward Community College (2012–2017)

Barbara Bekken, Virginia Polytechnic Institute and State University (2013–2017)

Joshua Caulkins, University of Rhode Island (2012–2017)

Wendy J. Harrison, Colorado School of Mines (2012)

Megan Plenge, University of North Carolina at Chapel Hill (2014–2017)

Sian Proctor, Maricopa County Community College (2012)

Mary Savina, Carleton College (2012–2017)

Susan Sullivan, University of Colorado at Boulder (2012–2017)

Emily Geraghty Ward, Rocky Mountain College (2012–2014)

Karen Viskupic, Boise State University (2012–2017)

Project Liaison

Kristin O’Connell, Science Education Resource Center, Carleton College


  1. AAAS (1993) Benchmarks for science literacy. American Association for the Advancement of Science, Washington, DCGoogle Scholar
  2. AAAS (2011) Vision and change in undergraduate biology education: a call to action. American Association for the Advancement of Science, Washington, DCGoogle Scholar
  3. American Geosciences Institute (2009) Status of the geocience workforce. Accessed 9 Mar 2018
  4. Argyris C (1976) Single-loop and double-loop models in research on decision making. Adm Sci Q:363–375CrossRefGoogle Scholar
  5. Blaschke LM (2012) Heutagogy and lifelong learning: a review of heutagogical practice and self-determined learning. Int Rev Res Open Dist Learn 13(1):56–71Google Scholar
  6. Boix Mansilla V, Duraising ED (2007) Targeted assessment of students’ interdisciplinary work: an empirically grounded framework proposed. J High Educ 78(2):215–237CrossRefGoogle Scholar
  7. Bybee R, McCrae B, Laurie R (2009) PISA 2006: an assessment of scientific literacy. J Res Sci Teach 46(8):865–883CrossRefGoogle Scholar
  8. Champagne AB, Kouba VL (2000) Writing to inquire: written products as performance measures. In: Mintzes JJ, Wandersee JH, Novak JD (eds) Assessing science understanding: a human constructivist view. Elsevier, Burlington, pp 224–246Google Scholar
  9. Egger AE, Kastens KA, Turrin MK (2017) Sustainability, the Next Generation Science Standards, and the education of future teachers. J Geosci Educ 65(2):168–184CrossRefGoogle Scholar
  10. Egger AE, Bruckner MZ, Birnbaum SJ, Gilbert LA (this volume) Facilitating the development of effective interdisciplinary curricular materials. In: Gosselin DC, Egger AE, Taber JJ (eds) Interdisciplinary teaching about earth and the environment for a sustainable future, Association of Environmental Sciences and Studies (AESS) book series. Springer, ChamGoogle Scholar
  11. Fuhrman M (n.d.) Questionnaire to measure indicators for recruitment/retention in geoscience careers, developed to evaluate projects in NSF’s Opportunities for Enhancing Diversity in the Geosciences (OEDG) program. American Institutes for ResearchGoogle Scholar
  12. Gilbert LA, Gross D, Kreutz K (2016) Systems thinking. Accessed 1 Mar 2018
  13. Gilbert LA, Iverson ER, Kastens KA, Awad A, McCauley EQ, Caulkins JL, Steer DN, McConnell DA, Manduca CA (2017) Explicit focus on systems thinking in InTeGrate materials yields improved student performance. Paper presented at the GSA Annual Meeting, Seattle, WA, 22–25 October 2017Google Scholar
  14. Gosselin DC, Manduca CA, Bralower TJ, Egger AE (this volume) Preparing students to address societally relevant challenges in the geosciences: the InTeGrate approach. In: Gosselin DC, Egger AE, Taber JJ (eds) Interdisciplinary teaching about earth and the environment for a sustainable future, AESS Interdisciplinary Environmental Studies and Sciences series. Springer, ChamGoogle Scholar
  15. Houlton HR (2010) Geoscience major career choice: investigation of pathways that lead students to the geosciences. Paper presented at the 2010 GSA Annual Meeting, Denver, COGoogle Scholar
  16. InTeGrate (2012a) InTeGrate assessment team 2012 meeting notes.Google Scholar
  17. InTeGrate (2012b) Your module/course development timeline. Accessed 9 Mar 2018
  18. InTeGrate (2014) Using InTeGrate modules and courses. Accessed 12 Jan 2018
  19. InTeGrate (2015) Developing systems thinking webinar. Accessed 9 Mar 2018
  20. InTeGrate (2017a) Classroom data collection and grading tools. Accessed 9 Mar 2018
  21. InTeGrate (2017b) Implementation programs. Accessed 9 Mar 2018
  22. InTeGrate (2018a) Assessment and project evaluation. Accessed 9 Mar 2018
  23. InTeGrate (2018b) Essay assessments: interdisciplinary and systems thinking. Accessed 9 Mar 2018
  24. InTeGrate (2018c) Faculty surveys. Accessed 9 Mar 2018
  25. InTeGrate (2018d) Geoscience literacy exam (GLE). Accessed 9 Mar 2018
  26. InTeGrate (2018e) HBCU Working Group. Accessed 9 Mar 2018
  27. InTeGrate (2018f) The InTeGrate attitudinal instrument (IAI). Accessed 9 Mar 2018
  28. InTeGrate (2018g) Teach systems thinking. Accessed 9 Mar 2018
  29. Janzen R, Ochocka J, Turner L, Cook T, Franklin M, Deichert D (2017) Building a community-based culture of evaluation. Eval Program Plann 65:163–170CrossRefGoogle Scholar
  30. Johnson R, Snow J, Abshire W, Buhr S, Cullen H, Denning S, Holland M, Manduca C, Pennington P, Schoedinger S, Schultz P, Shepherd JP (2009) Essential principles and fundamental concepts for atmospheric science literacy. NSF: Geoscience Education ProgramGoogle Scholar
  31. Kastens KA, Manduca CA (2017) Using systems thinking in the design, implementation, and evaluation of complex educational innovations, with examples from the InTeGrate project. J Geosci Educ 65(3):219–230CrossRefGoogle Scholar
  32. Kastens KA, Manduca CA (2018) Leveraging the power of community of practice to improve teaching and learning about the Earth. Change: The Magazine of Higher LearningGoogle Scholar
  33. Kastens KA, Mara V (2016) InTeGrate evaluation report: underrepresented minoritiesGoogle Scholar
  34. Kastens KA, Mara V (2017) InTeGrate evaluation report: IAI responses of InTeGrate versus non-InTeGrate studentsGoogle Scholar
  35. Kastens KA, Mara V (2018) Where is InTeGrate having its impact on Earth and environmental motivation and action? An analysis of IAI data with respect to student’s reason for taking the courseGoogle Scholar
  36. Labin SN, Duffy JL, Meyers DC, Wandersman A, Lesesne CA (2012) A research synthesis of the evaluation capacity building literature. Am J Eval 33(3):307–338CrossRefGoogle Scholar
  37. LaDue ND, Clark SK (2012) Educator perspectives on Earth system science literacy: challenges and priorities. J Geosci Educ 60(4):372–383CrossRefGoogle Scholar
  38. Libarkin JC, Anderson SW (2005) Assessment of learning in entry-level geoscience courses: results from the Geoscience Concept Inventory. J Geosci Educ 53(4):394–401CrossRefGoogle Scholar
  39. Manduca CA, Iverson ER, Luxenberg M, Macdonald RH, McConnell DA, Mogk DW, Tewksbury BJ (2017) Improving undergraduate STEM education: the efficacy of discipline-based professional development. Sci Adv 3(2):e1600193CrossRefGoogle Scholar
  40. National Academies of Sciences, Engineering, and Medicine (2016) Science literacy: concepts, contexts, and consequences. The National Academies Press, Washington, DC. Scholar
  41. NOAA, National Geographic Society, COSEE, National Marine Educators Association (2005) Ocean literacy: the essential principals of ocean sciences K-12. Accessed 9 Mar 2018
  42. Novak JD, Mintzes JJ, Wandersee JH (2000) Learning, teaching, and assessment: a human constructivist perspective. In: Mintzes JJ, Wandersee JH, Novak JD (eds) Assessing science understanding: a human constructivist view. Elsevier, Burlington, pp 1–13Google Scholar
  43. On the Cutting Edge (2016) About the Cutting Edge embedded assessment project. Accessed 1 Mar 2018
  44. Orr CH, McDaris JR (this volume) Supporting implementation of program-level changes to increase learning about Earth. In: Gosselin DC, Egger AE, Taber JJ (eds) Interdisciplinary teaching about earth and the environment for a sustainable future, Association of Environmental Sciences and Studies (AESS) book series. Springer, ChamGoogle Scholar
  45. Patton MQ (2010) Developmental evaluation: applying complexity concepts to enhance innovation and use. Guilford Press, New YorkGoogle Scholar
  46. PISA (2015) Program for international student assessment,
  47. Preskill H, Boyle S (2008) A multidisciplinary model of evaluation capacity building. Am J Eval 29(4):443–459CrossRefGoogle Scholar
  48. Rockman (2013) On the Cutting Edge project: year 3 independent evaluation report. Accessed 10 Mar 2018
  49. SERC (2017) Serkit. Accessed Dec 2017
  50. Steer D, Iverson ER, Manduca CA (2013) Piloting a geoscience literacy exam for assessing students’ understanding of Earth, climate, atmospheric and ocean science concepts. Paper presented at the AGU Fall Meeting Abstracts, San Francisco, CA, 9–13 December 2013Google Scholar
  51. Steer D, Iverson ER, Egger AE, Kastens KA, Manduca CA, McConnell D (this volume) The InTeGrate materials development rubric: a framework and process for developing curricular materials that meet ambitions goals. In: Gosselin DC, Egger AE, Taber JJ (eds) Interdisciplinary teaching about Earth and the environment for a sustainable future, Association of Environmental Sciences and Studies (AESS) book series. Springer, ChamGoogle Scholar
  52. U.S. Global Change Research Program (2009) Climate literacy: essential principles and fundamental concepts. National Oceanic and Atmospheric Administration and the American Association for the Advancement of ScienceGoogle Scholar
  53. Viskupic K, Steer D, Kortz KM, Perkins D, Wirth K, Herbert B, Singer J, Manduca CA (2014) Monitoring undergraduate growth through the major using embedded assessment questions, Geological Society of America Abstracts with Programs, Vancouver, BC 46(6)Google Scholar
  54. Wiggins G, McTighe J (1998) Understanding by design. Association for Supervision and Curriculum Development, AlexandriaGoogle Scholar
  55. Wysession ME (2012) Implications for earth and space in new K–12 science standards. Eos Trans Am Geophys Union 93(46):465–466CrossRefGoogle Scholar
  56. Wysession M, Taber J, Budd DA, Campbell K, Conklin M, LaDue N, Lewis G, Raynolds R, Ridky R, Ross R, Tewksbury B, Tuddenham P (2009) Earth science literacy: the big ideas and supporting concepts of Earth science. NSF: The Earth Science Literacy InitiativeGoogle Scholar
  57. Wysession ME, LaDue N, Budd DA, Campbell K, Conklin M, Kappel E, Lewis G, Raynolds R, Ridky RW, Ross RM (2012) Developing and applying a set of earth science literacy principles. J Geosci Educ 60(2):95–99CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ellen R. Iverson
    • 1
    Email author
  • David Steer
    • 2
  • Lisa A. Gilbert
    • 3
  • Kim A. Kastens
    • 4
  • Kristin O’Connell
    • 1
  • Cathryn A. Manduca
    • 1
  1. 1.Science Education Resource CenterCarleton CollegeNorthfieldUSA
  2. 2.Department of GeosciencesThe University of AkronAkronUSA
  3. 3.Williams College, Geosciences and Marine Science at Williams-MysticMysticUSA
  4. 4.Lamont-Doherty Earth ObservatoryColumbia UniversityNew YorkUSA

Personalised recommendations