Skip to main content

Advertisement

SpringerLink
Log in

Concurrent think-aloud protocols to assess elementary design students

  • Published:
International Journal of Technology and Design Education Aims and scope Submit manuscript

Abstract

Initiatives to integrate engineering design in the elementary science classroom have become increasingly evident in both national reform documents and classroom practice. Missing from these efforts is a purposeful attempt to capture students’ designerly thinking and dialogues as they engage in the process. The purpose of this study was to investigate how elementary school students approach and engage in engineering design using concurrent think-aloud protocols. Data from seven concurrent think-aloud protocols among triads of elementary students across seven classrooms were analyzed to identify how students conceptualize design. Researchers employed a transfer problem and think-aloud protocol analysis to assess students’ transfer of learning from classroom science based engineering design-based experiences. Results indicate that elementary student triad design teams were able to define a design problem, identify constraints and criteria, and generate multiple design ideas to solve the problem. Protocol timelines were generated using NVivo software to capture sequence of the triads’ coded cognitive strategies crucial in understanding which triads used a systematic approach to solving the problem from triads that randomly brainstormed ideas. If design is to become a pedagogical approach to teaching science or other STEM-related subjects, attention must be given to how students learn design and function within design. Concurrent think-aloud protocol provides a promising means of assessment of such efforts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of the reflective practice. Design Studies, 24(3), 275–294.

    Article  Google Scholar 

  • Atman, C. J., & Bursic, K. M. (1998). Verbal protocol analysis as a method to document engineering student design processes. Journal of Engineering Education, 87(2), 121–132.

    Article  Google Scholar 

  • Ball, L. J., Ormerod, T. C., & Morley, N. J. (2004). Spontaneous analogising in engineering design: A comparative analysis of experts and novices. Design Studies, 25(5), 495–508.

    Article  Google Scholar 

  • Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience and school. Washington, DC: National Academy Press.

    Google Scholar 

  • Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research in education (Vol. 24, pp. 61–100). Washington, DC: American Educational Research Association.

    Google Scholar 

  • Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 1, pp. 77–165). Hillsdale: Erlbaum.

    Google Scholar 

  • Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usable test method: a technical communication approach. IEEE Transactions on Professional Communication, 53(3), 202–215.

    Article  Google Scholar 

  • Cross, N. (1994). Engineering design methods, strategies for product design (2nd ed.). Chichester: Wiley.

    Google Scholar 

  • Cross, N., & Cross, C. A. (1996). Winning by design: The methods of Gordon Murray, race car designer. Design Studies, 17(1), 91–107.

  • Cross, N., & Cross, A. C. (1998). Expertise in engineering design. Research in Engineering Design, 10(3), 141–149.

    Article  Google Scholar 

  • Cross, N., & Dorst, K. (1999). Co-evolution of problem and solution spaces in creative design. In J. S. Gero & M. L. Maher (Eds.), Computational models of creative design IV (pp. 243–262). Sydney: Key Centre of Design Computing and Cognition, University of Sydney.

    Google Scholar 

  • Department for education and skills. (1999). Design and technology the national curriculum for England. London: Department For Education and Skills.

    Google Scholar 

  • Dixon, R., & Brown, R. (2012). Transfer of learning: connecting concepts during problem solving. Journal of Technology Education, 24(1), 2–17.

    Google Scholar 

  • Donovan, M., Bransford, J., & Pellegrino, J. (Eds.). (1999). How people learn: Bridging research and practice. Washington, DC: National Academy Press.

    Google Scholar 

  • Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem–solution. Design Studies, 22(1), 425–437.

    Article  Google Scholar 

  • Dunker, K. (1926). A qualitative (experimental and theoretical) study of productive thinking (solving of comprehensible problems). Pedagogical Seminary, 33, 642–708.

    Article  Google Scholar 

  • Dym, C., Agogino, A., Eris, O., Frey, D., & Leifer, L. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 95(1), 103–120.

    Article  Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

    Google Scholar 

  • Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence. NJ: Erlbaum.

    Google Scholar 

  • Fortus, D., Dershimer, C., Krajcik, J., Marx, R., & Mamlok-Naaman, R. (2004). Design-based science and student learning. Journal of Research in Science Teaching, 41(10), 1081–1110.

    Article  Google Scholar 

  • Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston: Pearson Education Inc.

    Google Scholar 

  • Halfin, H. H. (1973). Technology: A process approach. (Doctoral dissertation, West Virginia University, 1973) dissertation Abstracts International, 11(1), 1111A.

  • Hennessy, S., & Murphy, P. (1999). The potential for collaborative problem solving in D&T. International Journal of Technology and Design Education, 9, 1–36.

    Article  Google Scholar 

  • Hill, R. B. (1997). The design of an instrument to assess problem solving activities in technology education. Journal of Technology Education, 9(1), 31–46.

    Google Scholar 

  • Hill, R. B. (2006). New perspectives: Technology teacher education and engineering design. Journal of Industrial Teacher Education, 43(3), 45–63.

    Google Scholar 

  • Indiana Department of Education. (2012, April). Retrieved from http://compass.doe.in.gov/dashboard/overview.aspx.

  • Jain, V. K., & Sobek, D. K., I. I. (2006). Linking design process to customer satisfaction through virtual design of experiments. Research in Engineering Design, 17(2), 59–71.

    Article  Google Scholar 

  • Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85.

    Article  Google Scholar 

  • Kangas, K., Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2011). Figuring the world of designing: Expert participation in elementary classroom. International Journal of Technology and Design Education, 23(2), 425–442.

  • Kangas, K., Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2013). Design expert’s participation in elementary students’ collaborative design process. International Journal of Technology Design Education, 23(1), 161–178.

    Article  Google Scholar 

  • Kilgore, D., Atman, C. J., Yasuhara, K., Barker, T. J., & Morozov, A. (2007). Considering context: A study of first year engineering students. Journal of Engineering Education, 96(4), 321–334.

    Article  Google Scholar 

  • Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntembakar, S., & Ryan, M. (2003). Problem-based learning meets case based reasoning in middle school science classroom: Putting learning by designTM into practice. Journal of the Learning Sciences, 12(4), 495–547.

  • Kruger, C., & Cross, N. (2001). Modeling cognitive strategies in creative design. In J. Gero & M. Maher (Eds.), Computational and cognitive models of creative design V. Australia: University of Sidney.

    Google Scholar 

  • Kruger, C., & Cross, N. (2006). Solution driven versus problem driven design: strategies and outcomes. Design Studies, 27(5), 527–548.

    Article  Google Scholar 

  • Lave, J., & Wenger, E. (1991). Situated learning. Legitimate peripheral participation. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Merriam, S. B. (2001, 1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass.

  • Ministry of Education. (2007). The New Zealand curriculum. Wellington: Learning Media.

    Google Scholar 

  • National Research Council. (2012). A framework for K12 science education: Practices, cross cutting concepts, and core ideas. Washington, DC: National Academies Press.

    Google Scholar 

  • Nicholl, B., Hosking, I., Elton, E., Lee, Y., Bell, J., & Clarkson, P. (2013). Inclusive design in the Key Stage 3 classroom: An investigation of teachers’ understanding and implementation of user-centered design principles in design and technology. International Journal of Technology Design Education, 23(4), 921–938.

    Article  Google Scholar 

  • Patrick, C. (1935). Creative thought in poets. Achieves of Psychology, 178, p. 74.

  • Penner, D. E., Giles, N. D., Lehrer, R., & Schauble, L. (1997). Building functional models: Design an Elbow. Journal of Research in Science Teaching, 34(2), 125–143.

    Article  Google Scholar 

  • Perez, R. S., Fleming Johnson, J., & Emery, C. D. (1995). Instructional design experts: A cognitive model of design. Instructional Science, 23(5–6), 321–349.

    Article  Google Scholar 

  • Prevost, A., Nathan, M., Stein, B., Tran, N., & Phelps, A. (2009). Integration of mathematics in pre-college engineering: The search for explicit connections. In Proceedings of the American Society of Engineering Education (ASEE) Annual Conference and Exposition. Austin, TX: ASEE.

  • Rennie, L., Venville, G., & Wallace, J. (2012). Integrating science, technology, engineering, and mathematics: Issues, reflections, and ways forward. New York: Rutledge.

    Google Scholar 

  • Robertson, W. C. (1990). Detection of cognitive structure with protocol data: Predicting performance on physics transfer problems. Cognitive Science, 14(1), 253–280.

    Article  Google Scholar 

  • Ross, B. H. (1984). Reminding and their effects in learning a cognitive skill. Cognitive Psychology, 16(1), 371–416.

    Article  Google Scholar 

  • Royer, J. M. (1986). Designing instruction to produce understanding: An approach based on cognitive theory. In G. D. Phye & T. Andre (Eds.), Cognitive classroom learning: understanding, thinking and problem solving (pp. 83–113). Orlando, FL: Academic Press Inc.

    Google Scholar 

  • SchÖn, D. (1983). The reflective practitioner. New York: Basic Books.

  • Smith, C.O. (1971). The structure of intellect processes analyses system. A technique for the investigation and qualification of problem solving processes. Doctoral Dissertation, University of Huston.

  • van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to modeling cognitive processes. London: Academic Press.

    Google Scholar 

  • Welch, M. (1999). Analyzing the tacit strategies of novice designers. Research in Science & Technical Education, 17(1), 19–34.

    Article  Google Scholar 

Download references

Acknowledgments

This work was made possible by a Grant by the National Science Foundation (Grant # 0962840). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Technology Leadership and Innovation, Purdue University, Young Hall, Room 342, 155 S. Grant Street, West Lafayette, IN, 47907-2114, USA

    Todd R. Kelley

  2. Department of Curriculum and Instruction, Purdue University, Beering Hall, 100 North University St., West Lafayette, IN, 47907-2098, USA

    Brenda M. Capobianco

  3. Purdue University, 155 S. Grant Street, West Lafayette, IN, 47907-2114, USA

    Kevin J. Kaluf

Authors
  1. Todd R. Kelley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brenda M. Capobianco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin J. Kaluf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Todd R. Kelley.

Appendices

Appendix 1: Design a prosthetic leg to kick a soccer ball

figure a

Boiler BioTech, a company in Warsaw, Indiana, needs assistance in designing a leg for a young child. The prosthetic leg will need to be designed so that it will be able to kick a soccer ball. Everyone faces challenges every day. To help us, engineers have designed glasses for people who need help seeing, hearing aides for people who need help hearing, crutches and canes for people who need help with bearing weight, and artificial limbs for people who have lost a limb. Designing aids for all of these human needs requires understanding what function you are augmenting and lots of creativity. In this unit we are going to learn about the musculoskeletal system and then you will be given an opportunity to test your design skills by building a prosthetic leg and test it by using it to kick a ball.

During the lesson you will:

  • Design a prosthetic leg to kick a ball.

  • Measure the volume of different types of balls.

  • Find the weight of the balls.

  • Kick different types of balls with your prototype to see which one goes the farthest.

Design constraints:

  • The leg should hinge like a real joint (move back and forth).

  • The leg is being designed to strike a ball (move the ball or propel it on it’s own). Typically, the rubber bands would be used to make the spring loaded leg snap to propel the ball.

  • A list of items and potential monetary value is provided. Students may be asked to determine how much their design costs and they can “buy” additional items, if needed.

  • Elapsed time can be recorded for further math exercises.

  • The lesson is meant to design something that functions like a leg when it kicks a soccer ball.

  • Students do not have to mimic anatomy.

Appendix 2: Paper football kicker

figure b

The problem

Your younger brother Joey is the Recess Paper Football Champion of his grade, but he’s bummed he can’t play since he broke his right “kicking” finger playing basketball. Joey’s friends say if he can come up with something that flicks the football for him, they’ll let him keep playing, but Joey knows he can’t kick paper footballs with his opposite hand for accuracy. Joey heard you talking about learning about prosthetic limbs, so he thinks you can help him out by designing a device that will kick the paper football for him.

Recess Paper Football game is played using two goals posts—one 3 ft away, and one 5 ft away—so your device must be accurate to these varying lengths.

Your brother Joey is looking for the following design features for this paper football kicker. Your design should be able to:

  • hinge like a real-jointed finger that is flicking the paper football.

  • be designed to strike a paper football and propel it far enough to go through the goalposts.

  • be accurate at various distances (3–5 ft) Take up the floor space no larger than a typical textbook.

Your task

Describe how you would design a paper football flicker to flick paper footballs different distances in a fun and creative way. Please describe aloud how you would start the design task—where would you begin? How would you design the device to include all the features listed above? What types of tests would you conduct to ensure that your device works for both desired distances?

Appendix 3

Cognitive processes identified by Halfin’s (1973) study of high-level designers (nine of the 17 total codes that emerged in the CTA sessions)

Proposed mental methods

 

Definition

Analyzing

AN

The process of identifying, isolating, taking apart, breaking down, or performing similar actions for the purpose of setting forth or clarifying the basic components of a phenomenon, problem, opportunity, object, system, or point of view

Computing

CO

The process of selecting and applying mathematical symbols, operations, and processes to describe, estimate, calculate, quantity, relate, and/or evaluate in the real or abstract numerical sense

Defining problem(s)

DF

The process of stating or defining a problem which will enhance investigation leading to an optimal solution. It is transforming one state of affairs to another desired state

Designing

DE

The process of conceiving, creating inventing, contriving, sketching, or planning by which some practical ends may be effected, or proposing a goal to meet the societal needs, desires, problems, or opportunities to do things better. Design is a cyclic or iterative process of continuous refinement or improvement

Interpreting data

ID

The process of clarifying, evaluating, explaining, and translating to provide (or communicate) the meaning of particular data

Modeling

MO

The process of producing or reducing an act, or condition to a generalized construct which may be presented graphically in the form of a sketch, diagram, or equation; presented physically in the form of a scale model or prototype; or described in the form of a written generalization

Predicting

PR

The process of prophesying or foretelling something in advance, anticipating the future on the basis of special knowledge

Questions/hypotheses

QH

Questioning is the process of asking, interrogating, challenging, or seeking answers related to a phenomenon, problem, opportunity element, object, event, system, or point of view

Testing

TE

The process of determining the workability of a model, component, system, product, or point of view in a real or simulated environment to obtain information for clarifying or modifying design specifications

Appendix 4

See Fig. 4.

Fig. 4
figure 4

Black and white pie charts for Fig. 1

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelley, T.R., Capobianco, B.M. & Kaluf, K.J. Concurrent think-aloud protocols to assess elementary design students. Int J Technol Des Educ 25, 521–540 (2015). https://doi.org/10.1007/s10798-014-9291-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10798-014-9291-y

Keywords

Search

Navigation