Learning Science in Grades 3–8 Using Probeware and Computers: Findings from the TEEMSS II Project
- First Online:
- Cite this article as:
- Zucker, A.A., Tinker, R., Staudt, C. et al. J Sci Educ Technol (2008) 17: 42. doi:10.1007/s10956-007-9086-y
- 270 Downloads
The Technology Enhanced Elementary and Middle School Science II project (TEEMSS), funded by the National Science Foundation, produced 15 inquiry-based instructional science units for teaching in grades 3–8. Each unit uses computers and probeware to support students’ investigations of real-world phenomena using probes (e.g., for temperature or pressure) or, in one case, virtual environments based on mathematical models. TEEMSS units were used in more than 100 classrooms by over 60 teachers and thousands of students. This paper reports on cases in which groups of teachers taught science topics without TEEMSS materials in school year 2004–2005 and then the same teachers taught those topics using TEEMSS materials in 2005–2006. There are eight TEEMSS units for which such comparison data are available. Students showed significant learning gains for all eight. In four cases (sound and electricity, both for grades 3–4; temperature, grades 5–6; and motion, grades 7–8) there were significant differences in science learning favoring the students who used the TEEMSS materials. The effect sizes are 0.58, 0.94, 1.54, and 0.49, respectively. For the other four units there were no significant differences in science learning between TEEMSS and non-TEEMSS students. We discuss the implications of these results for science education.
KeywordsTechnology Probes Elementary Middle
At its core science is about investigating, exploring, asking questions, analyzing, and thinking. Digital technology is uniquely able to support observation and inquiry in ways that are largely lacking in elementary science teaching (Linn 2003; Lunetta et al. 2007). A substantial body of research shows that probeware can facilitate student learning of complex relationships (Adams and Shrum 1990; Beichner 1990; Friedler et al. 1990; Krajcik and Layman 1993; Laws 1997; Linn et al. 1987; Millar 2005).
The goal of the Technology Enhanced Elementary and Middle School Science II (TEEMSS) project was to address the need for inquiry-based instructional materials for elementary and middle school science teaching that make use of computers and probeware. For the TEEMSS project the Concord Consortium created and is disseminating 15 easily implemented technology-based instructional units for teaching and learning science in grades 3–8. The project also supported associated teacher professional development, as well as research focusing on the effectiveness of this approach to learning science.
The principal research question investigated by the project was whether students using probeware and the TEEMSS materials would learn more science than students learning the same science topics in matched comparison classrooms not using TEEMSS and probeware. This question is important because, notwithstanding many research studies focusing on upper grade levels, there is a scarcity of research and data about the efficacy of using probes and computers to teach and learn science in elementary and middle schools. What research does exist has often been done with small numbers of students. For example, a recent study of probeware (also called microcomputer-based labs, or MBLs) that reported significant positive impacts favoring the use of probes was based on a sample of only 65 fourth-grade students (Nicolaou et al. 2007). One goal of TEEMSS was to conduct trials and research with much larger numbers of students.
The TEEMSS Units
The Technology Enhanced Elementary and Middle School Science II project materials are designed to work with whatever computers and probeware schools choose to use. The project selected age-appropriate, standards-based topics for which technology offers real advantages. The units are modular so they can be integrated with existing curricula or used on their own. The TEEMSS learning strategy is based on student investigations of real phenomena using probes (and, in the Adaptation unit, a virtual environment based on mathematical models). To support student learning, the project produced software (including SensorPortfolio, the computer-based learning environment for the units), curriculum materials, and an online course to prepare teachers.
The Technology Enhanced Elementary and Middle School Science II project was an extension of a prior TEEMSS pilot project funded by the NSF IMD program (grant no 9986419). The pilot project demonstrated the soundness of the overall approach and funded the development of a structure for the materials that was used in TEEMSS. The pilot project focused on developing two units and testing them with three groups of teachers, supporting one group of teachers with a face-to-face workshop and the other groups exclusively with online courses. Units were developed to address middle school force and motion and energy transfer standards of the National Science Education Standards (NSES). Because teachers report that these standards are difficult to achieve, they represented a challenging test of our approach. Pre- and post-tests aligned to the standards were given to students to measure their learning gains (Metcalf and Tinker 2004).
The Technology Enhanced Elementary and Middle School Science II project produced 15 units keyed to the NSES that take advantage of computers, sensors, and in one unit, Adaptation, an interactive computer model. Five units each were developed for grade levels 3–4, 5–6, and 7–8, with each set of units targeting the five NSES standards: Inquiry, Physical Science, Life Science, Earth and Space Science, and Technology and Design, as well as the National Council of Teachers of Mathematics (NCTM) standards. The units are now available, free of charge, at http://www.teemss.concord.org/. Many thousands of copies have been downloaded.
The TEEMSS curriculum units
Sound: Explore sound and vibrations with the SoundGrapher
Water and air temperature: Mix fluids and measure temperature changes with a temperature sensor
Air pressure: Explore soda bottle, balloons and lungs with a gas pressure sensor
Electricity: Explore light bulbs, batteries, and wires using a voltage sensor
Levers and machines: Design and test your own compound machine with a force sensor
Motion: Graph, describe, and duplicate motion using a motion sensor
Sensing: Compare electronic and human sensing of your environment using temperature and light sensors
Monitoring a living plant: Monitor a living plant in a plastic bag with relative humidity and light sensors
Adaptation: Explore population, selection pressure and adaptation with a computer model
Earth and space science
Weather: Observe and measure weather-related changes with temperature and relative humidity sensors
Sun, Earth, Seasons: Connect planetary motion to day/night cycles and seasons with a light sensor
Water cycle: Study water phase changes and relate to terrestrial phenomena with temperature and light sensors
Design a playground: Study your playground and build models of several pieces of playground equipment using force and motion sensors
Design a greenhouse: Build a working greenhouse model and monitor conditions using temperature, light, and relative humidity sensors
Design a measurement: Choose something to measure and devise a way to do it using any or all of the sensors
Technology and Science Education
Computers and other digital technologies are an essential part of modern science, but they are not yet widely used in elementary and middle school science education. For example, in 2000 fewer than one-third of science teachers in grades 5–8 reported ever collecting data using sensors or probes (Hudson et al. 2002). Yet there is widespread agreement that computer and information technologies should be an integral part of elementary and middle school science teaching, in ways that can greatly improve learning. The NSES (NRC 1995), the Benchmarks for Science Literacy (AAAS 1993) and many state education standards require the integration of technology into science teaching and learning starting as early as first grade, as a way to facilitate student inquiry and enhance students’ understanding of temperature, pressure, and many other phenomena.
The TEEMSS pilot study was useful in that it found that handhelds and probes can be effective in inquiry learning environments at the middle school level (Metcalf and Tinker 2004), and that online teacher professional development could effectively prepare teachers to use inquiry-based materials. Nonetheless, the pilot study did not include a comparison group. The current TEEMSS project was funded in part to conduct more rigorous research on the efficacy of using probes and computers to teach science in grades three through eight and report evidence of the effectiveness of technology-based materials.
Another major goal of the TEEMSS project was to explore ways of bridging the “digital divide” by reducing the cost to schools of implementing ICT-based instructional materials. This objective led us to examine to what extent inexpensive handheld computers could be used instead of networked desktop computers. As a result, all TEEMSS classroom units used handhelds, but the probeware and curriculum materials could be used on desktops (Mac or PC) as well. Both handhelds and desktop computers were used in the research.
The Technology Enhanced Elementary and Middle School Science II project was designed so that probes from any of the major vendors could be used. Our commitment to reducing costs also led to the development of an innovative design for low-cost probeware.
A variety of other TEEMSS features also made use of many of the affordances of computers. For example, students were able to use TEEMSS software to draw graphs that they predicted would show the data collected and displayed by the probes and software. Graphs based on actual data collected by the probes could then be overlayed (in another color) on students’ predictions. The TEEMSS software could save these and other data (including responses to embedded assessment questions) and then retrieve the data for use at a later date.
As part of the project, TEEMSS units were used in classrooms during academic years 2004–2005, 2005–2006, and 2006–2007. Altogether 66 teachers, who were located in more than a dozen school districts in three states, used one or more of the units during those years. Data from these teachers and the students they taught provide a rich source of information about both the implementation of the units and student learning outcomes. The data about student learning reported in this paper come from a subset of the 66 teachers, as described below.
An online course was developed to provide teachers with information about the TEEMSS materials, pedagogy, and technology (including sensors and probeware). In 2005–2006, among the teachers taking the online course, the average number of “hits” per teacher was 261.
This paper focuses on eight TEEMSS units, those for which pre- and post-test data are available for classes using the TEEMSS materials and for comparison classes learning the same topics but without using the TEEMSS materials. (Three other TEEMSS units—the design units, which focus on the NSES standard for Technology/Engineering—used embedded performance assessments but did not include pre- and post-tests. For the remaining four units, data were not collected from classes learning the science content but without using TEEMSS materials, making it impossible to compare learning of TEEMSS and non-TEEMSS classes.)
Teachers and Students Participating in the TEEMSS Research
Number of teachers (number of students) used in data analysis
Human and electronic sensing
Water and air temperature
Levers and machines
Monitoring a living plant
A number of comparisons were made between students studying the same science topics either without using the TEEMSS materials or with the TEEMSS materials. This paper focuses on the most rigorous comparison, namely cases in which groups of teachers taught science topics without TEEMSS materials in school year 2004–2005 and then the same teachers taught those topics again using TEEMSS materials in school year 2005–2006. (These teachers are included under the 2004–2005 non-TEEMSS column in Table 2 and then again in the 2005–2006 TEEMSS column.) By focusing on groups consisting of the same teachers teaching in two different ways, one important potential threat to the validity of the findings can be eliminated because in this case there were no differences between teachers in the experimental (TEEMSS) and the comparison (non-TEEMSS) classes.
Measures of Student Learning
Items on the TEEMSS unit tests were primarily drawn from 12 existing standardized tests, including NAEP and TIMSS, as well as regional and state tests with similar item construction (Kreikemeier et al. 2006). SRI International, a subcontractor for the project, prepared binders that collected the more than 1,500 items initially identified as potentially relevant and organized these items by TEEMSS curricular unit. Concord Consortium staff reviewed the binders to determine item appropriateness for and alignment with TEEMSS curriculum. Approximately 380 items were identified as sufficiently aligned, and these were selected for use during the piloting stage of test development. Because several units still did not have adequate numbers of potential assessment items, SRI and Concord collaborated on developing unit-specific questions developed for TEEMSS to supplement those collected from other sources.
Tests for the eight units shown in Table 2 varied in length from 5 to 9 items. All of the tests were piloted in November 2004. Small numbers of students were asked to “think aloud” while answering test questions in order to identify student comprehension of questions. Finally, 60–100 students from Massachusetts completed each test. Validity testing was used for multiple purposes: to select questions that were appropriate for the target grade level, to evaluate inter-rater reliability for scoring, and to compare student performance on matched pre/post variations of questions. The items that performed the best across this range of priorities were included on the final tests. (Also, a few items were included on the final forms that had not been field-tested or reviewed by the outside science expert for scientific accuracy.)
The differences between the pretest and posttest forms used in the study were minimal, limited primarily to the order of the answer choices and the presentation of slightly different surface features (e.g., changing values of temperature readings for the prompts on a multiple choice test).
SRI International was responsible for scoring student work. Scoring conformed to standard practices; namely, each rater scored a single item for all student samples before being trained, raters were then trained to score using anchor and discussion papers, raters were allowed to score actual student work only after scoring qualification samples with 80% reliability, and a minimum of 20% of student-constructed (open-ended) responses were scored by two raters whose scores were checked by a third person who resolved any discrepancies in the scores. Before the student data were analyzed, inter-rater reliability was verified for constructed response items scored by two people. The results indicated that average agreement was 74% across all units on the pretest and 76% across all units on the posttest, somewhat lower than the 80% usually desired.
Scoring was done blind. That is, the scorers did not know whether the papers they scored were pre- or post-tests, or the names of teachers or students.
Measures of Teachers’ and Students’ Attitudes and Experiences
The project collected information about students’ and teachers’ experiences with TEEMSS from several sources. Teachers completed written surveys both before and after they participated in the project. In addition, after teaching a TEEMSS unit the teachers completed a special survey (“signpost”) providing information about their experience with that unit. A post-survey was also administered to nearly 1,000 students at the end of the 2005–2006 school year.
Data reported here about teachers’ and students’ experiences using the materials come from all the teachers and students using TEEMSS during the 2005–2006 school year. However, data about students’ science learning reported below are from a subset of all participants, namely for the groups of teachers described in the Research Design section.
Teachers’ and Students’ Attitudes and Experiences
Teachers reported that the TEEMSS materials were age appropriate. On a scale of 1 (much too easy) to 5 (much too hard), teachers rated the units at 3.3, or almost exactly age appropriate.
Teachers’ ratings of TEEMSS technology
Easy for me to learn the sensors
Easy for me to learn the software
Easy for my students to learn
Easy to use once we’ve learned it
Applicable to a broad range of science content
Useful for teaching science content
I was able to solve technical problems that occurred
Students’ ratings of TEEMSS
Seeing the graphs
Designing your own experiments
Working with a partner or team
The students were very surprised that still air and moving air were the same temperature. They kept measuring it to make sure.
We were working with the water cycle unit when a storm began to move in. The kids themselves noticed how much higher the relative humidity readings became and how much more ‘muggy’ the room became.
Both TEEMSS and non-TEEMSS students showed statistically significant gains on all eight of the unit tests (from pre- to post-). In other words, students learned science content whether taught through traditional means or using TEEMSS.
A comparison of students’ gains in two conditions, non-TEEMSS and TEEMSS
Gain, mean (SD)
Gain, mean (SD)
t (152.039) = 4.282, p < 0.001
t (134) = 5.406, p < 0.001
No sig. difference
t (99) = 7.737, p < 0.01
No sig. difference
No sig. difference
No sig. difference
t (97) = 2.369, p = 0.02
For the other four units for which these comparison data are available there were no statistically significant differences between students who did and did not use TEEMSS materials. Those four unit were sensing (grades 3–4); levers and machines, and monitoring a living plant (grades 5–6); and pressure (grades 7–8).
There does not seem to be a simple answer to the question why students using probeware and computers scored higher than non-TEEMSS students on some units but not on others. A number of possible explanatory factors can be ruled out. For example, teachers reported spending the same amount of time on the units that did not show benefits for TEEMSS as on the units where the TEEMSS students did better. Similarly, teachers’ reports of students’ levels of engagement also did not explain why sensors and probeware were more beneficial on some units than others. One likely explanation is that students benefit from the use of sensors and probeware when they study particular topics and particular concepts more than others. For example, students using TEEMSS developed a better understanding of graphs than those who did not use TEEMSS, a result consistent with prior research (Beichner 1990; Brassell 1987; Mokros and Tinker 1987; Nicolaou et al. 2007). Graphs typically play a more important role in learning some science concepts than others. The TEEMSS unit on water and air temperature relies heavily on students’ understanding graphs of temperature changes over time and the effect size was greatest for that unit, as shown in Table 5. In contrast, sensors and probeware had little or no benefit on recall of science-related facts, such as learning the names of classes of levers and simple machines. One of the units focused on levers and machines and there was no significant difference between TEEMSS and non-TEEMSS students for that unit.
Discussion and Conclusions
Computers are becoming more common in schools than ever before. Entire states are implementing laptop programs, including Maine (which has focused on middle and high schools) and Pennsylvania (which is focusing on high schools). Many other states (including Indiana, Massachusetts, Michigan, New Hampshire, New Mexico, Texas, and Vermont) and hundreds of schools and districts are implementing “one-to-one” computing programs on a smaller scale (for further information see, for example, http://www.ubiqcomputing.org, http://www.k12one2one.org/ or Bonifaz and Zucker 2004). Moreover, virtually every school in the United States is now connected to the Internet. Internet access is available not only in almost all schools but in almost all (94%) classrooms (Wells et al. 2006). The rapid growth of wireless networks in schools has made it much less expensive to provide ubiquitous Internet access throughout school buildings.
In this technology-rich environment, it is more important than ever to document ways that technology can enhance the teaching and learning of science. Data from the TEEMSS project demonstrate that using computers and probes in elementary and middle school classrooms can result in substantial learning gains. For certain curriculum units, use of computers and probes results in larger learning gains than instruction on the same topics without computers and probes.
Findings from this study are consistent with earlier research about the use of technology in science education. Many prior studies have reported positive impacts of using digital technology for teaching science, including a recent study reporting the value of using computer-based visualizations to teach science to middle and high school students (Linn et al. 2006). Notably, a meta-analysis of 42 studies of computer-assisted instruction (CAI) in science education, yielding 108 effect sizes, found an overall effect size of 0.27 standard deviations, meaning that “a typical student moved from the 50th percentile to the 62nd percentile in science when CAI was used” (Bayraktar 2001). The meta-analysis noted that simulations and tutorials in science were more effective than drill-and-practice.
The effect sizes reported here are larger than in most prior studies of the impacts of technology in science education. The effect size of 0.27 reported in the 2001 meta-analysis, for example, is usually considered small. In this study, for the four TEEMSS units on which statistically significant differences favored TEEMSS students, two of the effect sizes are considered medium in size and two are large.
Further research and development are called for. It would be good to show learning gains in additional science topics due to technology enhanced instructional units. More development work may be needed to reach that goal. Further research is also needed to see whether instruction in elementary and middle schools enhanced with computers and probeware can be tied to learning gains on standardized tests, such as those that will soon be required in science under provisions of the No Child Left Behind Act of 2001.
Support for the TEEMSS project, including both development of the units and the research reported here, was provided by grant no 9986419 from the National Science Foundation awarded to the Concord Consortium.