Introduction

In recent years, educational software has offered learners increasingly open-ended, exploratory learning opportunities (Jonassen et al., 1999), often presented without any systematic guidance. Where once students marched through automated drilling programs, many now engage in collaborative exploration in complex discovery environments. One interactive medium that has been the focus of recent research is the educational multi-user virtual environment or MUVE. Educational MUVEs have emerged in recent years as a promising platform for science and math learning. Their design draws on a foundation of work on text-based virtual worlds called multi-user dungeons (MUDs) (Fanderclai, 1995) and MOOs (multiple-user dungeons, object-oriented) (Bowers, 1987; Bruckman, 1996; Falsetti, 1995). MOOs, an offshoot of game-based MUDs, evolved into text-based virtual communities and more recently into places for collaborative learning. The term MUVE, or multi-user virtual environment, refers primarily to graphical MOOs (Brdicka, 1999). Educational MUVEs incorporate a virtual world in which learners control characters that represent their online personas. Through these ‘avatars’, learners can explore the world, interact with objects, communicate with other users, and generate and test hypotheses while they explore.

Many researchers have written of the potential benefits of learning in exploratory environments, such as educational MUVEs, but few have conducted empirical studies on what added leverage for learning this interactive medium may provide (Barab et al., this issue; Bruckman, 1994; Horwitz and Christie, 1999; Jonassen, 1999a). Where learning effectiveness has been studied, results from early educational MUVEs show mixed effects (Bruckman, 2000). While some students benefit from learning in educational MUVEs, for other students MUVEs have proven no more effective than paper-based curricula (Dede et al., 2003). One factor that might be contributing to the gap between expectations and results may be a lack of suitable computer-based guidance within the environments. After decades of educational software featuring individualized hints designed to help students work toward specific educational goals (Anderson et al., 1995; Dennen, 2003; Mory, 2003; Shute and Psotka, 1996), embedded guidance has all but disappeared in many exploratory learning environments, including educational MUVEs (Kirschner et al., 2006).

This paper describes an exploratory study in which an embedded guidance system and MUVE-based scientific inquiry curriculum were implemented with a sample of middle school students to investigate the impact of the guidance system on learning and explore the patterns of guidance use among the study participants. In this paper, the MUVE-based guidance system is investigated by addressing the following questions:

Experimental:

  • Do students with access to an individualized guidance system (IGS) containing extensive levels of ongoing guidance within the presentation of a MUVE-based science curriculum have greater learning on a test of science achievement than students who had access to the same system with moderate levels of guidance or with no guidance?

Exploratory:

  • Do students viewing more guidance messages within the presentation of a MUVE-based science curriculum earn higher scores on a test of science achievement than students who view fewer messages?

  • Does learning and use of the individualized guidance system vary by gender?

Theoretical framework

Educational MUVEs

A number of researchers investigating MOOs and MUVEs as learning environments have explored the design, functionality, and potential impact of such environments on student learning and motivation (Bers, 1999; Bers and Cassell, 1998; Corbit and DeVarco, 2000; Slator et al., 2004). For example, the research team behind the Quest Atlantis (QA) MUVE has published a series of studies about the environment describing its benefits on student motivation, learning outcomes, and on ‘making learning fun’ (Barab et al., 2005a, b). Quest Atlantis is an educational MUVE-based game in which students can take part in a large number of quests to save the people of a virtual Atlantis from destruction through environmental, moral, and social decay (Socially-Responsive Design Group, 2004). Elsewhere in this issue, Barab et al. describe a multi-level study investigating the learning benefits associated with a QA curriculum designed to support scientific inquiry practices situated in realistic, socially relevant issues (a ‘socio-scientific’ approach). Students in the study completed a two week curriculum designed to support their development of environmental awareness and real-world science inquiry skills while investigating an interactive narrative in the QA environment. Results of the study show promising findings related to student engagement in the MUVE-based curriculum, sophisticated explanations of curricular processes and outcomes, and statistically significant improvement on classroom and standardized assessments.

In a previous Quest Atlantis study, Tuzan (2004) conducted a design ethnography to identify the motivational elements that supported student participation in Quest Atlantis. In his study, he identified a number of motivational elements in the MUVE-based educational game, including identity presentation, playing, learning, rewards, immersive context, fantasy, uniqueness, creativity, curiosity, control and ownership, context of support, and social relations (Tuzan, 2004). Barab et al., (2005a) reported on the design-based research approach they took in designing and redesigning the Quest Atlantis environment in an ongoing effort of generating, testing, refining, and evolving theories of participation that work to “preserve the joy and meaning” in the processes of learning. Qualitative analysis of three rounds of studies into Quest Atlantis resulted in a better understanding of some weaknesses in the design of the world, including the need for a stronger and more engaging ‘backstory’ to the world.

The River City MUVE Project

The educational MUVE used in this study is the River City curriculum. Dede and his colleagues created River City and investigated its effect on middle school students’ motivation and learning about science (Dede et al., 2003). The River City curriculum is centered on skills of hypothesis formation and experimental design, as well as on content related to national standards and assessments in biology and ecology. The main learning goal for students exploring River City is to discover why residents of the virtual town are getting ill (Nelson et al., 2005).

The River City virtual world is set in the late 1800’s and named for the river that runs through most of the town. River City includes a main street with shops, a library, and elementary school, along with institutions such as a hospital, university, and city hall (Figure 1). Upon entering the city, the students’ avatars can interact with computer-based agents (residents of the city), digital objects (pictures and video clips), and the avatars of other students. In exploring, students also encounter visual stimuli, such as muddy dirt streets, and auditory stimuli, such as the sounds of town residents coughing, which provide tacit clues as to possible causes of illness. Content in the right-hand interface-window shifts based on what the student encounters or activates in the virtual environment, such as a dialogue with an agent or a view of historic photos with accompanying text that provides additional information about the town and its residents.

Fig. 1.
figure 1

The River City application interface.

Full size image

Students work in teams of 2–4 to develop and test hypotheses about why residents are ill. Three different illnesses (water-borne, air-borne, and insect-borne) are integrated with historical, social and geographical content, allowing students to develop and practice the inquiry skills involved in disentangling multi-causal problems embedded within a complex environment (Clarke et al., 2006). Over the course of a three-week long curriculum, students experience a year of virtual time in River City. First visiting River City in October 1878, student teams return several times to find that 2–3 months have passed in River City on each subsequent visit. A final sharing day at the end of the project allows students to compare their research with other teams of students in their class and to piece together some of the many potential hypotheses and causal relationships embedded in the virtual environment.

Results from pilot implementations in three public school classrooms in Boston, MA indicated that the River City MUVE is highly motivating for students, particularly students with lower academic backgrounds (Dede et al., 2003). However, the River City MUVE appeared to help only a subset of students in their learning. While low-achieving students using the MUVE did experience positive results, use of this version of the virtual environment did not lead to statistically significant improvements on science content test scores for the student population as a whole (Dede et al., 2003).

Guidance in Virtual Environments

While no studies have been conducted on the use and potential benefit of embedded guidance specifically designed for educational MUVEs, researchers have investigated the design of agents that inhabit virtual environments and coach students as they complete tasks (Luck and Aylett, 2000; Rickel and Johnson, 1997, 1999). For example, Johnson and his colleagues created the STEVE (Soar Training Expert for Virtual Environments) system. STEVE monitors student actions and offers individualized guidance in virtual environments (Rickel and Johnson, 1997, 1999). STEVE has been used in environments designed to teach students in the US Navy how to operate ship engines (Johnson et al., 2000). The STEVE research centers primarily on the functions, limitations, and potential benefits of the system, without a formal analysis of learning outcomes. Johnson’s group argues that the technology is too early in its development to conduct empirical studies (Johnson et al., 2000).

Guidance Types in Exploratory Learning Environments

The curricula used with exploratory learning environments such as educational MUVEs typically has students learning through collaborative, open-ended exploration of a software environment with limited embedded guidance support (Jonassen, 1991, 1994). From a theoretical perspective, these curricula can be viewed as being built on a combined foundation of socio-constructivist and cognitive-processing concepts. In developing an embedded guidance system for the River City MUVE, several types of guidance found in other exploratory learning environments designed from both theoretical frameworks were examined, including tacit, collaborative, and reflective guidance.

Tacit Guidance

Radical constructivists adhere to what has been labeled “Without Information Given” (WIG) constructivism (Perkins, 1991) or “endogenous” constructivism (Dalgarno, 2001). Constructivist followers of this viewpoint believe that students should construct personal interpretations of the world in a ‘discovery learning’ mode, absent any overt guidance. In this view, guidance implies correct and incorrect answers and singular outcomes, ideas that are anathema to radical constructivism. However, a philosophic position that advocates totally eliminating guidance in software design seems unachievable in practice. All computer-based constructivist environments provide at least a tacit form of guidance. Any software program includes tools and functions designed to be used in specific ways, consequently restricting the range of knowledge construction possibilities. This narrowing of feasible knowledge constructions is a kind of implicit guidance.

The GenScope learning environment provides a good example of a constructivist program with tacit embedded guidance. GenScope includes tools that enable students to manipulate models: the genetic workings of a fictitious breed of dragons. Students can manipulate multi-level, inter-connected models of dragon chromosomes and DNA to develop and test theories of how the rules of dragon (and human) genetics and reproduction work (Hickey et al., 1999; Horwitz and Christie, 2000).

Although no direct guidance messages are included in GenScope, and students are free to interact with it in any way they wish, a subtle form of guidance still exists. The models of dragon genetics are designed to follow simplified rules of ‘real-world’ genetics. By interacting with these models, students can likely construct a personal understanding that meshes closely with the views of the program designers. Students are guided toward a specific understanding of how genetics ‘works’ (Horwitz and Christie, 1999). The models support a limited range of performable interactions, through which students are likely to construct knowledge in similar ways. This form of guidance occurs to some extent in any program.

Although learning environments such as GenScope, based on a purely “WIG” view of constructivism, include only tacit guidance that is designed to support the individual learner as he/she constructs knowledge, exploratory environments designed from socio-constructivist and cognitive apprenticeship frameworks often feature forms of collaborative guidance.

Collaborative Guidance

An important aspect of both socio-constructivist and cognitive processing thinking is a belief in the social nature of learning. In addition to individual knowledge construction, curriculum developed from these theoretical perspectives has learners developing personal interpretations of the world through social interactions with others (Brown et al., 1989; Jonassen, 1999b; Lave and Wenger, 1991; Vygotsky, 1978). In keeping with this idea, some designers promote group-based collaborative knowledge-building in open-ended learning environments (Bielaczyc and Collins, 1999; Nelson, 1999; Scardamalia and Bereiter, 1991, 1994). Collaborative environments allow students to form interpretations of material in partnership with groups of fellow learners.

In such environments, students essentially guide one another. Students can voice their own understandings to the group and can strengthen their individual comprehension through guidance from the group. For example, a student using a collaborative environment for learning English may develop a partial interpretation of how past tense grammatical structure works. The student can turn to his or her peer group and to mentors within the software environment to check that interpretation with those of the others. This kind of group knowledge-construction can help students to check the ‘viability’ of their own understandings within the group. One potential pitfall is that the viability of understandings formed within a small group of students may be weak or inaccurate.

Programs that support collaborative, student-to-student guidance include general-purpose tools such as text-based chat programs, threaded discussion tools, and videoconferencing applications. In addition, designers have created special-purpose collaborative knowledge-construction programs. Primary among these are Scardamalia and Bereiter’s (1991, 1994) Computer Supported Intentional Learning Environments (CSILE). In CSILE and its more recent successor, Knowledge Forum, groups of student contribute data to a communal database of knowledge. Students can link pieces of information together and can comment or add to other students’ database contributions. In addition, CSILE supports individual, reflective guidance through note-taking tools, plan-building scaffolds, and meta-cognitive questions.

Reflective Guidance

Another type of guidance found in both constructivist and cognitive processing-inspired educational environments is “self-directed”, or reflective, guidance (Jonassen et al., 1999). Reflective guidance provides tools to support a student’s hypothesis-generation and testing processes without necessarily offering direct answers or making judgments about particular student actions. This view of reflective guidance echoes those applied to cognitive ‘scaffolding’, which Puntambekar and Hübscher (2005) define as a system of prompts and hints that support learning.

More specifically, reflective guidance messages generally ask students to reflect upon their own learning, describe how they will proceed, and use graphics and/or text to map out their growing understanding (Baylor, 2000; Jonassen et al., 1993).These guidance tools serve to externalize the invisible cognitive processes taking place within each student (Hannafin et al., 1997; Jonassen, 1991), and to scaffold students’ thinking as they develop their own understanding of content present in a given learning environment. With respect to this study, the River City MUVE had previously been designed with only elements of tacit and collaborative guidance included. As described previously, in exploring River City, students encounter visual and auditory stimuli (such as the sounds of town residents coughing) that provide tacit guidance to assist them in completing the project curriculum. In addition, students in the River City environment are encouraged to collaborate with their teams through use of an embedded text-based chat tool and by sharing messages in an electronic team notebook.

The early River City MUVE did not contain reflective guidance messages. The lack of such guidance may have contributed to the uneven results found in the research. Research has shown that lack of such guidance in computer-based exploratory learning environments can hinder learning. For example, Brush and Saye (2000) report that such environments do not provide enough structure to support students as they complete activities, thus increasing frustration. Kirschner et al., (2006) argue that constructivist learning environments with minimal guidance are doomed to fail because they ignore the cognitive processing research into the structure of human memory systems. According to Mayer (2004, cited in Kirschner et al., 2006), in a series of empirical studies dating back to the 1950s, unguided learning has repeatedly been shown to be inferior to guided learning. In one recent example, Horwitz and Christie (1999) found that use of the GenScope program, including only inherent guidance, did not offer any additional benefit for learning over non-computer based instruction. In diagnosing the disappointing results for learning seen with the program, Horwitz suggested that the lack of embedded reflective guidance in GenScope made it difficult for students to identify where and when to focus their attention (Horwitz and Christie, 1999).

Like other open-ended exploratory environments, MUVEs without embedded guidance may hinder student learning. Hence, for this study a guidance system designed to offer reflective guidance messages was added to the River City MUVE. As described previously, reflective guidance in exploratory learning environments differs from direct instruction in that it does not provide overt answers or make judgments about particular student actions. Reflective guidance messages instead focus on assisting with student meta-cognition, asking students to reflect upon their learning, describe how they will proceed, and use graphics and/or text to map out their growing understanding (Baylor, 2000; Jonassen et al., 1993).

Gender and Educational MUVEs

The question of ‘what girls want’ from computer software is a vital one that underlies current research into gender and educational learning environments. In its report on educating girls in the new computer age, The American Association of University Women (AAUW) educational foundation lists a number of design suggestions for computer games for girls (American Association of University Women, 2000):

  • Rich narrative, intricate games

  • Customizable, personalizable female characters

  • Opportunity for collaboration and communication

  • Social interaction on-screen and between players

  • Opportunity for positive social action

  • Appropriate level of difficulty

  • Opportunities to design or create

  • Strategy and skill requirements

In addition, the report finds that girls prefer games that closely simulate real life and games that allow for role-play (American Association of University Women, 2000; Subrahmanyam and Greenfield, 1998).

Educational MOOs and MUVEs support many of the features suggested by the AAUW report as useful for ‘girl-friendly’ software. Both have tools for social interaction, communication, and collaboration. In addition, graphical MUVEs are typically designed to simulate real-world events and contexts and center on role-play and on-screen interaction in a virtual narrative. Consequently, it is not surprising to find that recent educational implementations of MOOs and MUVEs have found greater equality in learning outcomes and participation rates among girls than in more traditional learning environments. One such study on gender and programming achievement in a text-based MOO found that girls spend significantly more time than boys communicating with others in the environment (Bruckman et al., 2002). In addition, gender was found to play no role in learning outcomes related to the programming tasks at the center of the MOO-based curriculum.

Barab’s “Socially-Responsive Design Group” (2004) conducted an extensive analysis of gender participation in their Quest Atlantis (QA) MUVE. The analysis showed no differences in terms of overall participation rates in the MUVE between boys and girls. However, looking specifically at participation as reflected by online communication, it was found that girls used chat more than boys (p < 0.01) and sent more e-mail messages (p < 0.01) than boys. In terms of learning and achievement, the QA MUVE was equally effective for boys and girls. Girls who participated in a three-quest learning unit about plant and animal cells saw gains on pre- to post-test scores that were statistically equivalent to those of the boys. Finally, Barab’s group found that girls wrote more in their online notebooks when completing quests and engaged in longer metacognitive reflections about their work in the MUVE.

Dede’s initial River City project also found that girls performed as well as boys through the River City curriculum. Pilot studies of the MUVE in 2001 found no significant differences between boys and girls in learning outcomes, motivation, or self-efficacy toward science learning (Dede et al., 2004). A large-scale implementation of the River City MUVE in fall 2004 reinforced the early study, again finding no significant differences in learning outcomes between girls and boys.

Research into educational MUVEs is still in its infancy, and the data on the gender ‘friendliness’ of MUVEs, while promising, is sparse. In addition, no research has been conducted on patterns of use and potential effectiveness of MUVE-based embedded guidance by gender. Consequently, while the primary focus of this study was a general exploration of the role of individualized, reflective guidance in an educational MUVE, the role of gender in MUVE-based learning was also explored, centering its possible interaction with the use of an embedded guidance system.

Muve environment and guidance system

Software Environment

The River City MUVE was built in the Active Worlds environment. Active Worlds (www.activeworlds.com) provides a robust, relatively simple authoring system for constructing 3-D graphical MUVEs. Using Active Worlds, designers can produce MUVEs incorporating multiple objects, sounds, animations, images, and ‘agents’ (automated 3-D characters that inhabit the MUVE). Large numbers of users can simultaneously occupy and explore a given MUVE. Users can walk, run, and fly through an Active Worlds MUVE, entering buildings, climbing mountains, and swimming through virtual bodies of water. Individual users can chat with other users, or broadcast messages to all nearby ‘citizens’ and ‘tourists’. In addition, users can click on specially designated objects containing hyperlinks. Clicking on these objects will trigger the appearance of web pages, images, or web-based applications in a web browser window embedded in the software. This browser sits to the side of the main 3-D MUVE window (Figure 1).

As part of the River City project, Active Worlds Corporation built a customized ‘plug-in server’ that provides additional functionality to the MUVE environment. The most relevant to this research is the data-tracking system. With the data-tracking system, it is possible to collect, store, and retrieve information on the activities of each student as s/he explored the MUVE. These data form the basis of a personal MUVE history of each student that followed them from session to session and were used by the individualized guidance system to select guidance messages available to students.

Individualized Guidance System

To investigate the effect of reflective guidance on student learning, an individualized guidance system (IGS) was added to the River City MUVE. The system utilized data collected on each student’s activities to offer real-time, reflective prompts about the students’ own learning in the world, with the content of the messages based on in-world events and basic event histories of each student. To understand the methods used by the guidance system, it is important to first understand the features through which students could find general clues about illnesses while exploring the MUVE.

The River City virtual world is designed as a late 19th century town, divided into several geographic zones (Figure 2):

  • Wealthy homes and River Overlook:

  • River City University

  • Train station and hospital

  • Middle class homes

  • Downtown: shops, library, school, city hall, library

  • Tenement district and dump

Fig. 2.
figure 2

The River City layout

Full size image

Within each zone, students could interact with historical photos, books and charts, and interactive tools (water sampling stations, fecal testers, bug counters, Figure 3) to gain information about the illnesses in River City.

Fig. 3.
figure 3

Interactive water sampling station.

Full size image

In addition, students could use the chat system in the MUVE (Figure 1) to ask a limited set of questions of the River City citizens. There were 32 computer-controlled citizens (software agents) scattered throughout town. These citizens could offer bits of information about happenings in town and the illnesses. For example, the agents could respond with a short answer to the question, “What’s new?” The information offered students by the in-world objects and citizens changed from session to session, but was not individualized in any way. Each student saw the same information from a particular object or citizen.

To create the individualized guidance system, all the items with which students could interact were programmatically tagged with identification codes. Every time a student clicked on an object or ‘spoke’ to a River City citizen, a record of the event was stored in a server-side database. The cumulative record of events resulted in a personalized history for each student. A guidance model, operated by an invisible software agent, was triggered after each student interaction event in the MUVE. A subset of events was associated with guidance scripts. The guidance model used these scripts to offer a specific selection of messages to each student. The scripts contained a set of rules for selecting guidance, based on a student’s history of interactions with objects or River City residents.

The following brief scenarios offer examples of the kind of guidance messages a student might receive, along with an explanation of how the guidance model individualizes guidance messages.

Guidance Example 1: Akiko enters River City for the first time. Through her avatar, she wanders around the town for several minutes to get her bearings. The first building she enters is the hospital. When she clicks on the admissions chart in the hospital, the Individualized Guidance System appears in the upper right-hand corner of the MUVE web pane with the title “Admissions Chart Guidance.” Below this headline are three buttons, “Hint 1”, “Hint 2” and “Hint 3”. Clicking on the first hint button, a message appears in the guidance system window stating, “What kinds of symptoms do you see in the chart?” Clicking on the second hint button reveals the message “Where do most of the sick people live?” Clicking on the third button reveals the message “Is there anything about this hospital that is different than the ones you have seen?”

Guidance model methods: When Akiko clicks on the hospital admissions chart, she triggers a guidance event. The guidance model uses an identification tag associated with the event to add the fact that it was triggered to Akiko’s personal history, and to look for any guidance scripts associated with the event. Finding that there is a script associated with the event, the model runs it. Because Akiko has not triggered this event before, and there are no relevant events recorded in her personal history, three default guidance message links are displayed.

Guidance Example 2: The next day, Akiko reenters River City and explores the tenements, asking questions of a couple of residents. She then returns to the hospital. This time, when she clicks on the hospital admissions chart, a new set of messages is available. Clicking on hint button 1 reveals a message stating, “Welcome back, Akiko. I noticed that you have talked to some of sick tenement residents. How many people are sick from that area?” Clicking on button 2 reveals the message, “Last time you were in the hospital, you talked with the doctor. What does she have to say this time?” Clicking on the third link reveals the message “Have the symptoms of the patients changed since last time?”

Guidance model methods: The model agent records all tenement interactions and events to Akiko’s personal history. It also checks for guidance scripts associated with the interactions, and shows related message links. When Akiko returns to the hospital and clicks on the admissions chart, the agent records the event and runs the guidance script. The guidance script contains rules for three individualized messages. Akiko’s previous visit to the tenement and her past interaction with the doctor causes two of individualized messages to appear. The third individualized message appears because the season in the virtual town has changed from fall to winter.

Method

In the initial stage of investigation, a randomized experiment was conducted in an effort to establish a link between access to individualized, reflective guidance in an educational MUVE and learning outcomes. Consequently, students were randomly assigned, within chosen classrooms, to participate in one of the three guidance access groups. Subsequent to this experimental study, post-hoc exploratory analyses were conducted to explore patterns of guidance use, including: (a) how students who viewed greater numbers of guidance messages performed on content tests compared to those who viewed less, and (b) whether boys and girls accessed the guidance to the same degree, and whether there was an interaction between gender and guidance use on content scores.

Participants

287 public middle school students (147 girls and 140 boys) in a mid-Atlantic state took part in the River City MUVE curriculum. At the participating school, overall SES is relatively high (3% of students receive free or reduced lunch), the student body is predominantly Caucasian (80%), and less than one percent of the students are non-native English speakers. Further, in 2004, 69% of 8th graders at the school were at, or above, grade level standards in English language arts, and 73% were at, or above, grade level standards in math (www.greatschools.net/cgi-bin/ny/other/1390, accessed Feb. 03, 2005). Statistical power analysis suggests that a sample of 287 students provides sufficient power to detect medium sized effects at the .05 alpha level (Light, Singer & Willett, 1990). Among the students assigned to the treatments under investigation in this study, 272 completed the curriculum and were included in the data analysis.

Procedures

Three computer-based treatments were designed for this study. These include No Guidance (NG), Extensive Guidance (EG), and Moderate Guidance (MG). Within each of the participating classes, students were assigned at random to use a given variant of the curriculum. The No Guidance variant was based on the baseline curriculum used in earlier River City implementations. Students in the No Guidance control group had no access to the Individualized Guidance System. Students in the Extensive Guidance treatment group used the same control curriculum, but were provided access to a guidance system featuring continuously updated guidance links and associated messages. Students with access to the Extensive Guidance version of the IGS could view three guidance messages per pre-defined information ‘hot-spot’ object in the River City virtual world. Whenever students clicked on a specially tagged object, the Guidance System would flash alternating colors to indicate that new hints were available. The Moderate Guidance treatment was identical to the Extensive guidance treatment, except that students had access to only one guidance message per ‘hot-spot’ object. In both guidance variants, the IGS did not automatically show specific guidance messages, but instead displayed ‘hint’ buttons linked to messages (Figure 4). To view guidance messages, students needed to click on these hint buttons. In this way, the number of messages students actually viewed guidance messages could be recorded, along with information on which messages they saw.

Fig. 4.
figure 4

The individualized guidance system interface.

Full size image

Measures

Both qualitative and quantitative data were collected from students and teachers over the three-week implementation period. Pre- and post-intervention, the students completed an affective measure that was adapted from three different surveys: Self-Efficacy in Technology and Science (Ketelhut, 2004), Patterns for Adaptive Learning Survey (Midgley et al., 2000), and the Test of Science Related Attitudes (Fraser, 1981). To assess understanding and content knowledge (science inquiry skills, science process skills, biology), a 30 question content test was administered pre- and post-intervention with an internal consistency reliability of .80 in a middle school population. The multiple-choice and short-answer content questions centered on scientific inquiry processes, hypothesis formation, and disease transmission.

Data Analysis

A combination of experimental and exploratory quantitative analyses were conducted in this study. Multilevel regression analysis was used to answer the first research question. Multilevel modeling is required in the experimental component of this study in order to accommodate for the clustering of students within classrooms in the research design, as well as in an attempt to make causal conclusions. In this analysis, student test score gains were regressed on access to an individualized guidance system within the River City MUVE. To answer research question 2, a post-hoc regression model was fitted to include the number of views of guidance completed by each student in the moderate and extensive guidance access treatments.

In addition, exploratory models were fitted to examine the role of increased guidance use for students in the moderate and extensive guidance groups who chose to ‘take-up’ the guidance system by viewing at least one hint. Finally, to investigate patterns of guidance use among boys and girls, a post-hoc exploratory regression model was fitted regressing gender and guidance use on score gains. Although this was predominantly an exploratory study using quantitative analysis techniques, anecdotal data from student interviews and classroom observations were also collected. These data were gathered in an effort to “add meat to the bones” of any quantitative findings.

Results

Guidance System Exposure

In model M2 of Table I, the predicted content test score gains of control (No Guidance) students are about 0.14 score points (intercept), while students exposed to Moderate Guidance (MG) show predicted gains of 0.45 (=0.14 + 0.31), and students exposed to Extensive Guidance (EG) are almost identical to the control treatment students, with a gain of 0.13 (0.14–0.01). A general linear hypothesis test confirms that the joint impact of the treatment predictors on the outcome is NOT statistically significant (p > 0.05). When comparing the impact of the control, Moderate and Extensive Guidance treatments on student learning of scientific inquiry methods and disease transmission, within the limits imposed by statistical inference, all treatments had an identical impact on student learning.

Table I. Taxonomy of Fitted Multilevel Models Describing the Relationship for Students Exposed to an Individualized Guidance System Between Science Content Pre- to Post-test Score Gains and Placement in Guidance Access Treatments and the Number of Times They Viewed Guidance Messages in an Educational MUVE, Controlling for Gender (n students = 272)
Full size table

Consequently, the answer to the question

Do students exposed to an individualized guidance system containing extensive levels of ongoing guidance within the presentation of a MUVE-based science curriculum have greater learning on a test of science achievement than students who are exposed to the same system with moderate levels of guidance or with no guidance?

is ‘no’. What the data reveal is that, on average, students with access to high levels of guidance (EG) did no better than the students with access to moderate levels of guidance (MG) and no better than students with no access to guidance (NG).

Level of Guidance System Use and Student Learning

Descriptive analyses suggested that students in the two treatments with exposure to the guidance system did not tap the guidance at a significant level, on average. Students with access to the Extensive Guidance System viewed an average of 15 messages over the course of the implementation. Similarly, students in the Moderate Guidance access group viewed an average of 12 guidance messages total. Although unlikely, it is possible for students in the Moderate Guidance students to view more than 200 messages over the course of the implementation, and for the Extensive Guidance students to view more than 600 total messages. To do so, however, students would need to visit all possible guidance ‘hotspots’ in each of the 6 River City virtual worlds, and choose to view every message presented. Still, the average number of views of guidance messages in the two guidance access groups is quite low.

Moreover, analyses revealed that the Moderate and Extensive Guidance groups consisted of two sub-groups: those students who used the guidance system and those who did not. Approximately a quarter of the students with access to the guidance system did not view any messages. To explore the potential relationship between levels of guidance system use and learning outcomes, this researcher thought it valuable to distinguish students with access to guidance who ‘took up’ the guidance at least once from those who never used the system. After all, students who did not look at any guidance messages experienced a curriculum identical to students in the control treatment.

Thus, to investigate the second research question, Do students viewing more guidance messages within the presentation of a MUVE-based science curriculum earn higher scores on a test of science achievement than students who view fewer messages?”, a post hoc regression model was fitted to explore the effect on learning outcomes of student utilization of the guidance system for guidance ‘takers’ in the extensive and moderate guidance access groups (Model M3, Table I).

In this exploratory analysis, for students in the Extensive Guidance treatment who took the initial step of choosing to use the guidance system, a statistically significant positive, nonlinear, relationship (p < 0.05) was found between level of guidance system utilization and science content score gain. Figure 5 displays this relationship. At the same time, however, the benefit of viewing additional guidance messages by users of the guidance system in the Extensive Guidance group was not seen for students in the Moderate Guidance group who utilize the guidance system (p > 0.05).

Fig. 5.
figure 5

The fitted relationship between content test score gains and levels of guidance system use by students with access to extensive or moderate levels of guidance who chose to “take up” the guidance at least one time, in a MUVE-based curriculum (n = 272).

Full size image

It is important to emphasize that the investigation into levels of guidance use and its possible association with content scores was not conducted under experimental conditions. Students who chose to use the guidance system self-selected into its use, and may have therefore shared some unobserved (and unanalyzed) commonalities aside from use of the guidance system. Consequently, this post-hoc analysis should be viewed as exploratory in nature. That said, a number of other variables were investigated as possible predictors of content test score gains, including socio-economic level, prior grades in science, experience with computer games, familiarity with computers, and gender. Among these, only gender appeared to have any interaction with level of guidance use on content test scores.

Guidance Viewing and Score Gains by Gender

Basic descriptive statistics revealed that boys and girls used the guidance system differently. Boys with access to the guidance system viewed fewer guidance messages, on average (10.79), than girls (16.26). Also, a smaller percentage of boys (76%) than girls (84%) chose to use the guidance system at least once. Of the students who opted into the system, boys still viewed fewer messages on average than girls (14.29 vs.19.25).

To investigate the question “Does learning and use of the individualized guidance system vary by gender?”, a post hoc model, Model M4, (Table I) was fit describing the effects of guidance system use on content score gains for boys and girls who chose to use the guidance system. This model predicts a weak (p < 0.10) negative relationship between gender and content test score gains overall, such that the predicted mean gains of girls (controlling for other predictors) are about 0.54, while boys show a negative GAIN of −0.27 (0.54–0.81). Consequently, in this study boys performed more poorly than girls overall, regardless of exposure to guidance.

At the same time, exploratory analyses revealed that increased viewing of messages as a ‘taker’ of the guidance system in the Extensive Guidance treatment group showed a positive association (p < 0.05) with content score gains for both boys and girls. This fitted relationship is plotted for boys and girls in Figure 6. Students in the Extensive Guidance group who viewed more guidance earned higher scores than those who viewed fewer messages, and the benefit of guidance system use varied by gender, with boys doing worse, on average, at each level of guidance viewing. Model M4 predicts essentially zero (−0.04) effect of Moderate Guidance group membership on score gains, and a very small (0.19) positive effect of utilization of the guidance system on GAIN for these students. While girls outperformed boys at each level of guidance exposure, girls and boys who ‘took up’ the guidance system benefited from increased viewing of messages. Exploratory models fitted with the two-way interaction between gender and guidance viewing included (not shown) revealed that the disadvantage faced by boys was consistent across all levels of guidance system use. In other words, in this study boys had steadily lower average gains on content test scores than girls at each level of guidance system use.

Fig. 6.
figure 6

The fitted relationship between content test score gains and levels of guidance system use by students with access to extensive or moderate levels of guidance who chose to “take up” the guidance at least one time in a MUVE-based curriculum, by gender (n = 272).

Full size image

Again, this is a non-experimental analysis, with students self-selecting into guidance use. Consequently, causal claims cannot be made. However, among the predictors investigated (SES, prior grade in science, experience with computer gaming, familiarity with computers), only guidance viewing level and gender were found to be associated with content test score gains. Clearly, this suggests that the difference among boys and girls with respect to guidance viewing and learning is a fruitful area for future research.

Summary

In this study, it was shown that simple access to guidance in an educational MUVE did not have any measurable effect on learning outcomes; Students with access to extensive levels of guidance saw no greater gains on content test scores than those with access to moderate levels and were indistinguishable from students with no access to guidance at all. In spite of this lack of causal findings, exploratory investigations into patterns of use of the guidance system and the nature of its use as a predictor of content score gains did reveal tentative findings that merit further investigation. Post hoc analyses of the relationship between levels of guidance use and content test score gains showed a significant positive relationship between increased viewing of guidance and test scores. Although students self-selecting into use of the guidance system may have shared unanalyzed common traits that contributed to higher test scores, in this study only guidance viewing levels and gender were found to have any impact on scores.

Analysis of content test score gains by gender revealed that boys did worse than girls in this study. The gender gap did not disappear for boys who viewed more guidance. Although increased viewing of guidance messages among boys was associated with higher score gains, at each level of guidance viewing their average score gains were still lower than that of girls at the same level. These findings, though exploratory, echo those of earlier studies that found educational MUVEs to be relatively ‘girl friendly’ learning environments (Bruckman et al., 2002; Dede et al., 2004; Socially-Responsive Design Group, 2004).

A surprising finding in this research was the low number of overall guidance views by participating students. Roughly a quarter of the students with access to the guidance system never looked at a single message. Further, a large proportion of the students with access to the system viewed only a fraction of the available messages. Why might this be the case? Anecdotal information gathered about student use of the guidance system in interviews and classroom observations revealed that, for many students, the guidance system design did not lend itself to frequent use. With much to do and see in River City and relatively little time to do it, many students focused their attention on the 3-D environment and chat system almost exclusively. Several interviewed students reported not understanding the purpose of the guidance system, or that they simply ignored it. If, as the exploratory findings reported here suggest, increased use of an embedded guidance system may contribute to better learning outcomes in a MUVE-based curriculum, then it is important to explore ways to encourage students to use the guidance.

Conclusions & Implications

The contrasting findings from this study offer useful implications for future research into the design of embedded guidance in educational MUVEs. On one hand, providing access to a guidance system had no effect on how well students performed on science content tests. On the other hand, exploratory analyses suggested that increased use of the guidance system among students with access to ‘extensive’ guidance helped those students outperform those who used the guidance system less. It is impossible to generalize this finding or treat it as causal, since students self-selected into use of the guidance system. However, the findings do suggest that future studies would do well to conduct a systematic investigation of the factors that influence a student’s decision to use embedded guidance initially, as well as the factors that contribute to their continued use of guidance as they explore the MUVE.

From a technical standpoint, MUVEs support the ability to keep minutely detailed records of the moment-by-moment movements and actions of each student in the environment. To explore questions of initial guidance take-up and continued use, the large database of information on in-world interactions could be mined more fully. A longitudinal study could be conducted in which the interplay between guidance viewing and subsequent actions in the MUVE is analyzed. Through these data, researchers could discover when students first choose to use guidance, when and why they stop using it, and how use of the guidance affects subsequent actions and continued take-up. Combining these data with more basic student-level predictors (gender, SES, age, ethnicity, etc.) and with a more robust program of qualitative analyses, would allow for a more complete picture how student characteristics interact with guidance take-up and continued use.

Exploratory analyses in this research also suggested that in the River City MUVE and curriculum, boys were at a disadvantage overall and specifically in terms of guidance system use. Based on these findings, future studies could investigate how the benefits seen with guidance use can be extended more fully to boys, while continuing to be enjoyed by girls. Because the content of guidance messages can be individualized to such a great extent, it would also be interesting to explore the effect of gender-specific guidance messages.

Finally, quantitative findings showing a low level of average guidance system use, combined with anecdotal interview and observational information on possible reasons for this low level of use, reveal that the design and placement of the guidance system did not promote its usage by all students. These ‘sideline’ findings suggest that future designers may benefit from more closely integrating the guidance system into the 3-D environment. Creating a stronger visual or narrative link between the guidance system and the 3-D world could do this. For example, the guidance system could be introduced as part of a package of tools in each student’s inventory used to make sense of the world. Alternatively, the guidance system could be embedded directly into the 3-D world itself. Instead of a world-external system that flashes to announce the availability of new hints, the hints could appear above the related object inside the world. Another possible design change would have the hints could coming via an in-world pedagogical agent, similar to the STEVE agent described earlier (Johnson et al., 1998, 2000). The agent could take of the form of an in-world companion, accompanying each student as he or she explores the environment.

Research into the theory, design, and practice of individualized guidance systems for use with educational multi-user virtual environments is still in its infancy. The findings from this exploratory study provide an important first step from which to embark on future studies. A new NSF-funded study offering an updated version of the individualized guidance system and the River City MUVE is being implemented with approximately 30,000 students across the U.S. and internationally between 2006 and 2008, providing opportunities for a more rigorous experimental design, a more representative sample, opportunities for more sophisticated quantitative analyses and greater attention to qualitative analyses, and greater ability to generalize findings. By evolving embedded guidance system design based on findings from this initial exploration, and by conducting additional studies, researchers can improve upon the current MUVE-based system to provide added leverage for learning for all students.