Keywords

1 Background

There is little doubt in academia or among the public at large about the usefulness of computer technology as a tool for learning [7]. Across many disciplines, but notably in the geosciences, computer technology as a tool for access to data and Web-based resources, and computational problem solving, is the lifeblood of the classroom. Today, students in higher education have access to real-time and legacy datasets, sophisticated visualization applications, high-bandwidth networks, and high-speed computers. The so-called “Millennials” or the Net Generation (NetGen’ers) have grown up with computers and are technologically savvy in a way prior generations could not be [15]. They are accustomed to operating in a digital environment, communicating with cell phones, text messaging, and myriad mobile and home devices. NetGen’ers enjoy being part of communities using multiple social media applications. “Millennials are putting [video games] at the center of their entertainment preferences, but it is a new kind of gaming that is more social, interactive and engaging [17, p. 9].” They are experiential learners: they prefer to learn by doing as opposed to learning by listening. By contrast, and despite huge investments in communication and computer hardware made by universities and schools, most formal teaching and learning still use methods that would be familiar to a 19th century student: reading texts, listening to lectures, and participating in highly scripted laboratory exercises [9].

There is no shortage of ideas on what can be done to better understand how students learn in the digital age [1, 3, 15]. Converging evidence from educational psychology suggests that computers can play an important role in developing critical and creative thinking skills in students, such as scientific inquiry. Recent advances in learning and cognitive science research recommend individualized instruction, subject-matter experts, and rich curricular activities for improving education [17]. Unfortunately, many recommendations have not been widely adopted because they prove too expensive or are difficult to integrate into traditional teaching approaches that too often still ignore findings of learning research [10].

Applied prudently and intelligently, technology holds great promise as a means to improve education and can be implemented without unrealistic increases in spending. Prensky [16] has framed the significance of computer technology in terms of the fundamental characteristics of effective learning: active engagement; participation in groups; frequent interaction and feedback; and connections to real-world contexts [2]. Simulations can improve learning by encouraging students to “learn by doing.” Advocates of electronic games suggest that gaming could increase student enthusiasm for educational materials, which could in turn increase time on task and lead ultimately to improved motivation and student performance [17].

Educators have already begun introducing games into instruction (e.g. “Discover Babylon©, Civilization II™, SimCity™, and Immune Attack™”), and will continue to benefit from commercial inroads into gaming in education so long as such applications are based on a sound understanding of which features of these systems are important for learning and why [18]. However, even if we fully understand how best to use simulated environments, the challenge of actually building technically accurate and visually compelling simulations is enormous [9].

The term “gamification” has recently been coined to describe both the use of games as tools (e.g. simulations, team-building exercises) for business, education, etc. and for the use of game techniques within a system [2, 8]. Deterding [4] suggests that gamification is “the use of game design elements in non-game contexts.” He further describes one level of gamification as using game interface design patterns to incorporate common, successful interaction design components into non-game applications. It is in this context in which the GEOPOD project has been developed. While the interface elements are derived from the realm of computer gaming, game elements such as scores, levels, leaderboards, etc. have not been included. Still, the motivational factors of providing such a game-like interface are well documented [4].

2 Project Description

The GEOPOD project creates an interactive interface (GEOpod) that can probe a 3-D immersive world of authentic geophysical data using a roadmap of rich curricular materials to motivate learners to explore, query, discover, and report on geoscience concepts, processes, and phenomena. The GEOpod has the ability to navigate the data volume defined by actual geospatial coordinates and map referencing; collect and store real data by means of virtual sensors; and actuate devices for measurement and sample collection. The interface simulates a navigable pod, or 3-D vessel, with six degrees of freedom. The experience is designed for instruction that will immerse students in a 3-D exploration environment where they can explore atmospheric features such as jet streams and frontal boundaries; deploy devices to retrieve vertical atmospheric profiles; follow isosurfaces; discover relationships and connections within and between phenomena; and collect and record data for analysis.

While GEOpod is not a game, it features a game-like interactive, virtual environment with a first-person perspective, one with which many students are familiar. Such environments have the potential to enrich instruction by creating for the student an immersive environment of pictorial dynamism and sophistication that is fun, interactive, and the next-best thing to reality. Learning through performance requires active discovery, analysis, interpretation, problem-solving, memory, and physical activity [5]. “Video games are complex systems composed of rules that interact. Gamers must think like a designer and form hypotheses about how the rules interact so they can accomplish goals and even bring about emergent results. Thinking like a designer in order to understand systems is a core 21st-century skill [17, p. 12].”

Fig. 1.
figure 1

Initial GEOpod interface

Full size image

The project had three main phases: (1) initial development of the data visualization and GEOpod simulator software, (2) a usability evaluation, and (3) an educational evaluation. This paper reports primarily on the second phase, describing the interface and the usability study that was conducted to evaluate and improve it.

3 Initial Interface

Figure 1 shows the GEOpod display that was initially developed. It is meant to represent the view of the pilot out the front windshield of the pod, the various instruments providing data readings of the atmosphere, and most of the controls or operations available to the pilot. This is in addition to the controls (particularly navigational) that are provided to the user through keyboard input, which mimic those of other video flight simulators or provide additional operations.

The 6 areas of the display are configured as follows:

  1. 1.

    Data Volume – contains the data used to create the isosurface (a specific volume and set of conditions of the atmosphere that is currently being explored). The pod pilot can sample parameters from the data volume as they move within it. The colored surfaces within this volume represent visualizations of atmospheric data.

  2. 2.

    Help – displays a help page.

  3. 3.

    Geocode Lookup – allows the user to lookup or navigate to a specific geographic location within the data volume.

  4. 4.

    Navigation Panel – shows the current position of the pod in latitude, longitude, and altitude.

  5. 5.

    Parameter Display – displays up to 9 different atmospheric parameters on a 3 × 3 grid. Some of the default parameters are temperature (T), geopotential height (Z), wind speed (WS), relative humidity (RH), and dewpoint (Td).

  6. 6.

    Primary Tools – includes buttons to provide access to a calculator, notepad, measurement devices (dropsonde instrument package and particle imager), and a navigational tool for setting up grid points within the volume.

The controls of this display are meant to be accessed using a typical pointing device such as a mouse.

3.1 Navigating the GEOpod

The GEOpod is a vehicle which allows the pilot to be immersed in the 3-D data volume. When using the GEOpod, the pilot can think of themself being at a certain location inside the 3-D world that cam see on the screen (as in Fig. 1). Using any one of various navigation controls, the pilot can move the GEOpod inside this world. When the pod moves, the pilot sees the image on the screen change, because they, in the GEOpod, are moving (much as buildings appear to fly past when driving a car). The navigation controls allow the pilot to move the GEOpod within the world, not alter the world or the data volume itself. All of the controls are relative to the GEOpod’s current position and orientation within this 3-D world (i.e., relative to the pod’s local coordinate system). For example, moving “up” will increase the pod’s altitude if the pod is level with the earth’s surface, or move the pod across the surface if it were above the earth looking straight down.

A keyboard and mouse interface was used instead of using an interface device such as a joystick or game controller for several reasons:

  1. 1.

    Potential users may already be familiar with this style of navigation through prior experience with computer games,

  2. 2.

    The additional physical dexterity such devices require,

  3. 3.

    The expense of supplying the devices for every system in a lab, and

  4. 4.

    Requiring adopters of the system to buy additional hardware would decrease the dissemination potential of the system.

3.2 Keyboard Navigation Functions

The four main navigation keys are w, s, a, and d. The w key moves the GEOpod forward, in whatever direction it is facing. The s key moves the GEOpod backward. The a key moves the GEOpod to the left. The d key moves the GEOpod to the right. The f and c keys move the GEOpod “up” and “down”, respectively, where these directions depend on the pods current attitude (orientation).

For a complete description of all the GEOpod controls, see [6].

4 The Usability Study

This following describes the test plan used to conduct a usability test during the development of the GEOpod software system [11]. The goals of usability testing included establishing a baseline of user performance, establishing and validating user performance measures, and identifying potential design concerns to be addressed in order to improve the efficiency, productivity, and end-user satisfaction.

It is worth emphasizing that this study is not intended as a validation test of the correct behavior of the GEOpod controls, nor as a test of the user’s abilities per se, but is an evaluation of the effectiveness of the GEOpod interface from the user’s perspective. Usability goals include that the user will be able, after minimal training, to perform a series of guided activities within a “reasonable” amount of time, with a minimal number of errors, instances of “dead ends,” or resorting to the help menu. The reasonableness metric will be judged against the need for students to be able to accomplish a certain amount of work using the GEOpod system during a typical class lab period. A “control” study was done by having an expert user perform the timed trials as well, to provide a lower bound on the times to complete the tasks.

4.1 Methodology

Two types of usability testing were employed for this project. The first was an expert heuristic evaluation, looking for potential problems based on basic human-computer interaction (HCI) principles. The results from the expert walkthrough were used to inform some elements of the usability study, as well as to provide general recommendations for improvements to the system.

The second type of testing was a standard usability study using a small group of test participants selected from the population of potential users of the system.

The Usability Study.

There were 14 primary test participants, all of whom were asked to complete the same set of four test trials – 3 short trials and one trial that approximated how the GEOpod system would ultimately be used in a class lab assignment. The order of the test trials was the same for each participant, as effects of learning bias were not a concern in this study; indeed, it was assumed that participants would learn something from each trial that could prove useful in subsequent trials. Each test participant was asked for basic demographic information including age, handedness, gender, and experience with previous 3-D navigational systems. They were also given a satisfaction assessment as a post-test to gauge their level of satisfaction with the interface.

The participants were all earth sciences students from the sophomore, junior, and senior levels, ensuring that they had at sufficient knowledge of meteorology principles and terminology; their knowledge of meteorology was not being tested, only their ability to use the GEOpod system effectively to complete the tasks. There were an equal number of male and female test participants. Participants were provided with a brief training on the use of the GEOpod controls prior to the first trial, as well as being given a short interval for experimenting with the controls in an unstructured manner.

The test system was housed in a cubical enclosed on three sides in a distraction-free research lab. One of the walls of this cubical was a 5′ high partition. The test subject was positioned at a desk within the cubical. The test facilitator sat outside the cubical on the other side of the partition, unseen by the test subject. The facilitator had a monitor that was a mirror of the test participant’s monitor, as well as an active keyboard and mouse with which to interact with the system during trial setup and end-of-test housekeeping tasks. This allowed the facilitator to observe and record the action of the test without in any way being in direct contact with the test subject, either verbally or visually.

Recording was done with both a video camera trained on the mirrored monitor and through screen capture software. The GEOpod system is also equipped with a logging mechanism that records time-stamped (accurate to at least the 1/10th second) events detected by the system, including all keystrokes, button presses, etc.

Timings and error counting were done after the trials through protocol analysis of the captured video, ensuring that such measurements were consistently taken.

Results.

The main usability study was conducted to primarily get a feel for the “reasonableness” of the design, i.e. were students able to successfully use the system to accomplish typical educational tasks. Inversely, the study would also point out potential problem areas with the interface, e.g. participants had difficulty completing a task.

Table 1 summarizes the timings of the 14 participants over 4 trials. In addition, the table includes the times posted by an expert user (control). A value of double the control time was used as a reference target time for the participant performance [12].

Table 1. Timed trials results
Full size table

What we see from this data is that, in Trials 1 and 4, test participants met the goal of performing the tasks in no more than double the control times. This meets the stated performance targets.

In Trial 3, participants completed the tasks in 2.5 times the control time. This is slightly worse than the target. However, there was only one time that was significantly worse than all others (nearly double the next longest time). If this time were eliminated from the average, the result is only slightly worse (4:11) than the target time.

In Trial 2, participants performed at 3 times the control time. There were 4 participants who took nearly twice as long as all other participants. Eliminating these extra lengthy participants from the average produces a result of 5:07, just slightly worse than the target time.

The conclusion for this data is that the time required for the users to complete each trial is within acceptable limits of performance.

Table 2 summarizes the participants’ responses to the post-test survey which asked attitudinal questions about their experiences with the GEOpod system using a Likert scale. The full questionnaire can be found in [12].

Table 2. Questionnaire responses
Full size table

Questions 1–7 should result in agreement (at least a 4) from the user (with 5 being “Strongly Agree”) if the user had a positive attitude toward the GEOpod system. Four of the seven questions resulted in average responses better than 4 (very favorable). The remaining 3 indicators showed slightly less favorable results, but still on the positive side of the scale.

Questions 8–10 should result in disagreement (a 2 or less) from the user (with 1 being “Strongly Disagree”) if the user had a positive attitude toward the GEOpod system. Only one of the three indicators was clearly within the target response area (Question 9), but the other two indicators are still on the side of the scale indicating disagreement.

Question 11 asked the user directly to indicate whether they liked using the GEOpod system. An average response of 4.53 shows a strong affirmative for this question.

The conclusion for this data is that users were uniform in their positive attitude toward the GEOpod system. In general the participants liked the system and thought it was easy to use. It was clear not only from the user responses to the post-test survey but from analyzing video of the tests that navigation was a consistent problem for users. There was some indication, however, that such navigational difficulties may be temporary, as indicated both by one particular participant in the survey and from observing improved user navigational performance as the trials progressed – an expected result.

Video Analysis.

The video recordings of all trials were reviewed in order to uncover problems not revealed by the data above. In particular, evidence of user confusion, execution mistakes, or misunderstandings of the state of the system was cataloged. This analysis uncovered six prominent errors by users, summarized in Table 3, in order by frequency (highest at top).

Table 3. User errors
Full size table

User errors can be caused by a number of factors, not the least of which is simply inexperience with the system. User Errors (UE) 1 and 2 in particular can be accounted for by this. Such problems can usually be effectively eliminated through longer and/or more thorough training.

That being said, however, there are other issues related to user disorientation (UE 2) – a pervasive problem during the trials – including a lack of adequate feedback related to the GEOpod’s orientation. This problem is discussed in more detail below.

Another indicator of concern is User Error 3, indicating a failure to complete at least one step during the trial. Usually this was because the user skipped a step (either intentionally or unintentionally), though sometimes it was because they did not perform the correct prior actions in order to successfully complete a step. Users who noticed their error and corrected their mistake were not flagged with this error.

Some of these failures can be attributed to issues not related to the system’s interface. One solution, certainly, is more thorough training. Another is that the phrasing of task statements may not have been adequately understood by some participants. Examples include phrases such as “note the location” or “parallel to the isosurface.” These problems can be overcome when designing assignments using the GEOpod system through a combination of training and changes in wording for certain tasks.

Table 4 shows the most frequent interface errors identified through video analysis. The most frequent error encountered is a focus problem related to the data entry fields (latitude, longitude, altitude, and gridpoints). As a consequence of this problem, users frequently ended up accidently entering command keystrokes into these data fields. While in all instances the users noticed the problem and were able to correctly fix it, this is still a significant problem of the interface as it slows users down significantly.

Table 4. Interface errors
Full size table

The second significant problem users had was in selecting the current values in these data fields prior to modifying the fields. It is not clear whether anything can be done about the current awkwardness of this interaction.

Recommendations.

The expert review, usability study, and video analysis were all aimed specifically at identifying potential problems with the interface design. As a result of these studies, 28 specific recommendations were made for changes to the interface. The most noteworthy problems were

  1. (1)

    Buttons that didn’t act like buttons

  2. (2)

    Data entry boxes that created confusion as a result of focus problems

  3. (3)

    Navigational confusion due to lack or inconsistency of controls

As an example of the kind of navigational problems encountered, a frequent problem for users was disorientation in terms of the current position and orientation of the GEOpod. For instance, it was possible to get the pod upside-down without knowing it, since there was no feedback telling the user the pod’s orientation. In such instances, many navigational commands would often do the opposite of what was expected, which both confused and frustrated users.

A complete list of recommended changes to the interface can be found at [12].

4.2 Updated Interface

Figure 2 shows a redesigned interface for the system. Most of the elements of this display were redesigned or enhanced as a result of the study. Particularly note the area designated as feature 9, the new navigational display, which now includes both a compass and a horizon (attitude) indicator. In addition, a mini-map in the upper left corner (feature 2) provides an overhead view of the volume overlaid on a world map.

Fig. 2.
figure 2

Updated GEOpod interface

Full size image

These aids were added specifically to overcome the navigational problems experienced by users. In addition, all buttons were redesigned to be reactive, and additional controls such as a speed indicator (feature 6) were added.

5 Educational Outcomes

A detailed study of the educational benefits of the GEOPOD system for meteorology students has not yet been conducted. Plans are being made to conduct such a study.

A limited beta test of the system was conducted with 64 students in two meteorology classes. Among the results of this beta test, a post-use questionnaire asked students to respond to questions regarding whether the GEOPOD system enhanced content understanding. An “overwhelming majority of students (75 %) responded to this question in the positive… the majority felt that the GEOpod was helpful because it was visually compelling (41), some (7) expressed that it gave them a chance to explore patterns and relationships in the data displayed and gain a deeper understanding of the interrelationship among concepts. Still others (5) responded that they “enjoyed the active or kinesthetic aspects of the technology (e.g. flying around inside the jet stream and being able to set parameters)” [13].

6 Future Work

The project hopes to implement two additional studies. The first is a second usability study to determine whether the redesign does indeed correct the user problems noted in the first study. Secondly, a more comprehensive educational study of the use of the GEOPOD system in actual meteorology classes would be used to validate the overall educational benefits of the system.

7 Conclusions

Overall, student reactions to using the GEOPOD system have been overwhelmingly positive. The 3-D visualization of the system was particularly appealing to students, as was the opportunities to explore the data through directed activities. The system shows promise in assisting students to gain a deeper understanding of meteorological principles.