Keywords

1 Introduction to the TIME Program

With 17.6 million Americans insured under the Patient Protection and Affordable Care Act as of September 2015 [1], it has been noted that there will be a shortage of primary-care doctors available in the foreseeable future [2]. To address these concerns, education curricula are focusing efforts to accelerate clinician programs [3]. In conjunction with the acceleration, curricula are providing a larger focus on patient-centered care by emphasizing patient safety and evidence-based practices [4, 5].

In collaboration with the University of Texas Southwestern Medical School (UT Southwestern), the University of Texas at Dallas (UT Dallas) joined the Transformation in Medical Education (TIME) initiative. The goal of this initiative is to increase the amount of clinical caregivers through the dual admission into the undergraduate BA in biology and MD training program. Through this initiative, students accepted into the program will have exposure not only to the traditional premed classes typically offered, but they will also engage in material that is designed to enhance their medical professionalism so that the students are better prepared in administering clinical care [6].

2 What Is the UT TIME Portal?

The University of Texas Transformation in Medical Education Portal (UT TIME Portal) was developed to complement the clinician programs at UT Southwestern. The UT TIME Portal is asynchronously accessible via a web dashboard and consists of: a pre-survey to gather demographics and obtain general knowledge/attitude levels, an introductory episode on platform usage, two instructional gameplay episodes, a discussion board, and a post-survey to gain feedback and measure knowledge/attitude change. The UT TIME Portal gives premed students the opportunity to interact with a virtual patient in the scenario of a medical interview, as well as engage in a scenario regarding a medical professional’s interaction with social media. As a complement to the learning modules of the UT TIME curriculum, the UT TIME Portal was tasked with offering the premed students the opportunity to practice interpersonal skills with asynchronous access.

The first gameplay episode that is experienced is the “Medical Interview Episode.” In the Medical Interview Episode, students assume the role of a practicing clinician who is responsible for interviewing a patient named Walter and his wife Susan. Walter self-reports the symptom of a painful headache while his wife, Susan, interjects subjective statements. The premed student’s responsibility is to conduct the medical interview using rapport, empathy, and active listening. Using these techniques will enable the premed student to: extract the relevant information from Walter, professionally respond to Susan’s interjections, and, ultimately, diagnose Walter’s ailment.

The second gameplay episode, the “Social Site Episode,” covers behavior in communications on a simulated social media site and illustrates potential harmful consequences to patient privacy. In the scenario, the premed student is presented with the opportunity to post vaguely about an interaction with a patient on an interactive social media site. The premed student then experiences a simulation of peers’ comments and conversations regarding their social media post. The conversation leads to the premed student divulging sensitive medical information for a patient, only to see the situation unfold into a serious HIPAA violation. The episode is designed to be unwinnable in order to demonstrate the dangers of posting to social media sites about patient care and how closely one’s professional life is tied to unprofessional social media conduct.

The UT TIME Portal offers a large variety of engagement measures, both quantitative: player and team score, badges, stars, and usage metrics; and qualitative: open-ended survey questions and a moderated discussion board to motivate and support learning objectives.

The impact of education-oriented serious games depend not only on the content they are designed to teach, but also the proficiency of the teacher and the context in which they are presented in the curriculum [7, 8]. In order for serious games to appropriately deliver their intended subject matter, the students need to be actively engaged in the software [9] in order to facilitate learning. By presenting students with the chance to actively engage in the material, effects such as self-reference enable stronger encoding processes [10]. By leveraging students’ motivation to learn the material and their role of active participation with the learning platform, one can facilitate deeper learning [11]. The learning portal is intended to engage active participation from the premed students, where users cannot simply click through dialogue trees; users must attend to the contextual scenario and respond appropriately in the given episode.

The UT TIME Portal is integrated within a summer curriculum where the students engage in a variety of learning modules, including the opportunity to conduct a standardized patient interview. With the placement of the UT TIME Portal in this curriculum, student engagement with the material and depth of learning practical clinical skills could be enhanced.

3 The Changing Nature of Media and Data Analysis

The development of new technology brings new forms of data [12]. As an example of new technology, since 2006, the platform of smartphones has been adopted by consumers, manufacturers, and developers, becoming a normal part of everyday use and an integral part in data collection [13]. Further, social networks– such as Facebook, Twitter, and LinkedIn – track user interactions, interests, and the creation of content to the degree where an individual’s online identity mirrors their real world identity to a striking degree [14].

Large searchable databases have enabled researchers to easily query to stored results and pull from datasets [15] that have unprecedented size and growth; Twitter stores upwards of 340 million tweets per day [15]. While the size of these datasets is immense, the syntactic nature of the stored data is difficult to analyze using traditional methodologies, as they were designed for much smaller sample sizes and number of variables. Novel analyses are still under development and review that are better capable of handling the size and complexity of these emerging datasets [15]. Our research here contributes to exploring new relationships between data gathered through the traditional i.e., survey, and emerging data collection methods, i.e., gameplay and qualitative metrics, collected through the UT TIME Portal.

4 Benefits of Quantitative Data

In spite of the changing nature of data and data analysis, quantitative data still serves as an important foundation for data collection in the UT TIME Portal. Quantitative research has been the standard for the collection of empirical data and “it is also known as the scientific research paradigm,” [16]. User metrics enable researchers to analyze the sample that has been collected through descriptive and inferential measures [17]. The design of the UT TIME Portal enabled the opportunity to collect large amounts of quantitative data. The implementation of the portal enabled quantitative data to be captured using objective instructions and stored in a structured database, which is paired to a corresponding user. Through purposeful design, researchers can capture every meaningful interaction that a given user has within the portal. These measures can give the researcher insight into user behavior which can address questions with precision and accuracy.

In the case of the UT TIME Portal, numerous usage metrics were quantified and captured. Details such as number of repeated sessions played, length and scores of the session, and badges received from each user session were informative as to how the UT TIME Portal affected self-reported knowledge gains on the post-TIME survey. Users’ discussion board posts were also categorically labeled with stars by the instructor of the course; the stars were awarded to students’ posts that were considered insightful in relation to the question posed by the instructor.

5 Unique Applications of Qualitative Data

The UT TIME Portal was also designed to capture users’ qualitative data. Qualitative research is carried out with the same goals as quantitative design – to yield data that is systematic, reproducible, and cumulative [18]. Many methods exist for qualitative data analysis, such as: constant comparison analysis, classical content analysis, keyword-in-context, word count, domain analysis, taxonomic analysis, and componential analysis [19]. Qualitative research seeks a better understanding of the context of a given dataset that a quantitative exploration may not capture. Qualitative data analysis software tools also ensure accuracy of the performed analyses by minimizing analyst errors [20].

While quantitative metrics are largely informative of a group of participants’ performance with the UT TIME Portal, quantitative data has difficulty capturing the experiences the user has through their interaction. Anecdotally, researchers at the Rotterdam Eye Hospital carried out a qualitative study on software designed to educate and support children suffering from an eye disorder; the researchers found that the qualitative study better captured the users’ interactions with the portal compared to what their quantitative data would have been able to yield [21]. Interviews revealed that children with the disorder were largely unaffected by it and felt little need to use the software, whereas quantitative data alone would only indicate low usage, which could have been attributed to a knowledge or technological barrier as opposed to a fundamental disinterest in the software.

In a similar manner, the impact of the UT TIME Portal on a user would have less depth if the portal was designed to exclusively capture quantitative usage metrics. In order to address this limitation, the researchers at UT Dallas also collected the participants’ qualitative data provided by their discussion board posts and responses to open-ended survey questions. These data offer opportunities in a variety of different areas that are of benefit to researchers. Each user’s qualitative data would help measure their engagement with the content, as the response the students’ provide will offer their level of awareness of the scenarios presented to them. Posts can also be used as a measure of how the students learned – i.e. were the students thinking critically about the material presented. As shown in Fig. 1, immediate emotional responses could also be recorded, as the students were asked for their reactions after experiencing the episodes. Finally, the posts were reviewed by a professor who was given the opportunity to award stars to posts that were deemed insightful; using this information, it was possible to evaluate the quality of posts the students were making.

Fig. 1.
figure 1

The UT TIME Portal captured student emotional response to help analyze their engagement and experience. Students demonstrated empathy towards the virtual characters and evoked strong emotions, indicating that they were actively engaged with the UT TIME Portal.

Full size image

6 Mixed Methods

Through the utilization of both quantitative and qualitative techniques, one can address the limitations of each methodology. In the UT TIME Portal, there were a total of 2572 lines of text from 423 discussion board posts, and two self-reported surveys completed by 74 premed students. Due to the text-based and syntactic nature of qualitative data, the sheer amount of possible records would be difficult to analyze. With qualitative data analysis software, it was possible to quickly organize and categorize this data in an efficient and thorough manner.

While a single user’s self-report of their experience is quantifiable, when a learning platform is intended to be experienced by larger samples or populations, more deliberate methodologies can be implemented in order to account for user behavior. To this end, the qualitative data received from the UT Time Portal was analyzed using the software suite NVivo 10. A review of 763 empirical articles has found that the number of studies reporting use of qualitative data analysis software is increasing each year, is published primarily in health sciences journals, and is most commonly used for data management and analysis [22].

The ability to meaningfully quantify emotional qualitative data is in its early stages, as researchers provide methods to capture and semantically assess qualitative entries [23]. Using prototypes for emotional categorization – a hierarchy of basic-level and subordinate-level emotions outlined by Shaver [24] – researchers have been able to develop automatic emotional identification algorithms in media analysis by implementing a combination of large training datasets and machine learning algorithms [15].

In spite of these developments, qualitative methodologies have largely been ignored by researchers as a method for analyzing serious games. A literature review of 129 papers reporting empirical evidence of serious games showed that only 8 of those papers reported qualitative data [25]. This could be attributed to a multitude of factors, such as the daunting datasets, lack of proven software and analytical methodologies, or a disregard of qualitative data as a whole.

7 Methodology

The data gathered from the 74 premed students using the UT TIME Portal was captured over the course of two annual sessions: in 2014, 54 students were enrolled, and 20 students were enrolled in 2015 [26].

NVivo 10 is a qualitative data analysis software package released in 2012. NVivo’s tools are designed to help users organize and evaluate large amounts of quantitative and qualitative data. NVivo’s “node” functionality allows users to collect and categorize important pieces of data into nodes that may easily be compared to datasets or each other. Through the use of nodes in conjunction with review and visualization tools, users may use NVivo to test hypotheses or correlations in large datasets, regardless of their type of data [19]. With this functionality, it is possible to evaluate this dataset in its entirety with its context intact, leading to a greater understanding of the process and experience of the participants, as a whole.

The UT TIME Portal – with its discussion board and post-surveys – generated numerous reports of qualitative data. NVivo’s “node” functionality was used to collect data pertaining to certain emotions and reactions. Nodes – such as, “Strong emotional response,” “Frustrated,” “Attitude change,” and “Attachment to Virtual Characters” – were created to test hypotheses and assess the effect the learning portal had on the premed students. These nodes made it possible to keep track of important pieces of data and generated new insights through comparisons to quantitative data.

When creating these nodes, NVivo’s word search functionality was used to quickly sift through discussion board posts to find particular themes and emotions. The broadness of a word search may be adjusted to include synonyms and/or related words, speeding up the process and highlighting instances of a word that may otherwise have been missed. These word searches helped to determine if the episodes were having the desired effect, as well as test hypotheses in regards to participants of different demographics reporting certain feelings or thoughts.

After importing and organizing the data and using word searches to fill in nodes, analyses could then be drawn by comparing qualitative nodes to quantitative data. By assigning gender and age attributes to the data, potential correlations between nodes and these attributes could also be tested. NVivo’s tools were used to quickly organize and sort data, validate hypotheses, observe data correlations, and gain new insights in an efficient and thorough manner.

8 Results

NVivo was used to search for commonly occurring themes in the data and create 11 nodes, grouping responses in ways that evoked engagement, displayed affection or attachment to virtual characters, and provided other insights into user experience. For example, a node titled “Attachment to Virtual Characters” was created to analyze if and how participants reacted to virtual characters. Searching for keywords such as “feel” when used in conjunction with the names of the virtual characters revealed that many users explicitly expressed attachment by empathizing with the characters, feeling bad for them, and often referring to them as if they were real.

The students shed much light on useful design considerations and improvements for keeping both the patient and the caregiver engaged during the Medical Interview Episode. As stated earlier, in this episode the student conducts a medical interview with Walter, and his wife, Susan. The student must keep the interview focused on Walter and his symptoms in spite of Susan’s interjections. Some students suggested non-verbal techniques like using body language and eye contact as means to switch attention back to the patient from the overly helpful caregiver.

Even though the caregiver Susan was modeled as a distractor in the medical interview, students still emphasized her importance and helpfulness in their discussion board posts, as well as concern for her feelings and making her feel included in the interview. For example, one student remarked that, “It is important to acknowledge the caregiver and use their input to ask appropriate questions so that the patient history is more complete.” Students also stressed being “respectful” to the caregiver and avoiding being “rude.”

These discussion board posts were also informed by the quantitative pre- and post-survey data. Both surveys asked the students to rate their agreement with the following statement: “I consider a caregiver a significant source of information during a medical interview.” In 2015, five out of 20 students reported negative movement in this area, meaning their ratings went down from pre- to post-survey (they rated “Strongly Agree” on the pre-survey, and went down to simply “Agree” on the post-survey). Even these students – whose opinions of a caregiver’s importance seemed to be affected by the Medical Interview Episode – qualitatively agreed that the caregiver was still an important source of information and that both her helpfulness and her feelings should be taken into account. One of these students wrote, “I would definitely take Susan’s observations of the chief complaint into account, as she brings an entirely different perspective of the chief complaint and can give information about the chief complaint that Walter either is not comfortable sharing or is not aware of.”

The Social Site Episode, as previously mentioned, simulates a social media site and demonstrates how sharing medical information online can lead to breaches of patient privacy. In this episode, Walter’s daughter, Wendy, learns that her father had a stroke when people start posting about it on social media. When analyzing the data for the Social Site Episode, NVivo’s built-in chart functionality was used to compare the “Attachment to Virtual Characters” node to quantitative demographic user attributes, such as gender and technological inclination, to discover that a user’s technical inclination or gender did not influence the level of engagement. Most responders, regardless of their technical inclination, preferences, or gender were concerned for Susan’s feelings, frustrated at the Social Site Episode, and wary of posting important things online for others to see. Notably, the least technological savvy respondent showed clear empathy while describing feelings for “poor Wendy” and her situation with Walter.

The majority of students had strong negative emotional reactions to the Social Site Episode dealing with professional online behavior, using words like “frustrated,” “hopeless,” and “despair” in their discussion board posts. These strong emotional responses also led many students to discuss professional behavior in general and the importance of keeping information private in any situation [26]. Some students took this as an opportunity to research and better understand laws regarding de-identification of patient information. Many students felt appalled by how quickly sensitive information can be compromised and disseminated, and agreed that it is best to not share any information if possible.

Fig. 2.
figure 2

In 2014, Students reported positive movement on the question, “Do you believe that a medical student should be held accountable for unprofessional behavior discovered through postings on his/her personal social networking page(s)?” In 2015, however, students reported no movement between pre- and post-surveys. However, analyzing qualitative data gives us additional insight into student learning gains.

Full size image

All students sympathized with the patient and some felt a need to show empathy: “this result caused me to feel regret for the patient and has driven me towards providing a sense of compassion and empathy to help the patient overcome this difficult time.” Another node, titled “Strong Emotional Responses,” consisted of references such as: “helpless,” “overwhelmed,” “very sad,” “out of control,” “frustrated,” “distressed,” “extremely irritated,” “angry,” “upset,” “shocked,” “uncomfortable,” “annoyed,” “frightened,” and displayed that most students (74 % from 2014; 90 % from 2015) had strong negative emotional responses to the Social Site Episode.

A substantial number of all discussion board posts were awarded stars by the instructor: 43 % of posts from 2014 received stars (133 out of 306), as did 51 % of posts from 2015 (59 out of 115). This categorical data provided by the instructor shows that a large portion of discussion board posts evoked engagement through their quality and insight.

Quantitatively, students were asked the question “Do you believe that a medical student should be held accountable for unprofessional behavior discovered through postings on his/her personal social networking page(s)?” and prompted to answer “yes,” “no,” or “not sure” on pre- and post-surveys. This question was posed in both the 2014 and 2015 UT TIME Portal sessions and yielded contrasting results. In 2014, there was a positive movement between the pre- and post-surveys: 74.55 % of students answered “yes” on the pre-survey, and 86.96 % of students answered “yes” on the post-survey, indicating a trend toward “yes.” However, in 2015, students reported no movement at all on this question, with 85 % of students reporting “yes” on both surveys, as shown in Fig. 2 [26]. As stated earlier, students received other learning modules outside of the UT TIME Portal within this timeframe as part of their educational experience.

9 Discussion

The qualitative data collected provided insight into the effect the episodes had on the 74 premed students in our study. Through analysis of quantitative and qualitative data generated by the UT TIME Portal, a deeper understanding of the students’ experience could be gleaned.

Emotional responses were consistently presented across the premed users, demonstrating an opportunity to capture and understand the interactions and impact of the learning platform in a way that quantitative methods would not allow. If the UT TIME Portal was not designed with a feature-set capable of providing both a suitable learning environment and a meaningful interaction with its users, the emotional responses from the users would not have been captured.

Technological predisposition had no effect on an individual’s ability to be engaged with the virtual characters and the scenarios presented to them. In the case of these premed students, having the opportunity to access meaningful content provided them additional hands-on experience.

The Medical Interview Episode was one learning module available to help students gain insight on how to provide meaningful clinical care for patients. When presented such a situation and given the opportunity to provide feedback, the premed students demonstrated their medical knowledge and offered meaningful suggestions that would improve the interaction with the patients in the Medical Interview Episode. One student suggested: “an active way to keep Susan happy without sacrificing focus on Walter would be to ask her what she has been doing to help Walter. This line of questioning acknowledges her role as his caretaker while probing information about Walter’s health simultaneously.” These insights show that students were thinking critically about the situations presented in the learning portal and demonstrating engagement.

Students also demonstrated empathy and patient communication skills through their responses to Susan, the caregiver. Even though Susan’s actions during the interview were distracting and overbearing, students still emphasized her potential helpfulness and offered ways to keep Susan’s feelings in mind while staying focused on the patient. Notably, these feelings and insights were shared by students who reported negative movement in this area on the pre- and post-surveys, meaning that their opinion of a caregiver’s helpfulness dropped after they played the Medical Interview Episode. Quantitative data analysis alone might have determined that these students found Susan’s overbearing nature to be detrimental to the medical interview, but qualitative data shows that their discussion board posts still emphasize Susan’s helpfulness and the importance of treating her respectfully, in spite of their negative movement between the pre- and post-surveys. This combination of qualitative and quantitative data generated new and important insight about the students’ experience and opinions.

The Medical Interview episode presented students with a difficult, nuanced situation. In such an interview, the clinician must extract relevant information from the patient and the caregiver while keeping the interview focused and efficient. The clinician must stay focused on the patient while keeping the caregiver’s feelings in mind and making them feel included. They must also think critically about the information they receive and quickly determine its relevance and implications. While the quantitative results show that students performed well and understood the concepts, the qualitative data is especially informative in that it gives insight into the students’ depth of understanding. Discussion board posts detailing strategies for the interview or additional questions to ask demonstrate deep learning and a thorough understanding of the nuances present in the interview. In this sense, the qualitative data informs the quantitative data and offers potential explanations of why students performed well or gave certain answers. Students may have performed well because they thoroughly understood the situation and the strategies necessary to work through it.

The Social Site Episode elicited a strong emotional response from the students, emphasizing the importance of electronic professionalism on social networking sites. The Social Site Episode presented students with a plausible scenario that was relevant to their future careers and the hyper-connected world they currently live in. To these students, this scenario unfolding for these virtual characters was emotionally impactful. This emotional response may have been noticeable through pre- and post-survey questions regarding social media and accountability, but through qualitative data collection and analysis the magnitude of these emotions could begin to be understood. Discussion board posts showed that students felt great empathy for the virtual characters and frustration towards the events in the Social Site Episode. Some students even made strong declarations about the potential dangers of social media and the importance of keeping the details of one’s medical career off of public forums – no matter how small. Qualitative data collection and analysis showed the true impact of the episode and allowed for assessment of its personal and emotional effects.

Quantitative pre- and post-survey data was informed by qualitative data. As mentioned earlier, when asked the question “Do you believe that a medical student should be held accountable for unprofessional behavior discovered through postings on his/her personal social networking page(s)?” students reported positive movement in 2014 but no movement at all in 2015. When looking at quantitative data alone, this seems to indicate that the Social Site Episode had little impact on the 2015 students. However, qualitative data shows that these students still reported empathy and strong emotional reactions to the episode, and understood the importance of information privacy. In this instance, the qualitative data reveals information about the students’ experiences that the quantitative data does not seem to indicate. In contrast to survey responses, the Social Site Episode still had a strong effect on the students, even if they did not report positive movement.

The stars add an interesting layer to the dataset, in the sense that this variable can be evaluated in both a quantitative and qualitative sense. As a quantitative metric, the stars provide a measure as to which students were thinking critically. When evaluated as a qualitative variable, one can quickly see which posts were considered as perceptive.

10 Summary

Through both qualitative and quantitative data analysis, a deeper understanding of student knowledge gains enhanced through learning interventions can potentially be obtained. In our study, qualitative data analysis indicated that students understood nuances of the gameplay episodes and possibly experienced deep learning – observations that could not be drawn through quantitative data analysis alone. New and developing media – such as serious games – provide a mix of qualitative and quantitative data and will likely continue to provide new types of methodologies to more fully understand participant experiences.